Agilisium’s AWS Big Data solution enables speed of thought insights for Universal Music Group

Data Convergence in AWS Cloud Computing

Highlights

data reconciliation time reduced from 48 to 7 hours
70%
lesser storage cost with Cloud DW
Query deadlock avoided
Quick results for data analytics
data security (encryption in s3)|

Client Profile

The client Universal Music Group (UMG) - a global music conglomerate and one of the big three music companies internationally - faced challenges in divining insights from the enormous amounts of data shared by its streaming partners.

Business challenges

  • The existing MS-SQL based on-premise data warehouse (DW) was struggling to scale up and process the exponential incoming volume of data from streaming partners (Spotify, Apple, YouTube etc.). This had a proportionate impact on all downstream processes and delayed key tactical business decisions.
  • Due to the slow processing time of the existing system, deep dive analyses such as Sales as of LYSD (Last year same day) was impossible.
  • Upgrading the existing system was ruled out due to the potentially exorbitant licensing cost of additional servers/tools to handle the new volume of data.
  • UMG was looking out for cost efficient, scalable solution that does not undermine speed and business agility.

Agilisium Solution

  • Agilisium devised a cloud based, elastically scalable architecture that offers faster analytics and business agility in a cost-efficient manner.
  • Agilisium was onboarded to consult, guide and execute the migration of the MS-SQL DW to Redshift. Initially, a data lake was created with the same on-prem table structure. This made it easy to integrate around 300 existing MicroStrategy reports minimizing or eliminate the impact on end users. Subsequently, the data was migrated from MS-SQL DW to Redshift. The streaming data was first enriched using Hive on EMR and loaded into S3 as multi-part files. The processed data was then moved into RedShift via a data pipeline using the bulk load copy command. The team followed best practices like enriching source data and bulk load copy command allowed for a rapid, high quality data migration.
  • In addition to the migration, Agilisium reworked the entire flow of the data to better serve multiple downstream services like Qubole which enabled UM
  • G analysts to query the raw data as needed for deeper Analytics, leveraging Data Lake built by Agilisium. At the end of the year-long migration a total 200+ TB was moved at the rate of 2 TB/day with an additional 250+ million records being added to the new RedShift DW every

Tech-stacks Used

emr
EC2
Qubole
DataPipeline(one of the component in AWS(like Airflow))
JAVA
Python
GitHUB
Redshift

Business outcome

  • Near real time synchronization of the records helped UMG gain insights at the speed of thought. Month-end data reconciliation time reduced from 48 hours to under 7 hours.
  • Cloud DW with flexible storage costs 70% lesser month on month than client’s previous set up.
  • Cost effective deep dive analysis now possible due to sub second response times to downstream services. This led to UMG’s better understanding of consumption patterns & affinity to decide where to focus & invest ad dollars.
Get the Most Out of Embedded Analytics with Amazon Opensearch 
From data rich visual analytics to interactive dashboards that make data fetching a piece of cake, Team Agilisium is with you, every step of the way. 
Talk to Us
Got a question? Don’t hesitate to give us a call today or shoot us an email. 
Please enter a business email
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.