– IFPI Global Music Report 2018
The music industry has experienced a revival after two decades of decline. This growth comes post the shift towards a subscription-based, rental services model to stay relevant in the face of changing customer consumption habits. A key challenge for businesses now is how to acquire Gen X - world’s first true digital natives - and turn them into fans with shrinking marketing budgets. In this scenario, marketing intelligence is crucial to making the right business decisions.
UMG’s existing on-premise system could only be scaled up with huge additional investments and even then, it wouldn’t process the incoming data at expected rates. UMG was looking out for cost-efficient new-age solution that would process data faster, provide insights at speed of thought, and enhance business agility. Agilisium was signed on to consult, execute and enable the migration of the client’s existing on-prem system to the cloud, due to its expertise in AWS.
- The existing MS-SQL based on-premise data warehouse (DW) was struggling to scale up and process the exponential incoming volume of data from streaming partners (Spotify, Apple, YouTube etc.). This had a proportionate impact on all downstream processes and delayed key tactical business decisions.
- Due to the slow processing time of the existing system, deep dive analyses such as Sales as of LYSD (Last year same day) was impossible.
- Upgrading the existing system was ruled out due to the potentially exorbitant licensing cost of additional servers/tools to handle the new volume of data.
UMG was looking out for cost efficient, scalable solution that does not undermine speed and business agility.
Agilisium devised a cloud based, elastically scalable architecture that offers faster analytics and business agility in a cost-efficient manner.
Agilisium was onboarded to consult, guide and execute the migration of the MS-SQL DW to Redshift. Initially, a data lake was created with the same on-prem table structure. This made it easy to integrate around 300 existing MicroStrategy reports minimizing or eliminate the impact on end users. Subsequently, the data was migrated from MS-SQL DW to Redshift. The streaming data was first enriched using Hive on EMR and loaded into S3 as multi-part files. The processed data was then moved into RedShift via a data pipeline using the bulk load copy command. The team followed best practices like enriching source data and bulk load copy command allowed for a rapid, high quality data migration.
In addition to the migration, Agilisium reworked the entire flow of the data to better serve multiple downstream services like Qubole which enabled UMG analysts to query the raw data as needed for deeper Analytics, leveraging Data Lake built by Agilisium. At the end of the year-long migration a total 200+ TB was moved at the rate of 2 TB/day with an additional 250+ million records being added to the new RedShift DW every day.
- Near real time synchronization of the records helped UMG gain insights at the speed of thought. Month-end data reconciliation time reduced from 48 hours to under 7 hours.
- Cloud DW with flexible storage costs 70% lesser month on month than client’s previous set up.
- Cost effective deep dive analysis now possible due to sub second response times to downstream services. This led to UMG’s better understanding of consumption patterns & affinity to decide where to focus & invest ad dollars