
Overview
The client is a leader in online marketing and online consumer acquisition platform. The high costs of their existing data warehouse on Teradata prompted the client to search for cost-effective alternates. Due to Agilisium’s expertise in modernizing enterprise data warehouse (EDW), we were tasked with performing a migration fit assessment from Teradata to AWS Redshift.
Challenges
The existing DW was platformed on Teradata. The storage cost of exploratory data was high, and system performance suffered due to long-running queries. Besides, dependency on traditional DW (Informatica / Custom SQL) platforms leads to an inability to handle Big Data workloads. Given the varied yet in-depth need for data analysis in an online marketing company, the client also wished for enhanced real-time data ingestion & analytics capabilities.
Solution
Challenged with a sweeping assessment on 30+ TB of data on a legacy DW, the four-member expert team from Agilisium set the following goals for the assessment
- Assess Redshift and related AWS services (S3 Data lake, Athena, Spectrum, Glue Catalog, EMR, Sagemaker) as a platform that supports innovation
- Assess potential performance uplift, cost-effectiveness, and ease of maintenance through benchmarking exercise
- Upskill client IT team, on Redshift best practices and DB maintenance activities through workshops, as required
Based on the above goals, the team delivered the following assessment via a benchmarking POC
- Conversion of Teradata schema to optimal Redshift equivalent using AWS SCT
- Migrate and validate data from 800+ tables into Redshift using custom utilities and optimize Redshift
- Compress and stage data in Glue Catalog & S3 Data Lake Fast facts
- Run benchmark tests (baseline, concurrency) using candidate ETL, BI, and Data science queries for different server instances and WLM settings
- Executive presentation on cost projections and potential performance uplift
- 2.5x performance uplift achieved via minimal code change
- One assessment to check fitment of AWS (Redshift and related AWS services) as a platform that supports innovation