Case Study
Built a solution to report the top performing shows and improve sales for an American mass media and entertainment conglomerate

About the Client

Our Client is an American mass media and entertainment conglomerate owned by Comcast. It is one of the world’s leading media and entertainment companies in the development, production, and marketing of entertainment, news and information to a global audience.

Challenges

  • Lack of Personalized user experience due to the absence of Personalized recommendation Engine
  • Need to Fetch information from different brands / platforms for a single user and provide recommendations which maps to the right brand was required
  • Revenue impact due to high churn rate – absence of personalized content
  • Need to handle real time data

Solution Highlights

  • mParticle collects Event / click stream data from 10 brands and 6 platforms
  • Iterable CRM data is routed thru mParticle’s web hook consumer to deliver data using Kinesis Stream
  • Kinesis firehose is created to each Brand source to write streaming JSON data into S3
  • Data is processed and staged into S3 and then copied into Snowflake table for standard reporting
  • Kinesis Data Stream connected to mParticle application to fetch subset of fields
  • Recommendation engine program built on Python to run on Spark is run on Local Spark instance using spark stream periodically throughout the day
mParticle application
  • Event/Click Stream Data from 10 Brands & 6 platforms
  • Kinesis firehose is created for each stream to write streaming JSON data into S3
  • JSON data from S3 is batch processed once a day on EMR and written to S3 Data Lake partitioned by Brand > Event Year > Event Month > Event Date > Platform in parquet file format
Iterable CRM
  • Iterable CRM data is routed thru mParticle’s web hook consumer to deliver data using Kinesis Stream
  • Kinesis firehose is created to each Brand source to write streaming JSON data into S3
  • JSON data from S3 is batch processed once a day on EMR and written to S3 Data Lake partitioned by Brand > Event Year > Event Month > Event Date in parquet file format

To fulfill SocialHi’5 need for a client self-service portal that was also easy to maintain, Agilisium’s 5-member expert team built a custom web application with a heavy focus on the visualization of campaign outcomes. They also developed in parallel a DevOps process to maintain, scale and operate this portal.

Web Application Architecture


A variety of AWS services and some open source technologies were used to build and run the web application. The web layer used the PHP framework, included a login and authentication system, and used AWS QuickSight to render its outcome dashboards.

The app layer was built on Python, and the backend services were run on Elastic Container Service (ECS) dockers with Auto Scaling and Auto Load Balancing (ALB) to ensure high availability of the portal. The database was run in a private subnet and used RDS MySQL as the database service.

DevOps Process:

As mentioned earlier, SocialHi5 necessitated that the solution offered was easy to maintain, scale, and operate. To that end, Agilisium’s DevOps engineers developed a 2-part DevOps process focusing on

  • CI/CD for web application development
  • Infrastructure Provisioning for maintenance.

Continuous Integration/Continuous Deployment (CI/CD Process)

All application (Web & App Tier) maintenance was articulated via AWS’s Code Pipeline. AWS’s Code Commit, Code Deploy, and Code Build services were invoked to automate the enhancement and maintenance of the self-service portal.

CI/CD Process Flow: Web Tier


CI/CD Process Flow: Web Tier


Infrastructure provisioning

All infrastructure was hosted on an exclusive SocialHi5 Virtual Private Cloud (VPC), to add an extra layer of confidentiality. AWS CloudFormation templates were used to spin up and maintain a host of AWS services utilized for the self-service portal.

Serverless Web application hosting: EC2, ECS, RDS, S3, SSM, VPC, NAT Gateway, ALB with Autoscaling Group, LAMBDA, Certificate Manager, Route53 were some of the services used to get the portal live.

Security: Web Application Firewall (WAF) was used with Cross-site scripting, Geo match, and SQL injection rules to protect from common cyber threats in conjunction with the AWS inspector service.

Monitoring and Logging: CloudWatch, OpsWorks, Config & Inspector services were also invoked to cover configuration management, logging, and monitoring of the application and infrastructure.


Real time Recommendation
  • Data is processed and stored back to S3, and then copied into Snowflake table for standard reporting
  • Kinesis Data Stream connected to mParticle application to fetch subset of fields
  • Kinesis Firehose is connected to it to sink the JSON data to S3
  • Recommendation engine program built on Python to run on Spark is run on Local Spark instance using spark
  • Final results are presented as JSON that is then pushed to DynamoDB for frontend application to present it to end consumers
Security
  • IAM best practices and principles are followed
  • AWS Certificate manager is being used for Load balancer and CDN certificates
  • All Data stores are in private subnet
  • Amazon KMS is used for encryption of data at rest
AWS services used:
  • Amazon Simple Storage Service
  • Amazon DynamoDB
  • Amazon Kinesis
  • Amazon Kinesis Data Streams
  • AWS CloudFormation
  • AWS CloudTrail
  • Amazon CloudWatch
  • AWS Identity & Access Management
  • AWS Key Management Service
  • Amazon Kinesis Data Firehose
  • AWS Lambda
  • Amazon Simple Notification Service
Results and Benefits
  • Personalized User recommendation which improved overall customer satisfaction
  • Kinesis Stream were leveraged to handle 5TB of real time data per day to process and provide near real time recommendation to the customer
  • Reporting feature enabled business users to identify the top performing shows, thereby overall improvement in the sales process and enabled business users to drive in sales
  • Overall improvement in Stats
  • Reduced TCO up to 20%
  • Improved Retention rate by 70%