Implementing long running ETL deployments with AWS CloudFormation

Enabling Devops in RSAC

Highlights

15 minutes

of Infrastructure deployment process reduced

98%

of deployment time potentially reduced

Faster ETL job deploy and execution process

Significant time saved for IT team due to automation

Allows for better planning and execution

Client Profile

Reliance Steel & Aluminum Co. (RSAC) is one of the largest metal services provider in North America. The company provides value added metals processing services and distributes a full line of over 100,000 metal products to more than 125,000 customers across various industries. RSAC is a Fortune 500

Business challenges

Agilisium Solution

Automated Infrastructure deployment using AWS Cloud Formation Templates.
Automated High Availability to failover onto a different AWS region using Cloud Formation Template.
Automated Application deployment using AWS AirFlow scheduler. Spark based ETL jobs triggered by AirFlow as and when needed.
EMR Spark based framework developed to run a distributed application environment.
All Source code repositories managed using Gitlab.
AWS Autoscaling features implemented to scale out the application using CloudWatch metrics.
Ansible playbook to manage and ease the deployment process.

Tech-stacks Used

glue

AWS – S3

AWS Cloudwatch

AWS EMR

eventbridge

lambda

stepfunction

SQS

SNS

spark

Airflow

Business outcome

Infrastructure deployment process reduced to 15 minutes, potentially reduction of 98% time when compared to manual deployment.
Faster ETL job deploy and execution process.
DevOps culture encourages collaboration & eliminates silos.
Significant time saved for IT team due to automation.
Allows for better planning and execution.
Lesser resources required. Developers can now focus on core activities.