Built an immutable data storage solution to simplify reporting and third-party integration for Reliance Steel and Aluminum Co.

Data Migration

• AWS Lambda was put in place to handle and process the core data flows.
• CloudWatch rule triggers the Lambda process every 5 minutes.
• Lambda connects to an on-prem SQL Server and extracts CDC data.
• Simultaneously, Lambda applies business rules within the extracted CDC data and stores the processed records to Amazon Aurora tables.
• EMR Spark-based framework is developed to run a distributed application environment.

Migration Readiness

• Migration Readiness was done using Agilisium’s Readiness template.
• Gathered preliminary information which was required for migration readiness and is segregated into success factors, workload, and security.
• Helped us to identify the type and volume of workload, stakeholders, and security-related information.
• Performance Consideration
• Measured and evaluated the optimum capacity units like memory size, the startup time for lambda.
• Controlled dependencies and Minimized deployment package size, which in turn would reduce the amount of time taken for the deployment package to be downloaded and unpacked ahead of invocation.
• The following metrics were captured and monitored as part of the Performance metrics.
• Service Response Time
• Database Query Throughput
• Memory usage (CPU Utilization)

Reliability

• The Custom Lambda framework had the capabilities of fault tolerance and failover.
• Failover decision: If the system goes through a failure, the restart process gets initiated from the point of failure as instructed in the custom Lambda framework.
• Disaster recovery was in place by leveraging parallel execution techniques.