Migrating to Amazon Aurora Postgres helped Reliance Steel & Aluminum Co. achieve Near real time Insights, Improved Scalability and Better Data Quality
AWS Serverless near real time architecture for migrating on prem SQL Server data changes to Amazon aurora.
Better Report experience
Improved database performance Decreases the database management cost
Reduced process burden on engineers
Improved scalability and performance of the system
Reliance Steel and Aluminum Company (RSAC) is North America’s largest metals service center company, offering 100,000+ products and value-added processing and serving over 125,000 customers primarily by providing metals processing, inventory management services, and quick delivery.
- On-prem and legacy data models were modifying/overwriting record data all the time. As a result, RSAC couldn’t have a single source of truth.
- RSAC required an immutable data storage/system where they can run Adhoc querying, reporting, and third-party integration.
- RSAC needed real-time transactions to data warehouses and low-latency data transfer to operational and analytics users with low production impact.
- Batch loads and periodic reloads with the latest data take time and often consume significant processing power on the source system.
- AWS Lambda was put in place to handle and process the core data flows.
- CloudWatch rule triggers the Lambda process every 5 minutes.
- Lambda connects to an on-prem SQL Server and extracts CDC data.
- Simultaneously, Lambda applies business rules within the extracted CDC data and stores the processed records to Amazon Aurora tables.
- EMR Spark-based framework is developed to run a distributed application environment.
- Migration Readiness was done using Agilisium’s Readiness template.
- Gathered preliminary information which was required for migration readiness and is segregated into success factors, workload, and security.
- Helped us to identify the type and volume of workload, stakeholders, and security-related information.
- Performance Consideration
- Measured and evaluated the optimum capacity units like memory size, the startup time for lambda.
- Controlled dependencies and Minimized deployment package size, which in turn would reduce the amount of time taken for the deployment package to be downloaded and unpacked ahead of invocation.
- The following metrics were captured and monitored as part of the Performance metrics.
- Service Response Time
- Database Query Throughput
- Memory usage (CPU Utilization)
- The Custom Lambda framework had the capabilities of fault tolerance and failover.
- Failover decision: If the system goes through a failure, the restart process gets initiated from the point of failure as instructed in the custom Lambda framework.
- Disaster recovery was in place by leveraging parallel execution techniques.
AWS Identity and Access Management