A prominent US-based biopharmaceutical company with a global presence was looking to overhaul and stabilize the technology stack that dealt with their international data. Agilisium was entrusted with this responsibility, owing to its extensive expertise in managing large workloads on the cloud.
The client’s international product, sales and marketing data is currently stored in Oracle, SAP, various internal systems and flat files. This data is used and updated constantly by the organization’s sales representatives during field visits to medical professionals.
Previously, the inbound data was staged via an S3 bucket and was processed through HDFS and moved to Redshift after applying SnapLogic based ETL logic and business rules. However, the HDFS system had very high latency as it did not work well with SnapLogic. In addition, the business criticality of the data meant that the business team would query data directly from HDFS while it was still being processed to RedShift. This led to additional performance issues.
Since this data included business critical pricing numbers and sales figures, there was no room for error. The organization also has very short 12-week budget cycles, which required the project to be completed on a very stringent deadline.
When the original system was built, HDFS was determined to be only suitable technology to process large amounts of data. However, AWS had matured since then. The same data processing capabilities were now offered by the Amazon RedShift. Consequently, the underperforming HDFS file processing system was disconnected. New snaps were built linking the S3 buckets that housed the inbound data directly to Redshift, which now fulfilled the role previously held by the HDFS system. RedShift in turn was connected directly to the user applications.
Apart from this, additional enhancements were performed on the application owing to business requirements. For example, previously, data from multiple international regions were stored in a single common RedShift database. This not only reduced the performance, but also increased the time taken for new development or changes.
The newer stable architecture separated the combined data and moved it into two distinct databases for different international regions.
The migration also brought about additional complications. For example, the pricing information was very sensitive and was required to be maintained accurate to the level of decimals once it was migrated. These requirements were also sufficiently resolved as a part of this migration project.
The new technology stack lead to a decrease in data processing time by over 50%.
The estimated timeline for a migration of this scope is typically more than 16 weeks. However, it was delivered in 12 weeks by ensuring that the migration processes were done in parallel and quality control was built into the process to ensure that things were done right the first time.
The client was extremely satisfied with the project and shared the following testimonial,
“The result is impressive, we have reduced the loading time by more than 50%, and eliminated all the issues related to Hadoop/HDFS. Your team has demonstrated responsibility and accountability for the delivery. With confidence in your delivery capability, we added many more people to multiple MVP 2 (May to July) teams……. Agilisium is the only partner we use for this MVP 2 technology transformation. “