The lack of a single source of truth, quality data and ad hoc manual reporting processes undermined top management’s visibility of integrated insights on sales, sales rep interactions, marketing reach, brand performance, market share, and territory management. Understandably, the client wanted to align information that has hitherto been in silos, to gain a 360-degree product movement view, to optimize sales planning and gain competitive edge.
The major setback for the Client was, discrete data, & also not in readily available & easily consumable fashion. The established markets were lacking single source of truth & the growing market was lacking proper data collection mechanism. The key challenge is to offer a Cloud based scalable Reporting & Analytics’ environment that can host established & new market data to provide 360-degree view of product, customer & Geography behaviour.
The existing system architecture was discrete and led to siloed data sources. The lack of strong governance undermined scope management and integrated insights generation. The key challenge was to offer a scalable Big Data Analytics solution that supports business objectives, while catalysing enablement of strong governance frameworks at enterprise level.
- Rigid data silos (CRM, database and flat files) resulted in lack of 360-degree view of product movement and sales progression.
- Key decisions on budgeting, incentive planning, understanding & leveraging various channels was not readily available, which needed log more manual effort & also lack of enough data history
- Existing architecture was not conducive for near-real time Advanced Analytics.
- Unable to tap the advance analytics capabilities like AI & Machine Learning so as to better service the customers & patients due to lack of proper Analytics solution.
- Lack of Digital Commercial capability; as targeted marketing plans required integrated insights, which were not viable in the existing architecture.
- Lack of Managed services delivery model and governance process for scope management.
A Cloud One strategy to progressively transition to a globally accessible and scalable Big Data Analytics platform was devised. Agilisium architected, designed and delivered an elastically scalable Cloud-based Analytics-ready Big Data solution with –
1) AWS S3 Data Lake as the single source of truth
2) A Massively Parallel Processing (MPP) Analytics Data Warehouse in AWS Redshift that supports downstream BI & Advanced Analytics, at scale
3) Snaplogic based Metadata-driven generic framework to streamline data validation, cleansing and ingestion into S3 and Redshift. Key generic framework features are as follows:
- The 3-step generic data validation, cleansing and ingestion framework leveraged metadata driven approach to do the below:
- Streamline data loading into S3 Data Lake and Redshift.
- Provide flexibility to define and add / edit business rules.
- Automate manual data validation / correction, and business rules application.
- The framework sends automated email notification to data stewards at each step, whenever a data quality issue is encountered. The stewards view error records through Tableau, upload and approve corrected records for loading.
- User and country-based security through IAM and row-level security is applied in both S3 Data Lake and AWS Redshift Analytics Data Warehouse.
- The data was processed and loaded in multilevel/layered as Raw/ Integrated Granular level data/ aggregated data with Business rules applied in AWS Redshift, to provide Multi-tier sematic layer for Tableau.
Technologies – AWS S3, AWS Redshift, Snaplogic, Tableau
Team size – 1 (Onsite), 3 (Offshore)
Data Size – About 50 MN records for 3 global markets
Project Duration – 5 months (ongoing)
Delivery model – Hybrid
How We Worked Together
Post a maturity assessment, Agilisium devised and enacted strong governance frameworks at executive, program and operational levels with scheduled check point review meetings, to close current gaps identified in scope management.
Scrum calls twice daily ensured that application owners were apprised about progress made at operational level. While monthly milestone review meetings aligned priorities at program level, the quarterly executive steering committee meeting clearly set the engagement priorities at executive level.
To ensure a smooth transition of new solutions, a 1-week workshop and demo on S3, Redshift and Tableau were given to the client’s business and technical teams.
- Holistic 360-degree view – 360-degree view on product movement and sales progression is now available with reduced dependency on IT.
- Quicker Insights – With access to data at lowest grain, business leaders answer their new questions with fewer clicks through self-service BI tool.
- 2x faster territory transition – Metadata has driven generic framework helped the transition of new territories in half the time than planned.
- Advanced Analytics Ready – The new architecture supports scalable Advanced Analytics, at the speed of thought.