Snaplogic, a leader in the Gartner’s 2019 Enterprise Integration Platform as a Service (iPaaS) Magic Quadrant, offers data integration services to cloud-based enterprise applications. With the advent of Saas based systems offering a plethora of customer services, Snaplogic wished to obtain rich insights out of their customers’ general usage trends, and in turn, serve them better with effective sales strategies.
Currently, Snaplogic uses a metadata layer consisting of a localized Database System, which is quite voluminous and transactional in nature. The incoming business data was growing in data size and volume (at a rate of 6%). This data was unstructured in nature and it needed to be properly streamlined and analytically processed, to derive rich customer insights.
This created the need for an explicit scale-up and a major revamp of Snaplogic’s existing tech stack.
While trying to address Snaplogic’s pressing need, the following challenges were faced:
- Complexities in the Data Structure: The source data structures where the customer usage details are stored, were heavily nested and transactional in nature. Given the massive volume of the incoming data, it was not possible to create a centralised repository, to try and make sense out of this source data and further convert it into understandable formats. These operations resulted in additional efforts and increased overheads for Snaplogic.
- Limited Visibility to Customer Usage Patterns: There was a lack of a structured platform and an established system, to build an in-house analytical engine to derive insights. Due to this, Snaplogic could not approach new customers, achieve maximum trial-customer conversions and carry out proactive sales.
- Lack of Support for effective Customer Tracking and Marketing Campaigns: Given the limitations of the existing architectural set-up, it was impractical to achieve what Snaplogic envisioned as a key business accelerator i.e., effective positioning and expansion of its customer base. This was possible by deploying customized sales strategies (customers tracking and acquisition) and build effective customer relationships.
- No Proper Reporting System for Insights Generation: To address the gaps in the transfer of information between the sales and marketing teams, and achieve a perfect synergy between them, it was necessary to create a centralised reporting and analytics platform to generate quality insights. This could bring the teams in perfect alignment with each other, support effective data driven business decision making and boost sales.
To address these pressing needs, Snaplogic felt that the best alternative would be a low-cost cloud-enabled turnkey solution. This was designed by Agilisium, by leveraging AWS to build an integrated Sales and Customer Usage Analytical platform and using Snaplogic as a data integration tool. The implemented solution addressed the challenges as follows:
This implemented solution brought about the following advantages:
- To address the complexities in the source system, the incoming customer data was stored into Amazon S3 as a data lake layer, a low cost and scalable data storage infrastructure. It functioned as a centralised data repository to build a dedicated analytics platform and carry out the requisite sales analytics.
- For better customer visibility, a Sales and Product Usage Analytics Engine was built on Amazon S3, to serve as a single source of truth. This Architecture, Design and Implementation activities of this cloud-enabled solution were entirely designed by Agilisium’s in-house experts.
- For enhanced customer tracking, the analytics engine was integrated with Snowflake data warehouse system. Snowflake is cost-effective and fully-automated, and can seamlessly integrate with Amazon S3 Data Lakes, while operating at lightning speed with minimal downtime hours.
- To establish an aggregate customer view, a full-fledged reporting layer was created Tableau, which uses graphical representations and customized dashboards. This enabled Snaplogic to visualize and clearly interpret the usage trends of its customers, with proper usage metrics.
In this high-level architecture, the system has been designed to handle over 1.2 PB scale data, collected over a period of 2 years. In this system, the incoming customer data was streamlined and stored into the cloud- based data lakes, for effective data integration processes. This was further processed to support high-value analytics and offer detailed reporting.
The entire system was operational in four stages, with the necessary integration pipelines built using Snaplogic’s intelligent connectors called ‘Snaps’.
- Source Data Ingestion: The incoming business data was gathered from multiple sources (Mongo DB, Salesforce CRM) in different formats (Semi Structured Files, Master Data Management) and directed into AWS’s Storage layers or S3, in their native formats.
- Storage into Amazon S3 Data Lakes: By applying suitable Transformation Logics, this unprocessed raw data stored in S3 was converted into Parquet formats, for further curation. These Parquet files were columnar and compressed in nature; and could be accessed via standard SQL operations using Amazon Athena. The following key aspects were covered as part of the AWS’s storage systems:
- Using Snaplogic’s homegrown Machine Learning (ML) component, further analytics were performed using predictive models on the datasets stored in S3. This scalable platform has been designed to extract meaningful customer insights and for other future needs.
- As part of Agilisium’s Best Practices, a Cloud enabled Data Governance Framework was established for the business data stored in the Amazon S3 enterprise data lakes. The necessary legal and regulatory requirements were fulfilled on par with the key Security and Compliance mechanisms.
- As an additional security enhancement, the sensitive business data was stored using selective role permissions, by following the “principle of least privilege”. This was offered by MFA (Multi Factor Authentication) enabled AWS Identity and Access Management, along with the requisite monitoring and audit tools such as Amazon CloudWatch and CloudTrail respectively.
- Transformation and Storage into Data Warehouse: From Amazon S3, the curated data was then transformed and stored into Snowflake Enterprise Data Warehouse. Specifically designed using star schema-based fact-dimensional modelling, Snowflake was able to perfectly store the transform customer data into logical patterns and arrangements.
- Visualization and Reporting: From Snowflake and Athena, the processed data was fed into Tableau Business Intelligence tool, to create canned (automatically generated) reports and dashboards. Snowflake, a cloud-built data warehouse as a service (DaaS) designed to handle massive big data workloads, could offer the requisite infra and software provisioning support, to integrate seamlessly with Tableau.
The following Key Value Additions were offered:
- Enhanced Customer 360 View: As a key value addition, the Customer Usage Executive Dashboards were created using Trend Charts as Graphical Representations. Using these charts, the Sales, Marketing Team and CXO heads at Snaplogic could obtain rich customer insights, by understanding the usage patterns of its Enterprise Customers based on fixed timelines, such as customer usage trends over the last 12 months and so on.
- From the portfolio of Top X and Bottom X customers (X specifying a count), the ranking of customers qualified as leads and prospects were also viewed graphically. From these detailed visual representations, Snaplogic’s active customers and users could be identified.
- Improved Customer Acquisitions: From this implemented solution, Snap logic could achieve about 80+% trial customer conversions. The conventional sales operations which were initially carried out reactively, are now done more proactively, creating scope for wider business opportunities.
- Effective Customer Tracking through Near-Real Time Updates: The manual processes, which initially took about 6-8 hours have now been replaced by automated efforts, achieving a reduced duration of about 30 min per data load. The implemented data re-engineering operations have helped achieve a significant performance improvement of about 94%, helping the marketing and sales teams at Snaplogic to obtain near real time customer updates, at a total time-lag of ONLY 3-4 hours.
- Enhanced Visibility and Better Customer Approach: With the improvised functionalities, Snaplogic can now better position itself geographically and demographically as an iPaas provider. It could also better understand sales trends, improve customer relationships (over 98% customer satisfaction) and drive future proof sales.
- Generation of end Report Views: The interactive customer usage dashboards can now provide instant sales analytics within seconds, to the market research experts at Snaplogic. With these features, Agilisium’s initiative as a customer-focused accelerator have been well realized at various levels of the hierarchy at Snaplogic, with additional efforts for enabling data driven decision making and better CEO focus.
Snaplogic was looking to build a centralized analytics platform to perform Sales & Product Usage Analytics. We wanted to work with an organization that has depth of experience in rolling out AWS- based Cloud Data Lake platforms. And, we found Agilisium to possess leadership and credibility with past implementations in this space to fulfill our vision.
Agilisium not only built the right platform to handle 1.2 PB scale of data with near real time updates, they also helped us bring about a cultural change towards becoming a true data-driven, information-centric organization.
Diby Malakar – VP Product Management, SnapLogic