Providing a Data Platform solution for a global leader in GCP industry to enable a holistic view of each customer in near real-time
Challenge
The customer’s objective was to overcome data silos and construct a unified Datalake that serves as a reliable and authoritative source of data for campaigns and planning. The desired solution needed to be cost-effective, secure, and dependable, while adhering to data stewardship standards. The client had high expectations for seamless integration with multiple third-party data vendors and the implementation of scalable, fully managed ETL solutions available through the Google Cloud Platform.
Our approach
To meet these objectives, we devised a comprehensive set of integration processes that guarantee efficient data ingestion, transformation, and storage. Our approach involved the creation and scheduling of more than 100 pipelines in Composer, leveraging the power of DataProc clusters, Google Cloud Storage buckets, and BigQuery. We seamlessly migrated historical data bundles using specialized Data Transfer tools. To ensure the utmost data security, we implemented Secret Manager, while also employing Logging to diligently monitor the processes.
The outcome
The seamless migration to the cloud facilitated the efficient processing of data on a massive scale, surpassing a petabyte. Over the course of 6 months, we accomplished the successful migration of roughly 1.5PB of historical data, restructured numerous ingestion processes, and orchestrated streamlined pipelines on the Google Cloud Platform. Consequently, data storage and computation expenses decreased by approximately 30%. The punctual and error-free delivery of data empowers a multitude of internal applications and significantly benefits more than 50 brands operating in over 100 markets.