Client Background
The client is among leading insurance companies in Australia and Europe. They serve millions of customers across more than 15 countries. Due to a Non-Disclosure Agreement (NDA), further details regarding the specific company cannot be disclosed.
Challenges
The client’s main challenge in building a greenfield DWH solution was the high data complexity caused by four inconsistent data warehouses with different structures and levels of standardization. This situation led to a lengthy loading process that required optimization to ensure efficient data ingestion. Additionally, data quality posed a significant issue, necessitating robust cleansing and validation mechanisms to establish a reliable and consistent analytical environment.
Our Solution
We improved the loading process with detailed data and business analysis combined with a metadata-driven approach. This allowed us to move from monthly data loads to daily, and later even twice per day. For data quality, we couldn’t find a good enough tool on the market, so we built our own top-tier automated testing tool (CAT). The whole solution is fully automated with DevOps Pipelines, runs on three environments, and is completely version-controlled in GIT, including PBI reports in TMDL. Of course, everything is thoroughly tested.
At a Glance
We built a high-performance DWH, consolidating four inconsistent warehouses into one in nine months, achieving 27% cost savings. Data ingestion increased 60x, moving from monthly to twice-daily refreshes, processing 1.5 billion records per hour.
The migration was seamless for 3 million clients, and we ensured top data quality with our automated testing tool (CAT). The solution is fully automated with DevOps Pipelines, version-controlled in Git, and optimized for business impact.
