“Since we are using data platform based on Data Mesh architecture designed and built by Transatlantix/Databricks we are able to efficiently share data and perform analytics across our businesses while the data is owned by businesses it is governed and secured centrally”

Chief Technology Officer - Leading professional services

Case Study:

World’s leading professional services firm in the areas of risk, strategy and people. With more than 85,000 employees and clients in 130 countries, annual revenue of over $20 billion. They help corporate and public sector leaders navigate an increasingly dynamic environment through four separate business units by providing professional services to Automotive, Insurance, Health care and financial institutions.

Problem Statement:

The current approach to conducting data analytics is through using diverse technologies and consolidating data from disparate sources across multiple departments within four enterprises with limited governance measures.

The objective is to develop an intelligent, integrated data platform solution that guarantees secure and democratized access to data for both internal stakeholders and external clients.

The primary challenge areas include addressing multi-tenancy issues, establishing a comprehensive data catalog, streamlining the onboarding process for applications, simplifying tool usage, ensuring platform reliability, and enabling multi-cloud capabilities.

Transatlantix Solution:

Transatlantix collaborated closely with the client's Technology team to facilitate this crucial architecture transformation.

The decision was made to adopt a *Data Mesh architecture leveraging Databricks as a pivotal foundation. This approach is essential for scaling the platform across diverse applications and business units while ensuring multi-cloud compatibility. Its primary objective is to manage, govern, and aggregate data from multiple business units, thereby forming a cohesive group-level perspective for analytics purposes

* Data mesh is a paradigm that describes a set of principles and logical architecture for scaling data analytics platforms https://martinfowler.com/articles/data-mesh-principles.html

Transatlantix solution of Data Mesh and Databricks Lakehouse:

Addressing the 4 principles:

 1. Domain-oriented data ownership:

Distributed architecture where domain teams, the data producers, have autonomy over their data and analytics workloads:

• Open flexible architecture for domains to collaborate within Databricks workspaces

• Distributed ownership of data assets and pipelines

2. Data as a product:

Applying product thinking to data, to make quality data easily accessible to data consumers via standardised interfaces:

•  Open standards and formats for FAIR data

• ACID guarantees, versions, and audits with Delta Lake

• Secure access beyond borders with Delta Sharing

3.  Self-service infrastructure platform:

Domain-agnostic approach to building, executing, and maintaining interoperable data products through common tooling:

• Managed orchestration with Databricks Workflows and Delta Live Tables

• Auto-scaling/tuning

• ‘Infrastructure as code’ (Terraform)

4.  Federated computational governance:

Creating an ecosystem that adheres to global rules through automatic execution of decisions by the platform:

• Discovery, access and lineage with Unity Catalog

• Global policy templates
  for access to data and compute resources

 Data Mesh Harmonized Topology:

Key Accomplishments / Outcomes:

1. Data sharing and accessibility across nodes without duplication.

2. Orchestration of pipelines leveraging cloud agnostic framework (e.g. Databricks workflows, Azure Data factory)

3. Data governance across hub and nodes (Unity Catalog)

4. Integrations (Qlik Sense, Dremio)

For more detailed information and guidance on Unity Catalog, Data Mesh, Databricks Lakehouse in Databricks please contact us here.