What is Data mesh?
Data is available in a humongous quantity, but to manage and use the data to make sound decisions requires a proper structure, such that data can be utilized to its optimum. Companies manage their data inadequately for the way the information is used. The data organizations use to inform their decision-making has remained centralized, stuck in a traditional monolithic data warehouse architectures. Many organizations have invested in a central data lake and a data team with the expectation to drive their business based on data. However, after a few initial quick wins, they notice that the central data team often becomes a bottleneck. The team cannot handle all the analytical questions of management and product owners quickly enough. This is a massive problem because making timely data-driven decisions is crucial to stay competitive.
Companies need to decentralize data and make it domain oriented, which can be done through Data mesh. Nextdata founder and Chief Executive Zhamak Dehghani has coined the term Data Mesh. Domain ownership principles require the domain team to take ownership of the data. According to this principle, analytical data should be aggregated around domains tailored to the limited context of the system, as well as team boundaries. A domain-driven distributed architecture shifts ownership of analytical and operational data from central data teams to domain teams. Just as microservices architectures combine lightweight services to provide functionality for business or consumer applications, data mesh uses functional domains to set parameters around data, treat it as a product, and make it accessible to users throughout the organization. In this way, data mesh enables more flexible data integration and interoperable capabilities, allowing users to instantly use data from multiple domains for business analysis, data science experiments, and more. The domain team is responsible for satisfying the needs of other domains by providing high-quality data. Basically, domain data should be treated as any other public API.
A dedicated data platform team provides domain-agnostic capabilities, tools, and systems to build, run, and maintain interoperable data products across domains. The data platform team uses its platform to enable domain teams to seamlessly consume and create data products. Federated governance principles enable interoperability of all data products through standardization promoted by governance groups across data networks. The main goal of federated governance is to create a data ecosystem while complying with organizational rules and industry regulations.
Benefits of Data Mesh:
Data democratization:
A data mesh architecture facilitates self-service applications from multiple data sources and extends access to data beyond technical resources such as data scientists, data engineers, and developers.
Cost efficiencies:
This distributed architecture moves away from batch data processing and instead, it promotes the adoption of cloud data platforms and streaming pipelines to collect data in real-time. Cloud storage provides an additional cost advantage by allowing data teams to spin up large clusters as needed, paying only for the storage specified.
Less technical debt:
A centralized data infrastructure creates more technical debt due to the complexity and collaboration required to maintain the system. Accumulation of data in the repository also slows down the entire system. By distributing data pipelines by domain owner, data teams can better serve the needs of data consumers and reduce the technical load on storage systems. It can also provide an API to interface with, improving access to data and reducing the overall volume of individual requests.
Interoperability:
Under a data mesh model, data owners agree on how to standardize domain-agnostic data fields upfront, which facilitates interoperability. Consistency across domains enables data consumers to interface with APIs more easily and develop applications to serve their business needs more appropriately.
Security and compliance:
: A data mesh architecture helps enforce data standards for domain-agnostic data and access controls for sensitive data, thus facilitating stronger governance practices