Embrace the power of data
In today’s data-driven world, organizations increasingly recognize the immense value of data. They seek innovative ways to leverage it for competitive advantage and innovation. As a result, the concept of “data as a product” has emerged.
- Anyesh SrivastavaSenior Solution Architect
In this article, when we write about “data products”, we refer to the conceptual framework articulated in the book “Data Mesh”. This concept entails adhering to superior practices and standards in relation to our data systems. Consequently, all references to “data products” throughout this document should be understood in this particular context.
The remarkable data evolution
At Rabobank, we are actively embracing transformative change toward becoming a data driven enterprise. As an employee and a passionate advocate for the transformative potential of data, I have the privilege of witnessing first-hand the remarkable evolution of data. This transformation has been particularly evident in the Area of Data and Factory Services within the Data and Analytics Tribe. The integration of data-driven practices and the recognition of data as a valuable asset have reshaped the bank’s operations and positioned it at the forefront of innovation in the financial industry.
I am proud to be a part of this journey and eager to share my experiences and insights through this blog, inspiring others to embrace the power of data and unlock its transformative potential in their organizations. Through this blog, my goal is to raise awareness among readers about the concept of “data as a product.” Additionally, I strive to provide guidance on the terminology related to data as a product, data products, and the architectural considerations that play a critical role in transforming data assets into valuable offerings. By offering this guidance, I hope to equip readers with the necessary knowledge and understanding to effectively navigate the realm of data products, enabling them to harness the full potential of their data assets and drive tangible value for their organizations.
The fundamental concept of data-as-a-product originates from the insights shared by Zhamak Dehghani in her book “Data Mesh: Delivering Data-Driven Value at Scale” and Piethein Strengholt in his book “Data Management at Scale.”
As I navigate the process of implementing data as a product within an organization, I find it beneficial to describe data as a product by consistently applying product management principles throughout the data creation process
What is a data product?
According to DJ Patil, author of “Data Jujitsu: The Art of Turning Data into Product” (2012), a data product is defined as “a product that facilitates an end goal through the use of data.”
In the context of data as a product, a data product is an offering that leverages data to provide valuable insights, drive informed decision-making, and deliver measurable business benefits. It is designed to address challenges, fulfil user needs, and empower users to extract meaningful information from data through features, visualizations, and self-service capabilities.
It is important to note that the terms “data as a product” and “data product” are closely interconnected yet carry distinct definitions. “Data as a product” holds the broader concept of regarding data as a valuable offering, emphasizing its transformation into a strategic asset. On the other hand, a “data product” specifically denotes a particular data asset that is purposefully developed to achieve objectives and deliver tangible value example, Dashboards for financial analytics. Data products represent the concrete results derived from implementing the concept of data as a product within an organization.
Guiding Principles for data as a product
So far, in this article, we have explored the concepts of data as a product and the data product. However, the journey doesn’t end there. It is crucial for us to delve deeper into the guiding principles and architectural aspects that encompass the entire lifecycle of these data products. To start, I choose DATSIS as the guiding principles for data product architecture because it serves as an ideal starting point for the journey of embracing data as a product. DATSIS stands out by specifically addressing the critical needs of data assets that transform them into valuable data products.
The accompanying picture offers a broad glimpse of the DATSIS acronym, and due to conciseness of this article, I opts to refrains from providing detailed definitions for each component.
Architecture based on DATSIS principles
The architectural concept of treating data as a product involves perceiving data as a consumable commodity within a business context. Not only should these data products be user-friendly and accompanied by metrics and metadata, but the underlying code and infrastructure used to generate them should also be accessible.
According to Datamesh architecture, data products serves as the building blocks of a Datamesh Architecture. These products are like small units of architecture that can be independently deployed and managed, referred to as Data Quantum. Within the architecture of these data products, domain-oriented data is held, along with the code responsible for necessary data transformations. Furthermore, the data and the policies governing it are shared within this architecture.
With this definition in mind, let us now delve into the high-level architecture of a data product and align it with the guiding principles mentioned earlier.
The architecture of a data product encompasses the following components:
- Data Transformation Pipelines framework for data transformation and preparation. These pipelines incorporate controls such as data observability, data reliability, and data quality, contributing to the effectiveness of the data products.
- Data Discovery Framework simplifies data browsing interactively, eliminating the need for specific permissions. It provides additional descriptions and access to data lineage, enhancing the data exploration experience.
- Control Plane Framework allows access to a REST API, enabling the management of onboarding data and data store(s). This framework offers control and flexibility over handling data within the product architecture.
- An interoperable Framework addresses consumption and ingestion needs, ensuring compatibility and seamless integration with other systems and data sources. This framework facilitates efficient data consumption and ingestion processes within the data product ecosystem.
Conclusion
By understanding the concepts and embracing the recommended architectural principles, organizations can effectively harness the potential of their data assets, drive innovation, and generate significant value. It’s crucial for organizations to approach data as a product with careful consideration, ensuring that data products are well-defined, easily accessible, and aligned with the needs of the business and customers.
As the landscape of data-driven decision-making evolves, adopting a comprehensive approach to data as a product becomes increasingly essential. By adhering to best practices and maximizing the utilization of their data assets, organizations can unlock the true potential of their data and drive growth in an increasingly data-centric world.
To conclude, I hope you gained valuable insights and guidance on the terminology associated with data as a product and data products, as well as the architecture responsible for transforming data assets into valuable data products. We explore the concept of data as a product and its significance in maximizing the value derived from data assets. For more information, please feel free to contact me.