The Data Tango: Harmonizing producers and consumers
We do have a lot of data in the bank. However, making reliable and trusted data available for all different business usecases in a governed way remains a challenge. To tackle this challenge, we have created a Data Platform that enables producers to publish their data and enables consumers to discover this data and apply it to their specific use-cases in a well-governed manner. In order to ensure success, producers and consumers should adopt a harmonized way of working. Let’s explore the pivotal roles that producers and consumers play for a successful Data Platform and how they contribute to the availability, reliability and usability of data on the platform.
- Hanna Mironchyk Lead Product Manager Data Management, GDAP Area Data Platform
Solving Data Availability Issues
Searching for data might be like searching for a dance partner. One might assume that our most obvious challenge is no dance partner or data available.
Surely, there will be no issue with finding data that we already own as a company? To our surprise while starting up with the Data Platform in the bank we discovered something else. Some users shared that they missed a holistic overview of all available data and who to contact with questions.
This is where the data catalog of the Data Platform comes into play. It provides this holistic overview, connects producers and consumers, and serves as the core of the Data Platform. It is also a place where we can derive information about what data was searched for and not found, which provides a great basis for gaining insight into what consumers are looking for.
The data catalog contains information on the bank domains and source systems from which data is available on the Data Platform, what is this data, which attributes are there and what do they mean, who do you talk to if you are not sure you can use this data. This marks the first interaction between producers and consumers. Think of it as a search platform for a dance partner for your data tango.
Ensuring Data Reliability and Trust
Just like an any dance, the tango between producers and consumers can only be successful if the partners are in harmony. A common concern among data consumers is, ‘Can I trust this data? Is it reliable?’
What can help to grow trust between producers and consumers?
Firstly, when producers and consumers use the Data Platform, they benefit from the standards and guidelines for data management and data preparation. Moreover, it is not only about setting the bar for reliable data with standards, but about providing automated processes, workflows, frameworks. The following automated processes and workflows on the Data Platform ensure that data management requirements are followed by design:
- Automated data schema scanning
- Clear ownership of data
- Governed Data Access
- Data Transfer Controls Framework
- Data Quality Framework and Engine, etc.
These standards, automated processes and workflows help not only consumers but also producers who can focus on building their data deliveries in an efficient manner. They also ensure that data access is fully-governed and data producers approval is always required to grant access.
Secondly, the Data Platform provides insights into producers’ compliance with these standards. The transparent insights give consumers input for their decision to start using certain data for their use-case or not. An experienced dancer might still be willing to have a dance with a novice and not be disappointed, if there is transparency about the level of proficiency and the realistic expectations are set in advance.
Governing data means understanding your data."
Enhancing Data Usability
Just as tango without the music is a collection of strange movements, the data without usability and understanding is barely anything. Data makes sense when you know how to put it together.
The Data Platform provides insights about data on the platform:
- who and for what purpose combines data on the platform,
- what sets of data are created,
- what business process it supports.
This helps in understanding business goals supported by the data. However, as business rules and logic become increasingly complex, maintaining them becomes challenging. This highlights the importance of another data governance aspects such as data lineage and definitions.
A strong feedback loop is crucial for data usability. On the Data Platform, data consumers can provide feedback to data producers by rating their data and leaving comments and suggestions for data sets (i.e., social metadata).
In conclusion, in today’s world, available, reliable and usable data is essential to obtain strategic advantage and foster innovation. Our experience shows that the cooperation between data producers and data consumers built on the solid foundation of the Data Platform is crucial for achieving this. By recognizing those roles and responsibilities and actively participating in the data ecosystem, we can collectively drive better business outcomes and innovation. Because as we know, it takes two to tango.