Redefining Data Discovery: Lessons from a Research-Driven Journey

When Feedback Speaks Loudly

Over the past few years, our data consumers have struggled to find the right data in our data marketplace. An issue echoed in feedback and satisfaction scores. Our usual method, user focus groups, wasn’t moving the needle. So last year, we pivoted to a research-driven approach.

Embracing a Research-Driven Approach

To guide us on this new path, we partnered with Rabobank’s Chapter Design. Together, we invested significant time and effort into rethinking our strategy and defining the right steps to truly understand and address user needs.

Our journey began with a clear problem statement. We conducted in-depth interviews with subject matter experts to uncover the root causes of their frustrations. This was followed by several rounds of user testing to validate our findings and refine our understanding.

In this article, let’s explore together the key “aha” moments and lessons we learned along the way and insights that are shaping the future of data discovery for our users.

Discovering the real data consumer journey

Our first “aha” moment came when we explored the actual journey of a data consumer. Based on our initial discussions and assumptions, we believed the process was fairly linear: a user becomes aware of the data catalog, onboards, searches for data, and then requests access. Simple, right?

But reality told a different story.

After becoming aware of the catalog, users are typically introduced to the tool not through our formal onboarding, but by a colleague. Even more surprising was how data discovery happens. Not through the catalog itself initially, but through conversations. Data consumers often start by asking peers whether a specific data product exists and gathering insights informally.

The surprises didn’t stop there.

As users move forward, searching for data and requesting access, they maintain ongoing communication with data producers, both within and outside the catalog. This continuous dialogue plays a crucial role in their journey. Additionally, onboarding isn’t a one-time event. Instead, it unfolds gradually, with users learning about the tool and its processes step by step as they engage more deeply.

Hunting in the dark

Another major insight we uncovered relates directly to data discovery. When users know the exact name of a data product, our data marketplace performs well. Search is quick and effective. But when the name is unknown, the experience changes dramatically. Exploratory search becomes a frustrating, often fruitless task.

Several factors contribute to this challenge:

• Overwhelming Interface: The abundance of features and information can make navigation feel cluttered and confusing, especially for new users.
• Ineffective Search Strategies: Users often struggle to choose the right search terms or refine their queries, resulting in an overwhelming number of irrelevant results.
• Poor Metadata Quality: Titles and descriptions of data products are frequently unclear, filled with internal jargon or abbreviations, and often lack essential metadata. This makes it difficult for users to assess whether a data product is relevant to their needs.

These issues highlight a critical gap in the user experience: while the infrastructure may be in place, the usability and discoverability of data still need significant improvement.

Human contact is irreplaceable

One of the most consistent themes we’ve heard from users is that our data catalog is not used in isolation. Direct communication between data producers and consumers plays a vital role throughout the data access journey. In fact, this human interaction is often a prerequisite for successfully finding and using the right data.

Here’s how and when this communication typically happens:

• Before the search begins: Data consumers often reach out to colleagues or data producers for guidance on where to start or which data product might be relevant.
• During data discovery: As users explore potential data products, they frequently contact producers to clarify details, request additional context, or discuss how the data can be used effectively.
• After requesting access: Even once a request is submitted, communication continues. Users often follow up with producers to understand timelines, requirements, or next steps.

These interactions highlight a key insight: while tools and platforms are essential, they can’t replace the value of human connection in navigating complex data ecosystems.

Where do we stand today?

With the key insights in hand, we organized them into “How might we…” opportunity areas, each representing a distinct challenge and potential for innovation. These opportunity areas include:

• Improving data literacy for more effective search within the data marketplace
• Enhancing collaboration by enabling users to find data through colleagues
• Standardizing data product names and descriptions to boost discoverability
• Fostering trust in the access request process through increased transparency and consistency

This set the stage for the creative phase, where we brainstormed a wide range of solutions, from practical improvements to bold, unconventional ideas. After carefully narrowing down our ideas, we created a set of design concepts and shared them with users for input. This step was key in revealing which directions resonated most and offered the greatest potential impact.

The result of this collaborative, research-driven journey is a brand-new recommendation plan—now ready for implementation.

We’re excited about what’s ahead. These insights are not just shaping our roadmap—they’re reshaping how we think about data discovery, usability, and the human experience behind every search. And this is just the beginning.