Shipping AI Features in Banking: Why Evals Are Your Only Defense Against AI Confident Hallucinations

That definition might sound straightforward. What is less obvious is how rarely it happens in practice and how clearly the people building AI at the largest scale have been saying so in public. When the CPOs of the two most influential AI labs in the world, the co-founder of OpenAI, and the CEO of Y Combinator all point to the same skill independently, it is worth paying attention to what they are saying.

Kevin Weil, CPO of OpenAI: writing evals is becoming a core skill for product managers a critical part of making a good product with AI.

Mike Krieger, CPO of Anthropic: if there is one thing he could teach people, writing evals would be it.

Greg Brockman, co-founder of OpenAI: evals are surprisingly often all you need.

Garry Tan, CEO of Y Combinator: evals are the real moat for AI companies. The best AI product leaders run them ruthlessly behind the scenes even when the public story is about taste and craft.

This is not a niche engineering opinion. It is an emerging consensus at the frontier and the investment data reflects it. Gartner predicts that by 2028, LLM observability investments will reach 50% of GenAI deployments up from just 15% today. That shift is not happening because the tooling got better. It is happening because the cost of shipping AI without knowing whether it works has become too visible to ignore.

In the previous article in this series, I wrote about how data problems kill AI initiatives long before the model is ever trained. That was the first gap the foundation beneath the feature. This article is about the second gap what happens after the feature is live and nobody has defined what good looks like.

What an Eval Is and What It Is Not?

Since the word will appear throughout this article, it deserves more than a one-line definition.

An eval gives you a number or a set of numbers that tells you whether your AI feature is improving, regressing, or drifting in ways nobody noticed. You give the AI representative inputs, score the outputs across specific dimensions you defined, and track how that score changes over time as the product evolves.

Two things it is not, and the distinction carries real weight.

It is not a public benchmark. A model scoring at the top of a leaderboard tells you nothing about whether it handles your specific data, your specific users, your specific edge cases. Benchmark performance and real-world quality are two entirely different measurements.

It is not production monitoring either. Monitoring tells you when the system crashes or returns an error. Evals tell you when the system is wrong and wrong answers never return errors. They arrive looking exactly like correct ones, delivered with complete confidence.

That second point is where confident hallucinations live. An AI feature can respond in under a second, in fluent professional language, and still give a customer a factually incorrect answer about something that matters. No alert fires. No dashboard changes color. The system looks healthy. The customer received the wrong answer stated with certainty.

That is not a model failure. It is a measurement failure. And measurement defining what good looks like and tracking it continuously is a product responsibility, not an engineering one.

KPIs Tell You the Bet Was Worth Making. Evals Tell You the Feature Is Worth Trusting

Most AI product reviews feel reassuring. The adoption numbers are up. Support volume is falling. There is a thumbs-up widget next to the AI response a small fraction of users interact with it and the team calls this feedback. The model passed QA before launch. It scores well on a benchmark.

These signals are real. None of them are evals.

What they measure is engagement and system behavior whether people showed up and whether the system ran. What they do not measure is whether the AI's outputs are correct, consistent, faithful to the source data, or appropriate for the specific customer's situation. That gap is where AI features fail quietly. Not dramatically. Not in a way that triggers an incident report. Gradually, invisibly, after the launch celebration has ended.

The distinction between KPIs and evals is worth being precise about.

KPIs operate at the initiative level. Did this investment deliver business value? Did the customer experience improve? Did cost per interaction fall? These belong in quarterly reviews and board presentations. They tell you whether the bet was worth making and they arrive months after the feature shipped.

Evals operate at the feature level, in real time. Is this specific output correct? Is it faithful to the source data? Does it behave consistently when the same question arrives phrased differently? These tell you whether the feature is actually working before a business outcome can even be measured.

Xavi Amatriain, Chief AI Officer at Expedia and formerly VP of AI at Google across the platforms powering Gemini and Search, made the product ownership case directly at VB Transform 2026: "The new PRD are the evals. You encode what you want the product to do through your evals. All your thinking is going to go into the evals."

This is not a technical reframing. It is a product ownership argument. The person who defines what the AI should do what a good answer looks like, what constitutes a failure is the product manager. If that definition is never written down in a form that can be measured, the feature ships without a contract between the team and the user. Changing the prompt, upgrading the model, adjusting the retrieval without evals, no one can tell whether any of those changes helped or hurt. Improvement becomes invisible. So does regression.

How to Actually Build Evals

Understanding what evals are leads naturally to the next question: how do you actually build them? The answer is more accessible than it sounds. There is a natural progression, and the value begins on day one.

The first level is human review. Someone on the team reads a sample of real AI outputs and scores them against criteria written in plain language. Not the clean outputs from a test environment the actual ones from real users, with the ambiguous phrasings and incomplete inputs that controlled settings never surface. This step matters most not because it scales, but because it forces a precise answer to the question teams tend to skip: what does good actually look like for this specific feature, for these specific users? Every automated eval that follows is built on the definition produced here.

The second level is rule-based checks. Automated logic applied at scale. Does the response include a required disclaimer? Is the length within acceptable bounds? Does it avoid outputs that violate a documented policy? These checks run fast and catch the obvious failures. They do not say anything about whether the response is correct, coherent, or appropriate for the customer's actual situation.

The third level is LLM-as-judge. A separate AI model evaluates the outputs of your feature against the criteria defined at level one now running continuously and at scale. The judge model reads each output, applies the definition of good the product team wrote, and returns a structured score. Because the criteria are in natural language, the product manager writes and owns them directly without depending on engineering to translate requirements into code. The technique scales like having a domain expert review every interaction, at a fraction of the cost. In a financial services context, that might mean assessing whether a response is grounded in the source document, whether it avoids stating specific figures without attribution, and whether it correctly identifies when a human advisor is needed rather than an automated answer three criteria that together define the difference between an AI feature that builds trust and one that quietly erodes it.

Who Owns Quality? The Question to Ask Before You Build

Knowing how to build evals is one thing. Getting everyone aligned on who needs to participate in them is another and it is where many AI initiatives quietly stall after a promising start.

Here is what tends to happen. The initiative gets approved. The pitch was compelling faster responses, better scale, lower cost. Development begins. Then, at some point, the product team explains that before this feature can be trusted at scale, the people closest to the customer will need to dedicate time to reviewing outputs, defining what good looks like, and setting quality thresholds. Suddenly the energy shifts. Priorities have changed. The eval work starts to feel like a tax on a project that was supposed to make things easier.

This is not a failure of commitment. It is a failure of expectation alignment and it starts before the first line of code is written.

The excitement about what AI can deliver and the investment required to evaluate it properly need to be established together, at the beginning. The teams closest to the customer the people who understand the use case, the edge cases, the moments where a wrong answer causes real harm are not optional participants in the eval process. They are the ones who define what good looks like. That definition cannot come from engineering alone. It cannot come from the product team alone. It has to come from the people who know the customer.

There is an additional layer worth naming. In many cases, the process the AI is replacing was never properly measured either. There were no quality thresholds for how accurate the human response was, no systematic review. So when evals arrive as a new requirement, they feel like something AI introduced rather than what they actually are: the measurement discipline that should have existed all along. The AI did not create the need for quality standards. It just made the absence of them impossible to ignore.

The most important conversation to have at the start of any AI initiative is not which model to use or which framework to build on. It is who owns the definition of quality for this feature and whether they understand that ownership before the team ships anything. These are the conversations we are navigating, as we experiment and learn what good actually looks like before anything reaches scale.

Conclusion: The Question That Separates Shipping From Knowing

Everything in this article traces back to a question that is surprisingly simple to ask and difficult to answer.

Pick an AI feature your team has shipped and ask three things. What score did it receive on its last eval run? What does that score actually measure? And what threshold does it need to reach before your team would be confident scaling it to more users?

If all three have clear answers the number, the definition, and the threshold your team is operating with a level of AI product discipline that still genuinely differentiates. If they do not, the gap is not in the tooling. It is in who owns quality for that feature, and what owning it actually means in practice.

Data determines whether an AI initiative can get off the ground. Evals determine whether it can be trusted once it does. Closing that gap not just shipping the feature, but knowing whether it works is what it means to build AI products rather than simply deploy them.