How a “Tiny Team” Manages Unity in a Large Financial Enterprise

Our small, tiny team ended up close to Unity at scale. Not as “admins from a simple tool.” Not as the owners of Databricks workspace. And definitely not as the owners of everyone else’s datasets. No…

… more like what our team name hints at: the universe.

Invisible thinness.

If you wanted us to stop existing, the workspaces would still be there. The data products would still be there. The universe remains.

But the often-implicit relations between all those (work)spaces and other spaces, like the shared conventions, the common language and the patterns that make things predictable, become harder to see, harder to align on, and harder to explain.

(Also: we’re not doing this in a vacuum. Security, risk, and other platform teams are part of the same system. They keep us honest, and they keep the whole thing grounded.)

This post is the tale of that responsibility and why Unity is not just another run-of-the-mill financial platform. It’s what Unity actually means once internal customers, auditors, and risk teams are part of the story. How do we try to give teams freedom without allowing the platform to become an ungoverned sprawl? Why a managed PaaS can still hide operational limits, governance gaps, and failure modes that only become visible when something important breaks. And what we’d do differently if we could start over.

A quick contextual note: I joined this team officially about one and a half months ago, but I’ve been collaborating closely with them for about a year.

So, some of what follows are outsider observations.

And some of it is what I’m learning on the inside.

Why Running Unity in Finance Isn’t Just “Another Admin Job

On paper, Unity is “governance.” In practice, in finance, it’s closer to plumbing.

Invisible when it’s working; catastrophic when it isn’t.

Because the expectations are clear:

1. Every dataset has a classification.
2. Every access path needs control.
3. Every change needs an audit trail.
4. Every exception needs an owner.

And then there’s a tiny thing that gets lost in translation when people say “the Universe team”: We don’t actually own the datasets.

The teams do. They own the meaning, the quality, the definitions, and the consequences.

What we try to provide is consistency in the shared layer: patterns, standards, and reusable building blocks that make it easier for teams to meet those expectations without reinventing each workspace from top to bottom.

Now add the detail nobody puts on the architecture diagram.

The team responsible for this is small.

Small as in: the same people are designing the model, operating it, responding to incidents, answering questions, and writing standards, all while trying to keep the whole thing transparent. So, “running Unity” stops being an admin task. It becomes a matter of maintaining consistency, since inconsistency is how risk sneaks in. One workspace does it “their way.” Another team copies it.

Three months later, you can’t explain why two similar datasets have different policies, different owners, and different access patterns. And then you’re back where you started: governance via meetings and a tiresome back-and-forth of e-mails.

The only sustainable path is repeatability, patterns, and some fine automation. Plus a shared language that transcends turnover, growth, and vendor change.

What Unity Really Means Inside a Regulated Organization

People describe Unity like it’s a feature set. There are permissions, a catalog to store your data, and a place to click “Grant.” That’s not wrong; it’s just incomplete.

Inside a regulated organization, Unity becomes the place where governance stops being an abstract slide and starts becoming something teams can apply.

And when things are applied consistently, evidence can be established.

It’s when teams (and the wider ecosystem around them) can express and rely on things like:

1. Governance: who can read what, who can write what, and under which conditions.
2. Lineage and traceability: being able to explain how a table was produced and what it feeds downstream.
3. Data management: ownership, sensitivity, lifecycle, and the link between policy and what’s actually implemented.
4. Consistency across Databricks workspaces: the same access model, the same naming conventions, the same semantics—at scale.

That last one is the quiet, daily win. Or, more honestly: it’s the quiet, daily goal because enterprises don’t have a single environment to manage.

They have a landscape. A landscape full of workspaces with multiple teams and multiple ways of working.

Without a common governance layer, every workspace becomes its own island with its own rules.

Unity is how we try to keep the islands from drifting apart.

We’re not fully there yet. But we’re headed straight towards consistency and something that “we” can explain (and want to defend).

How “We” Provide Freedom Without Losing Control

This is the point at which the story most often gets misunderstood and twisted. Governance teams don’t wake up one morning wanting to slow people down (even though that’s what most people think). Most of us get into this because we like enabling at a fast pace. But enabling without boundaries isn’t freedom.

It’s entropy.

At the same time, we’re not a police squad “enforcing governance.” Rather, we see what we maintain as connective tissue: the shared layer that makes the platform attractive and repeatable across teams.

Because the fastest way to lose autonomy is to create a high-profile incident that forces everything to become centralized, manual, and restrictive.

Our consumers are everyone who builds on the platform:

1. Data engineers and analytics engineers who ship pipelines.
2. Data scientists who explore, train, and iterate.
3. BI teams who curate data products.
4. Platform teams who run shared infrastructure.

They want autonomy, and they should have it.

But autonomy scales faster than governance unless governance is designed to feel like the default and not as a separate process.

So, we put our energy into a few durable levers:

1. Opinionated standards: naming conventions, workspace patterns, and “golden paths” that reduce decision fatigue.
2. Policy-as-code where possible: rules that are reviewable, repeatable, and testable.
3. Safe defaults and shared templates: designs that keep mistakes small and contained when someone tries something new.
4. Standards above our own team: partnering with other platform teams across the bank, so that “a standard” means the same thing everywhere.

And then there’s the thing small teams learn the hard way: the power of neat documentation.

In a tiny team, documentation isn’t an optional extra. It’s the fundamental system that remembers what our minds can’t retain forever. Having proper documentation is how you scale:

1. Onboarding without relying on institutional memory.
2. Breaking changes without unleashing panic.
3. Evidence-based audits instead of handwaving.
4. Self-service that doesn’t turn your product into a ticket queue.

We follow a documentation framework called Diátaxis.

It keeps us honest about what the reader needs in the moment: a tutorial to learn, a how-to guide, a reference or an explanation.

Not just “click here.”

But “this is why the pattern exists.”

The Vendor Problem

Let’s be precise once again about roles. We don’t manage every Databricks workspace day-to-day. That’s the job of the product and platform teams who own those workspaces.

What we try to facilitate is cross-workspace consistency: shared conventions, shared approaches, and the invisible relationships that make a multi-workspace reality feel like one platform.

And that’s where the vendor problem shows up.

Managed PaaS is the deal, right? Our organization runs mostly on Azure Databricks because we don’t want to run Spark clusters like it’s 2016. But the trade-off can be a real issue. When the vendor ships, our platform changes. Sometimes, it’s subtle. But sometimes, it’s by default. And in a regulated environment, “default” is not the standard.

New features can appear without you planning for them. Some get enabled by default. Workspace-level options change. New capabilities show up around model serving.

And suddenly, you’re back in the same loop: understanding impact, aligning with security and risk, updating guidance, and making sure teams don’t accidentally adopt patterns they that either “we” or “they” can’t defend later.

A real example: AI capabilities like Genie appearing that weren’t part of the original security model. It doesn’t even matter whether the feature is “good.”

The questions are always the same:

1. Which data could flow into it?
2. Where does that data go, and under what terms?
3. What is logged, and what is retained?
4. Can we disable it, scope it, or gate it in each workspace?
5. What evidence can we show that the configuration matches policy today, not three releases ago?

So, we try to be as proactive as possible: tracking upcoming releases, testing behaviors, verifying defaults, updating guidance, updating documentation, and communicating clearly. That’s not pessimism. It’s how you keep a fast-moving PaaS usable in a regulated enterprise.

If We Could Start Over

If I’d been in this position earlier, I would’ve pushed for self-service sooner. Not because I think tickets are evil. More because ticket-driven approaches quietly train people to work around the platform team. I could see the pressure building up long before I joined the team and started taking responsibility.

People want an answer sooner rather than later, but the queue leaves them waiting until next week. That’s when they start improvising. And improvising is where inconsistency starts to grow. The simpler, better goal would be:. Make compliance path the easiest path.

If I could start over, I’d invest sooner in the things that let a tiny team scale without becoming a bottleneck:

1. Automation for repeatable requests: workspace onboarding and defining policy templates that consumers could use.
2. A self-service API: internal teams request what they need through a controlled interface that enforces validation, approvals, and audit trails.
3. An understandable platform: show what exists, who owns it, and the expected usage patterns.
4. Enablement as a product: examples, reference architectures, and documentation that reduces “mystery admin work.”

To me, that shift changes the role of a governance team. From humans operating a gate, to humans designing a system where governance is embedded in the workflow. In a regulated enterprise, that’s the only model that scales.

Conclusion

Unity at scale is not just a catalog, an external location, or a storage credential. It’s an operating model.

It’s a decision about what you standardize, what you automate, and what you’re willing to let vary across teams.

The goal isn’t to control people.

It’s to keep the shared layer coherent, so that the compliant path is easy to follow, and the whole thing remains transparent after the fact.

If you’re carrying a similar responsibility, I’m curious to know: What did you automate first and what did you learn the hard way?