Data contracts
A Socratic walk-through of data contracts — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a producer promising not to change a field make a whole platform faster to build on?
A promise seems a weak thing to build a platform on. A producing team says: this field will keep this name, this type, this meaning, and if that must change you will get notice. No new technology, no faster pipeline — just an undertaking.
And yet teams that adopt this report that everything downstream gets faster to build. That is worth being suspicious of. A promise adds a constraint — it takes options away from the producer. How does removing someone's freedom make a whole system more productive?
Reasoning it through
REASONING #Start with what a consumer does in the absence of any promise. They build a pipeline over a table they do not control, whose schema can change tonight without warning. What do they build defensively?
They add checks for columns disappearing, coercion logic in case a type shifts, reconciliation because they cannot trust a count. And critically, they build all of that speculatively — guarding against changes that will probably never happen, because they have no way to know which ones might.
Now ask what that defensive work actually is. It is an attempt to infer a guarantee from observation: the consumer concludes, from the fact that a field has been an integer for six months, that it is an integer. That inference is not knowledge; it is a bet. And the producer never agreed to any of it and does not know it happened — so they refactor a column entirely reasonably, and something breaks in a team they have never spoken to.
Notice the asymmetry. The producer cannot know what is safe to change; the consumer cannot know what is safe to rely on. Both are guessing, and the guessing is expensive on both sides.
What does a contract change? Not the data. It changes the distribution of knowledge. The producer now knows exactly which fields are load-bearing, so they can refactor everything else freely — the contract gives the producer information they did not have, which is why this is not simply a tax on them. The consumer knows which properties are guaranteed, so the speculative defensive code stops being written.
But a stated promise is only as good as its enforcement, and a declaration nobody checks decays. So what makes a data contract more than a wiki page? Two things. It is machine-readable — a schema, plus assertions about nullability, ranges, uniqueness and freshness. And it is checked where a change would occur, in the producer's own pipeline: a violating schema fails the build rather than failing a consumer's job at three in the morning. That is the shift the term points at — moving detection from downstream runtime to upstream build time.
Then what about changes that genuinely must happen? A contract that forbids all change is a freeze, and would be abandoned within a quarter. So the useful ones govern change rather than prohibit it, using compatibility rules borrowed from schema registries: adding an optional field is backward-compatible and needs no coordination; removing a field or narrowing a type is breaking, and requires a version, a notice period, and a window in which both versions are served. The promise is not "this will never change" — it is "you will never be surprised".
The analogy
THE ANALOGY #Think of a supplier of machine parts who publishes a specification: this bolt is M8, this thread pitch, this tensile grade, and any change comes with twelve months' notice and a new part number. The specification does not make the bolts better. It lets every workshop downstream design a jig, cut a housing, and order in bulk without keeping a caliper on the bench. And it works in the supplier's favour too: because only the published dimensions are committed, they can change the alloy supplier, the coating and the factory entirely, knowing precisely which properties the world depends on.
a bolt's specification describes physical dimensions that a workshop could measure for itself if it had to, whereas a data contract must also carry meaning — that this timestamp is in UTC, that this status code excludes test orders — and semantics cannot be recovered by inspecting the data at all.
Clarifying the model
THE MODEL #Three refinements.
The first is that the semantic clauses do the quiet work. A schema check catches a type change; it cannot catch a producer who keeps the field name and type and changes what the field counts. That failure is silent, survives every automated test, and is the most damaging kind. Contracts address it only partly — by writing the definition down against a named owner, so at least there is something to violate and someone to ask.
The second is that a contract is an interface, and interfaces are only worth having where there is a real boundary. Imposing them on every table inside one team's own pipeline adds ceremony and buys nothing, because producer and consumer are the same people. The value scales with the distance between teams and the number of consumers — which is why this pairs naturally with domain ownership: a data product is a dataset with a contract on it.
The third is the honest cost. A contract makes some changes expensive on purpose: the producer who wants to drop a column must find the consumers, agree a timeline, and possibly run two versions. That friction is not a side effect to be minimised away, it is the mechanism — it moves the cost from an unpredictable breakage in someone else's system to a predictable negotiation in the producer's own planning. Teams that adopt contracts and then grant themselves routine exemptions have kept the paperwork and discarded the benefit.
Worth saying that the term is fairly new — common use dates from around 2022 — and practice is still unsettled. The underlying idea is much older: the same argument for published interfaces that software engineering settled decades ago, arriving late here because data was long treated as exhaust rather than as a product with users.
A picture of it
THE PICTURE #How to readStart at the parallelogram, and note that every branch below it happens in the producer's pipeline, before anything reaches a consumer — that relocation is the whole point. The first diamond sorts changes by kind rather than size: an added optional field needs no conversation, which is what stops the contract becoming a freeze. The red terminal is a success, not a failure — a build stopping is the contract working. The third branch is the honest weakness: a change that keeps the shape and alters the meaning passes every check, and the only defence is the store at the bottom, a written definition with a person's name on it.
What became clearer
WHAT CLEARED #The promise is productive because of what it lets everyone stop doing. Without it, both sides guess at each other — consumers writing speculative defences, producers avoiding changes out of vague fear or making them in blind confidence. A contract converts that guessing into shared knowledge: the producer learns exactly what is load-bearing and is free everywhere else, the consumer learns what may be relied upon and stops hedging, and a genuine breaking change moves from an unplanned outage to a planned negotiation.
Where to go next
ONWARD #- How schema registries implement backward, forward and full compatibility, and which to choose.
- What belongs in a contract beyond schema: freshness, volume expectations, semantics, escalation.
- Whether a contract without a named owner is worth anything at all.
Key terms
TERMS #| Term | What it means |
|---|---|
| Data contract | an explicit, machine-checkable agreement between a data producer and its consumers covering schema, semantics and service expectations. |
| Schema evolution | the rules governing how a schema may change over time without breaking existing readers or writers. |
| Backward compatibility | a change a consumer written against the old schema can still read, such as adding an optional field. |
| Breaking change | a change that invalidates existing consumers, requiring a version, a notice period, or a dual-serving window. |
| Semantic drift | a field keeping its name and type while its meaning changes, which schema checks cannot detect. |
Every term the collection defines is gathered in the glossary.