Error budgets

Why does deciding in advance how much failure is acceptable make a team ship faster and break less?

Two groups in the same company want opposite things, and both are right. The people who build features want to ship; every release is value delivered. The people who keep the service up want fewer changes; every release is a new way to break. Neither position is unreasonable, and the argument between them has no natural resolution — it recurs at every planning meeting and is settled by whoever is more senior or more recently burned.

Now consider a strange proposal: write down, in advance, exactly how much failure the service is permitted to have. Not zero. A specific quantity of badness, agreed before anyone knows where it will be spent.

Why would naming a permitted amount of failure make things better? The claim is not obvious, and working out why it holds is the whole subject.

Begin with the target. Why not aim for perfect reliability?

Two reasons, and the second is the interesting one. The first is cost: each additional nine of availability roughly multiplies the engineering effort. The second is that your users cannot perceive the difference anyway. Their phone's network is less reliable than your service already, so improving from 99.99 to 99.999 percent is invisible beneath the noise of the path between you. Reliability above the level at which users can detect it is expenditure with no recipient.

So a finite target is right. Now watch what a target does arithmetically. If the objective is 99.9 percent success over thirty days, then 0.1 percent of requests are permitted to fail — roughly forty-three minutes of full outage, or a wider smear of partial failure. That remainder is the error budget: not an estimate of what will go wrong, but an allowance of what is allowed to.

Here is the move that changes the argument. What if that allowance were treated as a resource the feature team may spend?

Then the two positions stop being opinions. Shipping consumes budget, through the risk each change carries; not shipping conserves it. The decision about whether to ship on Friday is no longer a negotiation between temperaments — it is a question with a factual answer: is there budget left? The disagreement has been converted into a measurement.

Notice what that does to incentives, because this is where the "ship faster" claim comes from. Under the old arrangement, the reliability team's only lever was to resist change, so they resisted all of it, including the safe kind. Under a budget, a team with plenty remaining has an earned right to move quickly, and nobody can veto it on a feeling. The budget protects velocity as much as reliability — most people meet it as a brake and are surprised to find it is equally a permission slip.

And the other half: why does it break less? Because the budget introduces a consequence that arrives before the disaster does. The usual policy is that when the budget is exhausted, feature work stops and reliability work takes priority until it recovers. That is a small, automatic, unarguable penalty attached to burning reliability — so the pressure to invest in testing, staged rollout, and fast rollback exists continuously, rather than only in the fortnight after an outage.

A subtlety worth pausing on: what does it mean to have budget left over at the end of a period?

It means you were more reliable than you promised — which sounds like success and is actually a signal that you were too conservative. You paid for reliability the user could not perceive, and you did it by shipping less than you safely could have. Chronic underspend means the target is wrong, in the same way chronic overspend does.

Now the honest limitations, because this mechanism is easier to describe than to run.

It only works if the indicator behind it genuinely reflects user experience — a budget computed from a metric that does not track users measures nothing, however rigorously it is enforced. It works badly for rare, catastrophic risk: a thirty-day budget says little about a once-a-decade data loss, and treating security incidents as budget spend is a category error. And the freeze is a social commitment, not a technical one. A budget that is exhausted and then waived under commercial pressure is worse than no budget, because it teaches everyone that the numbers are decorative.

Three clarifications.

The first is that the budget is a policy instrument, not a forecast. Nobody plans to use it evenly. Its function is to make a decision rule exist before the pressure that would otherwise decide it — and its value is highest at precisely the moment it is least welcome.

The second corrects a widespread misreading: that an error budget makes reliability optional. It does the opposite. It makes reliability a shared currency with a price, so every team faces the cost of the risk it introduces rather than externalising it onto whoever is on call. That is why it belongs in the vocabulary of organisational design and not only of engineering.

The third concerns burn rate, which is what makes the budget operationally useful rather than merely retrospective. Watching the rate at which budget is being consumed lets you alert on trajectory instead of on threshold: a fast burn over a short window means something acute is happening now, while a slow burn over a long window means chronic degradation that no single incident would ever trigger. Alerting on burn rate is how a budget stops being a monthly report and becomes the thing that pages you.

• Multi-window multi-burn-rate alerting, and why a single threshold alert is a poor budget alarm.
• Rolling windows versus calendar periods, and the gaming each invites.
• What to do about dependencies whose failures you inherit but cannot control.

Every term the collection defines is gathered in the glossary.