Control plane and data plane

Why can a platform keep serving traffic perfectly while being completely unable to change anything?

Every so often a cloud provider posts an incident note with a strange shape: existing instances continued to serve traffic normally, but customers could not launch new ones, change configuration, or scale. Nothing was down, in the sense that users of the running services noticed nothing. And yet the platform was, from the operator's chair, completely frozen.

Read that as a single system and it makes no sense — how can a thing be simultaneously healthy and paralysed? It only makes sense if there are two systems in the box, doing different jobs, with different failure consequences. The interesting question is not what they are called but why they are worth separating, and why a great deal of engineering goes into making sure one can survive the other.

Ask what a router, load balancer, or container platform actually does. Two things, with almost nothing in common except the traffic they concern.

One happens per request, millions of times a second: consult a table, pick a destination, forward the packet. Repetitive, latency-critical, and — crucially — it reads state rather than writing it. The other happens when the world changes: a new instance registers, a deployment shifts weights, a health check fails and something leaves rotation. Comparatively rare, tolerant of seconds of latency, and it writes the state the first job reads.

Now ask what each needs to be correct. The forwarding job needs the table and nothing else; given a table it runs indefinitely without talking to anybody. The change job needs consensus, because a membership table that disagrees between nodes causes real damage — and consensus means quorum, which means a dependency that can fail.

So the two have opposite dependency profiles. Build them as one component and the forwarding path inherits the coordination dependency, so a store that cannot reach quorum stops your traffic and not merely your deployments. Split them, and the forwarding path's only dependency is a table it already has.

That is the design, and it has a name worth knowing: static stability, described at length in Amazon's Builders' Library. A statically stable data plane keeps operating on its last known configuration when the control plane is unavailable. It stops learning; it does not stop working.

That implies something about how the state must be delivered. If the data plane asks the control plane on every request, the separation is nominal — the dependency is still on the hot path. So configuration must be pushed ahead of time and cached locally, and the data plane must prefer stale-but-serving over fresh-or-nothing, which means accepting a failure mode where it confidently routes to a topology that is no longer true.

There is a second, quieter reason to separate them. The control plane handles a tiny fraction of the requests but touches far more of the system's variety: every API shape, validation rule, workflow, integration. It is where most of the code and most of the bugs live while carrying almost none of the traffic, so keeping it off the path that runs millions of times a second is a blast-radius decision as much as a performance one. That also explains the asymmetry in real incidents: control plane failures are common and, when the design is sound, survivable, while data plane failures are rarer because the component is deliberately simpler.

But there is a sting. A frozen control plane is survivable only while nothing needs to change. The moment something does — an instance dies and cannot be replaced, a certificate expires, traffic spikes and autoscaling cannot respond — the outage arrives late, through the data plane, caused by the control plane failure that looked harmless an hour earlier. Static stability buys time, not immunity, which is why operators care so much about whether a failover can be executed without the control plane at all.

Three refinements.

First, this is a split of responsibilities, not necessarily of machines. The same box can run both, and often does at small scale; what makes the pattern real is that the request path does not synchronously depend on the configuration path. A single binary with a locally cached config and a background updater is already following the principle.

Second, the boundary is rarely clean. Health checking, rate limiting, and metrics sit awkwardly between: they observe per-request behaviour but produce decisions that change configuration. Each must be examined for whether it introduces a hot-path dependency by the back door — a rate limiter consulting a central counter on every request has quietly reattached the two planes.

Third, some designs add a management plane for the human- and API-facing part — accounts, quotas, billing, audit — distinct from the control plane that programs the data plane. The two-way split carries the failure-domain argument; the three-way split is mostly an organisational and security boundary, and the terminology is not used consistently across vendors.

• How configuration is pushed and cached so the data plane never blocks on a fetch, and what happens on cache expiry.
• Which recovery actions can be performed without a control plane, and how failover is tested against its absence.

Every term the collection defines is gathered in the glossary.