Split-brain

What happens when a system that must have exactly one leader briefly has two?

Many systems insist on a single leader: one node that accepts writes, hands out sequence numbers, or decides what happens next. The usual justification is that a single decider makes reasoning easy. Two deciders would be chaos.

So here is the question worth sitting with. Suppose the machinery that elects that leader is careful — majority votes, heartbeats, timeouts, all of it. Can the system still end up with two nodes each sincerely believing it is the leader? And if the answer is yes, what exactly is the failure: is it that the election was wrong, or something more uncomfortable?

Start with how a leader is normally deposed. The other nodes stop hearing from it, and after some timeout they declare it dead and elect a replacement. Now ask the awkward question: what did they actually observe? They observed silence. And silence has at least two causes — the leader crashed, or the network between them broke. From the outside, over an asynchronous network, those two are indistinguishable. That is not an engineering oversight; it is a proven limit. You cannot reliably tell a dead node from an unreachable one by waiting.

Follow the consequence. If the network partitioned rather than the leader crashing, the old leader is still running, still holding what it believes is its mandate, and still willing to accept writes from any client that can reach it. Meanwhile the majority side has elected a new leader that is also accepting writes. Two leaders, both honest, both following the protocol correctly. Nothing is broken in the sense of a bug.

Now notice something sharper. It is tempting to say quorum solves this: require a majority to elect, and the minority side cannot elect anyone. True — but that only stops a second leader from being elected. It does nothing about the first one, which was elected legitimately and simply has not learned it was replaced. Quorum bounds how many leaders can be created; it does not retract the ones already at large.

And the partition is not even required. Consider a leader that is paused — a long garbage-collection stop, a hypervisor freezing the VM, a disk write that blocks for thirty seconds. Its lease expires, the cluster promotes someone else, and then the paused process resumes mid-sentence, with no way of knowing that time passed. It sends the write it was about to send. From its own perspective, nothing happened at all.

So what does the damage? Not the existence of two leaders as such, but the fact that a shared resource downstream — a database, a file, a queue — accepts writes from both and cannot tell them apart. Each request looks perfectly valid. If both leaders write, the resource ends up with a state neither leader intended, and reconciling it afterwards is often impossible because the information about which write was legitimate no longer exists anywhere.

This suggests where the fix has to live. If the two leaders cannot detect each other, and the leader itself cannot detect its own staleness, then the only party in a position to notice is the resource being written to. Give every leadership term a number that only increases — a term, an epoch, a fencing token. The leader attaches its number to every request. The resource remembers the highest number it has ever seen and refuses anything lower. The stale leader is not asked to know it is stale; it is simply told no, at the moment it matters.

Which reframes the whole problem. We are not trying to prevent two leaders from existing. We are trying to prevent two leaders from both being effective. Those are different goals, and only the second one is achievable.

Timing helps but does not close the gap. Leases — leadership that expires unless renewed — shrink the window, and a leader that checks its lease before acting is much safer than one that does not. But the check and the action are separate moments, and something can preempt the process in between. Hardware fencing in high-availability clusters takes the blunter route: the surviving node power-cycles the suspect one, so the question of what it believes stops mattering.

Three refinements.

First, the misconception this most often corrects: split-brain is usually described as a bug in leader election, which makes people look for a smarter election. But the election can be flawless. The gap is between when leadership is revoked and when the revoked leader finds out — and that gap cannot be driven to zero, only made harmless.

Second, quorum and fencing solve different halves. Quorum stops a minority partition from minting a competing leader. Fencing stops an already-minted leader from acting after it has been superseded. Systems that have one and not the other are still exposed, which is why consensus protocols carry a monotonic term number through every message rather than only using votes at election time.

Third, the resource has to participate. A fencing token is worthless if the storage layer ignores it. This is why "we use Raft" does not by itself protect data that Raft does not mediate: any external system a leader touches must itself check the token, or the guarantee stops at the edge of the consensus group.

• How Raft threads its term number through every message, and what breaks if a message omits it.
• Leases and clock assumptions: what a bounded-drift assumption buys, and what it costs when violated.
• Reconciliation after divergence, and why some data models survive it while others cannot.

Every term the collection defines is gathered in the glossary.