Why adding a worker stops the others

Why does bringing another consumer online briefly halt every consumer already running?

You are behind on a stream, so you start one more consumer. The sensible expectation is that throughput rises, perhaps after a moment while the newcomer warms up. What the dashboards show instead is every consumer — including the healthy ones that were never going to be touched — stopping together for a second or several, and lag briefly getting worse.

Nothing failed. You added capacity and the system paused. That is odd enough to be worth explaining rather than tolerating, especially since the same pause happens on every deploy, every restart, and every time a consumer is momentarily slow.

Ask first what invariant the group is protecting. Each partition must be read by exactly one consumer in the group at a time. Not "usually one" — exactly one, because two readers on one partition would process the same records twice, commit conflicting positions, and destroy the per-partition ordering that is the whole reason partitions exist.

Now suppose membership changes. Someone must compute a new assignment, and every member must adopt it. What could go wrong during the changeover? If one consumer adopts the new plan while another is still acting on the old one, some partition is briefly owned by two of them. The invariant breaks in exactly the window where nobody is watching.

So how do you guarantee no overlap? The cheapest correct answer — cheap in reasoning, expensive in downtime — is a barrier: everybody lets go first, then everybody picks up. If no member holds anything at the moment the new plan is computed, no two members can hold the same thing afterwards. That is the eager rebalance protocol, and it explains the symptom precisely. Consumption stops group-wide because stopping group-wide is the proof of correctness.

Trace the states and it becomes concrete. A coordinator on one broker watches the group. A join, a departure, or a member that misses its heartbeat moves the group out of stable and into preparing to rebalance; every member must revoke its partitions and rejoin. Once all have rejoined — or the join window expires on the stragglers — an assignment is computed and distributed, and the group returns to stable. During the middle two states nothing is being consumed by anyone.

Which raises the sharper question: was the barrier necessary? Not entirely. Notice that most partitions are not moving. If a group of four gains a fifth member, the majority of assignments are unchanged, and revoking them was pure waste. The cooperative incremental protocol, introduced in Kafka 2.4 under KIP-429, exploits exactly that: members keep the partitions they are retaining and revoke only the ones being reassigned, at the cost of a second rebalance round to hand them over safely. Later work moves assignment computation to the broker itself and removes the group-wide synchronization barrier; that newer consumer group protocol, KIP-848, reached general availability in Kafka 4.0.

There is one more thing worth noticing, because it turns an inconvenience into an incident. Rebalances are triggered by more than deliberate scaling. A consumer whose processing loop takes longer than its allowed poll interval is treated as gone, which triggers a rebalance, which pauses everyone, which makes everyone further behind, which makes the next batch slower, which can evict another member. The loop is self-reinforcing. It is why a group can appear to be "constantly rebalancing" without anyone having deployed anything.

The misconception to correct is that the pause is a warm-up cost belonging to the joining consumer. It is not; it is a coordination cost belonging to the group, and it is paid in full even when the newcomer is instantly ready and even when its share of the work is tiny.

That reframing changes what you tune. Since the cost scales with the number of membership changes rather than their size, the useful moves are the ones that reduce how often membership appears to change: giving each instance a stable identity so a rolling restart does not look like a departure and an arrival, sizing the poll interval against your genuine worst-case processing time rather than the average, and starting several new consumers together rather than one every few minutes.

One qualification, since the newer protocols are easy to over-read. Cooperative rebalancing removes the group-wide stop, not the rebalance. Partitions that genuinely change hands still pause, and their new owner may still have local state to restore before it can process. What was eliminated is the collateral damage to consumers whose assignment never changed — which was most of the pain, but not all of it.

• How static membership lets a rolling restart avoid rebalancing entirely, and what it costs if an instance never comes back.
• Why restoring a local state store can dominate rebalance time, and how standby replicas shorten it.

Every term the collection defines is gathered in the glossary.