Losing it or doing it twice
A Socratic walk-through of losing it or doing it twice — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why must a system choose between occasionally dropping work and occasionally repeating it?
Every messaging system offers you two delivery guarantees and calls them at-most-once and at-least-once. Nobody wants either. Everybody wants the obvious third option, and the vendors that advertise it turn out to mean something narrower than the words suggest.
That pattern is suspicious. When an entire industry converges on offering only the two settings a customer does not want, the constraint is probably not effort. So what makes the third setting unavailable?
Reasoning it through
REASONING #Start with the smallest possible act: one party sends something, the other receives it and replies "got it". Now let the reply go missing — not the message, just the confirmation.
What does the sender know? It knows it sent. It does not know whether the silence means the message never arrived or means it arrived and the acknowledgement drowned on the way back. Those two worlds are, from where the sender stands, identical. No amount of waiting distinguishes them, and adding a confirmation of the confirmation just moves the ambiguity outward by one hop — the last message in any exchange is always unacknowledged.
So the sender has exactly two policies available for the ambiguous case: send again, or do not. That is not a shortage of imagination; it is the number of branches there are. Sending again risks a duplicate whenever the original had in fact landed. Declining to send again risks a loss whenever it had not. This is the Two Generals problem, and it is not merely hard — it is provably unsolvable over a channel that can drop messages.
Where does that leave the receiver? Facing the same fork in a different costume. It must decide when to acknowledge: before doing the work, or after. Acknowledge first and a crash mid-work destroys the message with the job undone — at-most-once. Acknowledge after and a crash between finishing and acknowledging leaves the sender still holding it, so the work is redone — at-least-once. Same fork, same two branches, no third.
Now, what about the guarantee everyone actually wants? Notice what it would require: not that the message is delivered once, but that its effect occurs once. That reframing is what makes it achievable, and it is achievable in one of two ways.
The first is to make repetition harmless. If the work is naturally idempotent — set this field to this value, mark this order paid — then delivering it five times leaves the world in the same state as delivering it once. If it is not naturally idempotent, you carry a unique identifier with the message and record which identifiers have been processed, so a repeat is recognized and discarded. That is real, and it is the workhorse.
The second is to make the work and the acknowledgement commit atomically, so the crash window closes. Kafka's transactional processing does exactly this: the output records and the consumer's offset advance land in one transaction, so the "I did it" and the "I did it" cannot disagree. But look closely at the boundary. That atomicity holds because both halves live in the same system. The moment the effect is a card charge, an email, or a call to somebody else's API, no transaction spans it, and you are back to identifiers and deduplication.
Two honest limits on deduplication, since it is doing so much load-bearing work. Its memory is finite — a set of seen identifiers has a retention window, and a duplicate that arrives after the window closes is invisible to it. And the record of "seen" must itself be committed atomically with the effect, or you have just moved the crash window rather than removed it.
The analogy
THE ANALOGY #Imagine phoning a bet to a bookmaker on a bad line. You say "fifty on the grey", and the call drops before you hear anything back. You have two options and only two: call again, and possibly stand a hundred; or leave it, and possibly stand nothing. The bookmaker faces the mirror of it — write the slip and then confirm, or confirm and then write.
the analogy makes the ambiguity look irreducible, but a person would simply ring back and ask whether the bet was placed — and that question is answerable only because the bet has a name. Attach a unique reference to the request and a machine can ask it too, which is precisely how at-least-once turns into an effect that happens once.
Clarifying the model
THE MODEL #The misconception to dislodge is that "exactly-once" names a stronger delivery guarantee. It does not. Delivery over an unreliable channel still cannot be exactly once; what a good system provides is exactly-once processing within a boundary it controls, built on at-least-once delivery underneath. Read any such claim by asking where its boundary is drawn, because everything outside it inherits the original fork.
A second refinement: the choice is not a matter of taste, it is a matter of which error is cheaper. Sampled metrics, telemetry, and cache invalidations can afford to drop one; a payment instruction cannot. Conversely, a duplicated "send the welcome email" is an annoyance, and a duplicated "ship the pallet" is a loss. Naming the cost of each error decides the setting, and no default the broker ships knows that answer for you.
Finally, at-most-once is not merely the lazy option. It is the correct one whenever a later message supersedes an earlier one anyway — a fresh sensor reading, a current price. There, redelivering stale work is worse than dropping it.
A picture of it
THE PICTURE #How to readStart at the rounded terminal and take the single decision that matters — whether the acknowledgement goes out before the work or after. The left branch reaches a risk node where the message is simply gone; the right branch reaches a risk node where it comes back. Both branches then meet at the same outcome, but only the right one can get there, and only by passing through the subroutine that recognizes a repeat by its identifier. That box is the entire difference between at-least-once delivery and an effect that happens once.
What became clearer
WHAT CLEARED #The fork exists because a sender cannot distinguish a lost message from a lost acknowledgement, and there are only two things to do about an ambiguity you cannot resolve. So the guarantee you pick is really a choice of which failure to accept. The way out is not a cleverer protocol but a change of subject: stop trying to deliver once and make the effect happen once, by naming each unit of work so a repeat can be recognized — and be honest about where the boundary of that trick ends.
Where to go next
ONWARD #- How a transactional consume-process-produce loop actually commits offsets and output together, and what it costs in latency.
- Why a deduplication window is a retention decision in disguise, and how to size it against your worst realistic redelivery delay.
Key terms
TERMS #| Term | What it means |
|---|---|
| At-most-once | acknowledge before processing; work may be lost, never repeated. |
| At-least-once | acknowledge after processing; work may be repeated, never lost. |
| Two Generals problem | the proof that agreement cannot be guaranteed over a channel that may drop messages, however many rounds are used. |
| Idempotence | the property that applying an operation repeatedly leaves the same result as applying it once. |
Every term the collection defines is gathered in the glossary.