Idempotent operations

Why must an action that might be retried be designed for it rather than simply repeated?

You send a payment request. The network hesitates. No reply comes back. Do you send it again?

Notice how uncomfortable that question is, and notice exactly why it is uncomfortable. It is not that you do not know what happened — it is that you cannot know, and no amount of waiting will tell you. The silence is identical whether the request never arrived, or arrived and was processed and the acknowledgement was lost on the way home. So the retry is a gamble: it either fixes a lost request or creates a duplicate charge.

That is the whole subject. The interesting move is not "retry carefully." It is to change the operation so that the gamble no longer has two outcomes.

Let us be precise about the impossibility first, because everything else follows from it. This is the Two Generals Problem, and it is genuinely unsolvable, not merely hard: over an unreliable channel, no finite exchange of messages can give both parties certainty that the other acted. Adding an acknowledgement to the acknowledgement just moves the uncertainty to the last message. So a caller facing silence has exactly two choices — give up, or retry — and neither is safe on its own.

Now ask a different question. What property would an operation need, so that retrying it were harmless? Say it plainly: applying it twice must leave the world in the same state as applying it once. That property has a name borrowed from mathematics — idempotence, from idem (same) and potens (power). An operation f is idempotent when f(f(x)) = f(x).

Test that against real operations and watch which pass. "Set the shipping address to 14 Elm Street" — apply it twice, same result. Passes. "Delete order 8812" — the second attempt finds nothing to delete and the world is unchanged. Passes, provided you agree that a second delete reporting "already gone" is success rather than an error. "Add 50 euros to the balance" — fails, plainly. "Append this line to the log" — fails.

Do you see the pattern? Operations that declare a desired state are naturally idempotent. Operations that describe an increment or an event are not. This is exactly why HTTP defines PUT and DELETE as idempotent and POST as not: PUT says "make the resource be this", POST says "here is one more thing." The distinction is not arbitrary convention, it is the same mathematical property showing up in a protocol.

Which brings the awkward case. Most operations that matter commercially are increments and events: charge a card, ship a parcel, send an email, publish a message. You cannot restate them as desired states without losing meaning — "the customer's total charges should be 50 euros" is not what a payment means. So what can you do?

You can give the operation an identity. The caller generates a unique key — a UUID, a client-side order reference — and attaches it to the request. The server records, atomically with performing the effect, that this key has been applied and what the result was. A second request bearing the same key is not performed again; the stored result is returned instead. The operation itself is still an increment, but the request has become idempotent, which is all the caller needed. This is what Stripe's Idempotency-Key header does, and what the deduplication identifiers in message brokers do.

Two details make or break it, and both are easy to get wrong. The key and the effect must be committed together — if you apply the charge and then crash before recording the key, the retry charges again, and you have merely narrowed the window rather than closed it. And the record must live at least as long as the caller might retry, which in practice means hours or days, not the lifetime of a process's memory.

One more honest point. People describe this as achieving "exactly-once delivery." It is not delivery that becomes exactly-once — messages are still delivered many times or none. What becomes exactly-once is the effect, which is why the careful literature says effectively-once. Delivery guarantees come in two real flavours, at-most-once and at-least-once, and idempotence is how you build something that behaves like the third out of the second.

A few refinements that catch people out.

Idempotence is not the same as being read-only. A safe operation changes nothing; an idempotent one may change a great deal, but converges to the same state however many times it is applied. Conversely, an operation can be non-idempotent while looking harmless — an "increment the view counter" endpoint has no obvious risk until a retry storm inflates your numbers.

It is also not the same as commutativity. Idempotence says a repeat is harmless; it says nothing about two different operations arriving out of order. "Set the address to A" and "set the address to B" are each idempotent, and applying them in the wrong order still leaves the wrong address. Retries and reordering are separate problems and want separate defences — often a version number or a conditional update.

And notice which end of the wire bears the cost. The caller cannot make an operation idempotent by retrying it politely; only the receiver can, by defining the operation as a state assertion or by keeping the key ledger. This is why "design for retry" is a contract, not a client-side habit — a client that retries against a service which never promised idempotence is doing something unsafe, however careful its backoff.

• How exponential backoff with jitter keeps safe retries from becoming a retry storm.
• What message brokers actually promise, and how their deduplication windows expire.

Every term the collection defines is gathered in the glossary.