Idempotency Keys Explained Simply

Every payments engineer eventually meets the same ghost. A customer taps "Pay" once, the network hiccups, the mobile app retries on their behalf, and suddenly your ledger shows two charges where the customer intended one. The money is real, the duplicate is real, and the support ticket that follows is very real. Idempotency keys are the boring, unglamorous mechanism that makes this entire class of problem disappear, and yet they remain one of the most misunderstood tools in our trade.

I have spent enough years operating regulated payment infrastructure to believe that idempotency is not an optimisation you bolt on later. It is a property you design for from the first endpoint. In this post I want to explain idempotency keys plainly, without the academic distributed-systems vocabulary, and show you how I actually implement and reason about them in production.

What Idempotency Actually Means

An operation is idempotent if performing it once produces exactly the same result as performing it ten times. Reading your account balance is naturally idempotent: asking the question repeatedly does not change the answer. Deleting a record by its identifier is idempotent too, because once it is gone, asking again to delete it changes nothing. The trouble lives in the operations that create something or move money, because those are the ones where a repeat genuinely does damage.

An idempotency key is a unique token that the caller attaches to a request so the server can recognise a retry of an operation it has already performed. The client generates the key, the server remembers it, and if the same key arrives twice the server returns the original outcome instead of doing the work again. The key turns an otherwise dangerous operation, like "charge this card for forty euros", into one that is safe to call as many times as the network forces you to.

The subtlety people miss is that idempotency is a contract between two parties, not a feature of one of them. The client promises to reuse the same key for what it considers the same intent. The server promises that a repeated key yields the same result. If either side breaks its half of the bargain, the protection collapses.

Why Retries Are Not Optional

It is tempting to think you can dodge the whole problem by simply not retrying. That instinct is wrong, and dangerously so. In any distributed system, the failure modes are not symmetrical. When a request times out, you genuinely do not know whether the server processed it. The response may have been lost on the way back to you even though the work completed. If you refuse to retry, you trade duplicate-charge risk for dropped-payment risk, and a payment that silently vanishes is just as damaging to trust as one that fires twice.

So retries are unavoidable. Mobile clients retry. API gateways retry. Message queues redeliver. Your own background workers retry. The only honest engineering posture is to assume that every request you receive might be a duplicate, and to make duplicates harmless. Idempotency keys are how you keep retries aggressive and safe at the same time, which is exactly what you want when money is on the line.

The network does not promise you exactly-once delivery. It promises you at-least-once delivery and lets you build exactly-once processing on top. Idempotency keys are the layer where you do that building.

Generating Good Keys

A good idempotency key is unique per logical operation and stable across retries of that operation. The cleanest approach is for the client to generate a UUID at the moment the user expresses intent, for example when they press the pay button, and to attach that same UUID to every retry of that one payment. If the user starts a genuinely new payment, they generate a new key. This maps the key precisely to user intent, which is the boundary that actually matters.

What you must avoid is deriving keys from data that changes between retries, such as a timestamp or a request sequence number, because then every retry looks like a brand-new operation and the protection evaporates. Equally, you should avoid keys that are so coarse they collide across genuinely different operations. I have seen teams key on the customer ID alone, which means a customer can only ever make one successful payment, because the second is treated as a duplicate of the first.

• Generate the key on the client, at the point of intent, before the first attempt.
• Reuse the identical key for every retry of that same intent.
• Use a high-entropy value such as a UUID v4 so accidental collisions are effectively impossible.
• Never recycle a key for a different operation, and never derive it from mutable request fields.
• Treat the key as opaque on the server; do not try to parse meaning out of it.

Storing and Comparing Keys on the Server

On the server side, the mechanism is a persistent record keyed by the idempotency key. When a request arrives, you look up the key. If you have never seen it, you record it, perform the operation, and store the response. If you have seen it and the operation completed, you return the stored response without doing the work again. The store needs to be durable and shared across all your application instances, which in practice means a database table or a distributed cache with persistence, not in-process memory.

There is an important detail that separates a toy implementation from a correct one. You should also store a fingerprint of the request body alongside the key. If the same key arrives with a different payload, that is a client bug or an attack, and returning the original response would mask it. The correct behaviour is to reject the mismatched request with an error rather than silently doing something the caller did not ask for.

Here is a representative schema and the comparison logic I use as a starting point. The exact column types vary by engine, but the shape is consistent across the systems I have run.

CREATE TABLE idempotency_keys (
    idempotency_key   UUID         NOT NULL,
    request_hash      CHAR(64)     NOT NULL,
    response_status   INT          NULL,
    response_body     JSONB        NULL,
    state             VARCHAR(16)  NOT NULL DEFAULT 'in_progress',
    created_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),
    locked_until      TIMESTAMPTZ  NULL,
    CONSTRAINT pk_idempotency PRIMARY KEY (idempotency_key)
);

-- A retry with the same key but a different body is a contract violation.
SELECT response_status, response_body, state, request_hash
FROM   idempotency_keys
WHERE  idempotency_key = @key
FOR UPDATE;

Handling Concurrent Duplicates

The hardest case is not the slow retry that arrives a second later. It is the two requests with the same key that hit your servers at the same instant, perhaps because an impatient client fired twice or two queue consumers grabbed the same message. If both threads check the store, both see no existing record, and both proceed to charge the card, you have built an idempotency layer that does nothing under the exact conditions it exists to handle.

The fix is to make the "claim the key" step atomic. You insert the key with an in-progress state as the very first action, relying on the unique constraint on the key column to let exactly one request win the race. The loser of that race gets a constraint violation, which it interprets not as an error but as a signal that another request owns this operation. It can then wait briefly and read back the result, or return a conflict status telling the client to retry shortly.

// .NET: claim the key atomically, then do the work.
try
{
    await _db.IdempotencyKeys.AddAsync(new IdempotencyKey
    {
        Key         = key,
        RequestHash = hash,
        State       = "in_progress",
        LockedUntil = DateTimeOffset.UtcNow.AddSeconds(30)
    });
    await _db.SaveChangesAsync(); // throws on duplicate key
}
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
{
    // Another request already owns this key.
    var existing = await LoadAsync(key);
    if (existing.State == "completed")
        return Replay(existing); // return the original response
    return Conflict("Request already in progress; retry shortly.");
}

var result = await _payments.ChargeAsync(request);
await PersistResultAsync(key, result);
return result;

Scoping and Expiry

An idempotency key should never be globally unique across your whole platform without context. It should be scoped, typically to the authenticated merchant or API consumer and to the endpoint. Two unrelated merchants could conceivably generate the same UUID, and even if that is astronomically unlikely with proper entropy, scoping costs nothing and removes the worry entirely. The practical key in your store is therefore the tuple of consumer, route, and the client-supplied key.

Keys should also expire. You are not obliged to remember every key forever, and doing so turns your idempotency table into an ever-growing liability. I generally retain keys for twenty-four hours, which comfortably outlives any realistic retry window for a synchronous payment while keeping the table bounded. The retention period is a deliberate decision: it must be longer than your longest client retry policy, or you will let a genuine duplicate slip through after the record has been purged.

Be careful that expiry interacts cleanly with stuck operations. If a request claims a key, marks it in-progress, and then the process dies before completing, you do not want that key locked forever. The locked_until column in my schema exists precisely so a later request can reclaim an abandoned in-progress record once its lock has lapsed, rather than returning conflicts indefinitely.

Idempotency Versus Deduplication

People often conflate idempotency with deduplication, and while they are cousins, the distinction matters. Deduplication usually means filtering out repeated messages in a pipeline based on some content signature, often best-effort and time-windowed. Idempotency, as I am describing it here, is a stronger and more deliberate guarantee tied to an explicit key the caller controls, with a defined response replay behaviour. Deduplication asks "have I seen something like this recently?" Idempotency asks "have I performed exactly this operation, and if so what did I return?"

The reason to care is that deduplication windows are heuristics and idempotency keys are contracts. When a regulator or an auditor asks how you guarantee a customer is not double-charged, "we dedupe within a five-minute window" is a weak answer. "Each payment carries a caller-supplied key, we store the outcome durably, and we replay the original result for any repeat of that key" is a strong one. In regulated environments, the difference between a heuristic and a guarantee is the difference between an accepted control and a finding.

Common Mistakes I Have Seen

The most frequent mistake is treating the idempotency layer as a thin cache rather than a transactional participant. If you record the key and perform the charge in two separate, uncoordinated steps, you create windows where the charge succeeds but the record does not, or the reverse. The key claim, the operation, and the result persistence must be reasoned about together, ideally with the database transaction boundaries drawn so that a crash anywhere leaves you in a recoverable state.

The second mistake is making idempotency optional in a way that callers ignore. If your API treats the key header as nice-to-have, most integrators will simply omit it, and you will discover the gap during an incident rather than a design review. For any endpoint that creates or moves money, I require the key and reject requests without one. The friction this adds at integration time is trivial compared to the cost of a duplicate-charge incident in production.

The third mistake is replaying the wrong thing. When you return a stored response for a repeated key, you must return the original response faithfully, including its status code, not a freshly computed one. A retry that originally produced a created-resource response should keep producing that same response, so the client converges on a single consistent view of what happened rather than seeing the state shift underneath it.

Conclusion

Idempotency keys are not glamorous, and they will never appear on a product roadmap as a headline feature. But they are the quiet foundation that lets a payment system retry fearlessly, survive network turbulence, and tell a customer with confidence that they were charged exactly once. The mechanism is simple to describe: a unique key, a durable record, an atomic claim, a faithful replay. The discipline is in applying it consistently to every operation that matters, and in resisting the temptation to treat it as an afterthought. Build it in from the first endpoint, scope and expire your keys deliberately, and make the dangerous operations boring. That is the whole job, and it is well worth doing properly.