Why Your Retries Need Idempotency Keys

A few years ago I got paged at 2am because a merchant had been charged three times for the same settlement batch. Not a huge sum, about 4,200 euros total, but it was the kind of number that ends up in a regulator's inbox if you handle it badly. The root cause took twenty minutes to find and it was embarrassing in its simplicity: our payout worker timed out waiting on the acquirer, retried, and the acquirer had actually processed the first request just fine. It just hadn't told us in time.

That incident is the reason I now treat idempotency keys as non-negotiable on any endpoint that moves money or mutates state. Not a nice-to-have. Not a Phase 2 item. If your service retries, and every distributed service retries, then you have already decided to send duplicate requests. Idempotency is just whether you decided what happens when they land.

Retries Are Not Optional, So Duplicates Aren't Either

People sometimes talk about retries as if they were a feature you can turn off. They aren't. TCP retransmits. Your HTTP client retries on connection reset. Your message queue redelivers when an ack goes missing. Kubernetes restarts the pod mid-request. A load balancer gives up after 30 seconds and the caller tries again. Somewhere in that stack, a request you thought happened once is going to happen twice, and you don't get a vote.

The uncomfortable truth is that the network cannot tell you the difference between "the request failed" and "the request succeeded but the response got lost." From the caller's side those two look identical: a timeout. So the caller does the only sane thing and tries again. If the first attempt actually went through, you now have a duplicate, and if your handler isn't built to notice, you've just double-charged someone.

This is why I get twitchy when an engineer tells me an operation is "basically safe to retry." Basically safe means unsafe. Either the operation is provably idempotent or it is a latent double-spend waiting for a bad Tuesday.

What an Idempotency Key Actually Buys You

An idempotency key is just a client-supplied identifier that says "this is the same logical operation as before, even if you're seeing it for the second time." The client generates it once, usually a UUID, and sends it on every retry of that specific request. The server's job is to guarantee that no matter how many times a request arrives with that key, the observable effect happens exactly once.

The subtle part, the part people get wrong, is that you must also return the same response. It isn't enough to skip the second charge. If the first call created a payment and returned a payment ID, the retried call has to return that same payment ID and the same status. Otherwise the client thinks its second attempt failed and, you guessed it, retries again. I once watched a system that correctly deduplicated the side effect but returned a generic 500 on the duplicate. The client read that 500 as failure and hammered the same key in a tight loop until someone noticed the graph.

An idempotency key isn't a lock on your database. It's a promise to the caller: send me this as many times as you need, and I will behave as if you sent it once.

The Key Must Come From the Client

Here is an opinion I'll defend loudly: the server must not generate the idempotency key. I've reviewed designs where someone hashed the request body server-side and called it idempotency. That breaks the moment two genuinely different operations happen to serialize identically, or the moment the same logical operation carries a timestamp or a nonce that changes between retries. Now identical operations look different, or different operations look identical. Both failure modes are ugly.

The client knows what a single logical operation is. It generates one key per operation and reuses it across retries. That's the whole contract. Stripe's API works this way, and it's the model I copy without apology. When a customer clicks "Pay" once, the frontend mints one key and holds it, even across page refreshes if it can, so a jittery user mashing the button doesn't produce five payments.

Storing the Key, and Scoping It Correctly

You need somewhere to record which keys you've seen and what you replied. The mechanism matters less than the guarantees. I've used a dedicated Postgres table with a unique constraint on the key, which is my default because the database does the hard part for you. Redis works too if you accept its eviction semantics, and I would not use Redis alone for money movement because a lost key means a double charge.

Scope is where teams cut themselves. A raw global key namespace invites collisions and, worse, lets one tenant's key interfere with another's. Think through these before you ship:

• Scope keys per merchant or per account, never globally. Key abc-123 from tenant A must never match tenant B.
• Bind the key to the operation type. A key used for a refund should not be honored on a capture.
• Store a hash of the request payload alongside the key, so a reused key with a different body is a hard error, not a silent replay.
• Give keys a finite lifetime, 24 to 72 hours is usually plenty, so your table doesn't grow forever.

The Race Condition Nobody Tests

The naive implementation is "check if the key exists, if not do the work, then save the key." That has a race the width of a truck. Two retries arrive within milliseconds, both check, both see nothing, both do the work. Congratulations, you built a double-charge machine with extra steps.

The fix is to make the first write of the key atomic with claiming the work. Insert the key first, with a unique constraint, in the same transaction that starts the operation. If the insert fails because the key already exists, you're the loser of the race, and you go read the stored result instead of doing the work. The database's unique index is doing the concurrency control, which is exactly what it's good at.

-- Postgres: claim the key atomically, then decide.
-- The unique index on idempotency_key is the whole trick.

INSERT INTO idempotency_records (idempotency_key, merchant_id, request_hash, status)
VALUES (@key, @merchantId, @requestHash, 'in_progress')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

-- If RETURNING gave you a row, you won: do the real work,
-- then UPDATE the row to 'completed' with the response body.
-- If it gave you nothing, someone else has the key:
-- read their row. If 'completed', replay the stored response.
-- If still 'in_progress', return 409 and let the client retry.

That "still in_progress" case is real. If the original request is genuinely mid-flight, you can't yet return its result. I return a 409 with a short retry-after and let the client back off. Yes, it's a little awkward. It's a lot less awkward than a duplicate payout.

What This Looks Like in .NET

In our services this lives as middleware in front of the money-moving handlers, not sprinkled inside them. Centralizing it means a new endpoint gets idempotency for free instead of relying on some developer remembering the pattern at 5pm on a Friday. Here's the shape of the decision, trimmed down.

public async Task<PaymentResult> HandleAsync(PaymentRequest req, string key)
{
    var claimed = await _store.TryClaimAsync(
        key, req.MerchantId, Hash(req));

    if (!claimed.IsWinner)
    {
        if (claimed.Existing.Status == "completed")
            return claimed.Existing.Response;   // replay, byte-for-byte

        // Original still running -> tell caller to wait.
        throw new InProgressException(retryAfter: TimeSpan.FromSeconds(2));
    }

    if (claimed.Existing?.RequestHash != Hash(req) && claimed.Existing != null)
        throw new IdempotencyConflictException();  // same key, new body

    var result = await _acquirer.ChargeAsync(req);
    await _store.CompleteAsync(key, result);
    return result;
}

Notice the request-hash check. Reusing a key with a different amount is not a retry; it's a bug in the caller, and I want it to blow up loudly rather than quietly do something the caller didn't intend. A 422 with a clear message has saved more than one integration partner from shipping their own double-charge.

Your Idempotency Is Only as Good as the Next Hop

Here's the part that keeps me honest. You can implement all of this perfectly and still double-charge if the system you call isn't idempotent. My payout worker was fine; the acquirer's timeout window was longer than mine, so I gave up and retried while they were still working. The real fix was passing our idempotency key downstream to the acquirer, who honored it, so the duplicate died at their door instead of ours.

Always propagate the key to any external call that mutates state. If the downstream provider doesn't support idempotency keys, that's a serious gap in their product and I'd weigh it in the procurement decision, not treat it as a detail. When you can't propagate, you fall back to reconciliation: a scheduled job that compares what you think you sent against what actually settled, and flags the mismatches for a human. It's slower and it's ugly, but it catches the duplicates your keys couldn't, and in this business you want two independent ways to be right.

What I Tell New Teams On Day One

Whenever I inherit a service, the first thing I ask is which endpoints move money or change balances, and which of those have idempotency. The answer is usually "some" and "we think so," which means no. So I make it a rule rather than a code review comment, because rules survive turnover and comments don't.

The rule is short. Any state-mutating endpoint accepts an idempotency key. Any call to an external provider forwards one. Any operation without one gets a written justification in the PR, and "it's read-only" is the only justification I accept. It sounds rigid, and it is, deliberately. The cost of the discipline is a few extra lines per endpoint. The cost of skipping it is a 2am page and a conversation with compliance about why a customer paid twice.

Conclusion

If I could tattoo one idea onto every backend engineer building payment infrastructure, it would be this: exactly-once delivery does not exist, but exactly-once effect is achievable, and idempotency keys are how you get there. The network will always give you at-least-once. Your job is to turn that into at-most-once side effects without dropping the ones that matter. The keys are cheap. The discipline is the expensive part, and it's the part worth paying for before your first bad Tuesday, not after.