Every payment platform I have run eventually develops a relationship with failure. Not the catastrophic kind that wakes you at three in the morning, but the small, persistent kind: a settlement API that times out under load, a card network that returns a transient decline, a database connection that drops mid-transaction. These failures are not bugs to be eliminated so much as conditions to be managed. The question is never whether a downstream dependency will fail, but how your system behaves in the seconds and minutes after it does.
Retry logic is the most common answer, and exponential backoff is the most common refinement of that answer. Both are deceptively simple to describe and surprisingly easy to get wrong. I have seen well-meaning retry code amplify a minor outage into a self-inflicted denial of service, and I have seen overly cautious retry policies silently lose payments that should have completed. What follows is how I think about the problem, the rules I hold teams to, and the specific patterns that have held up in regulated fintech production for years.
Why Retries Exist at All
A retry is a bet that a failure is transient. When a request fails, you are implicitly classifying it: either the failure is permanent and retrying is pointless, or the failure is temporary and a second attempt has a reasonable chance of succeeding. That classification is the entire foundation of retry logic, and most production incidents I have investigated trace back to getting it wrong. Retrying a permanent failure wastes resources and delays the inevitable error. Failing fast on a transient error throws away requests that would have worked.
In a payment context, the stakes are concrete. A transient network blip between our service and an acquiring bank is exactly the kind of failure a retry should absorb. The customer should never see it. But a response indicating insufficient funds or an invalid card is permanent within the lifetime of that request, and retrying it does nothing except generate noise in the bank's fraud monitoring. The art is in drawing that line correctly and encoding it in something more disciplined than a try-catch with a loop counter.
Distinguishing Transient from Permanent Failures
The first thing I ask any engineer proposing a retry is: which specific errors are you retrying, and why? A retry policy that catches every exception is not a policy, it is a hope. The classification has to be explicit and it has to be conservative. When in doubt, do not retry, because the cost of an unnecessary retry against a payment endpoint is almost always higher than the cost of surfacing a clean error.
For HTTP-based dependencies, I treat the boundary roughly like this:
- Retry on connection timeouts, connection resets, and DNS failures, since these almost always indicate transient network or capacity issues.
- Retry on HTTP 502, 503, and 504, which signal that an upstream is overloaded or briefly unavailable.
- Retry on HTTP 429 only when the response includes a Retry-After header, and honor that header rather than your own schedule.
- Do not retry on 400, 401, 403, or 404, because the request itself is malformed or unauthorized and a second attempt will fail identically.
- Treat 500 with caution: retry only if you know the operation is safe to repeat, because a generic server error may mean the work partially completed.
That last point is the one teams underestimate. A 500 from a settlement endpoint might mean the request never landed, or it might mean the payment was recorded and the response was lost on the way back. Those two cases demand opposite behavior, and you cannot tell them apart from the status code alone. This is why retries and idempotency are inseparable, which I will come back to.
Why Naive Retries Cause Thundering Herds
The most dangerous retry implementation is the simplest one: catch the error, wait a fixed interval, try again. The problem is what happens at scale. When a downstream service degrades, it does not fail one request at a time. It fails thousands simultaneously. If every one of those failed requests waits exactly one second and retries, you have just scheduled a synchronized wave of traffic to hit a service that is already struggling. The retry storm arrives, the service falls over harder, and the next wave is even larger.
This is the thundering herd, and I have watched it turn a thirty-second blip into a forty-minute outage. The service had recovered within seconds, but the retry traffic from the initial failure kept knocking it back down. The fix is to spread retries out over time and, critically, to spread different clients' retries across different points in time so they stop arriving in lockstep. That is precisely what exponential backoff with jitter is designed to do.
The cruelest outages are the ones your own recovery logic prolongs. A retry policy that ignores the health of the system it is retrying against is not resilience, it is a feedback loop waiting to oscillate.
The Mechanics of Exponential Backoff
Exponential backoff means each successive retry waits longer than the last, growing the delay geometrically. A typical schedule doubles the wait each time: one second, then two, then four, then eight. The intuition is that if a service has not recovered after one second, giving it two more is cheap, and if it has not recovered after eight seconds, hammering it every second was never going to help anyway. The exponential curve concentrates your early attempts where transient failures resolve quickly, then backs off gracefully as the failure starts to look persistent.
You always pair backoff with two bounds: a maximum delay, so you do not end up waiting minutes between attempts, and a maximum attempt count, so a genuinely dead dependency eventually produces a clean failure rather than an infinite loop. In my services the defaults are usually a base of around two hundred milliseconds, a cap of perhaps thirty seconds, and three to five total attempts depending on how time-sensitive the operation is. A synchronous request behind a customer-facing checkout gets fewer, faster retries; an asynchronous reconciliation job can afford to be patient.
The piece that separates a robust implementation from a fragile one is jitter. Pure exponential backoff still synchronizes clients, because they all double their delays in the same pattern. Adding randomness to each delay scatters the retries across the time window so they no longer arrive together. The variant I default to is full jitter, where each delay is a random value between zero and the current exponential ceiling.
Implementing Backoff with Jitter in .NET
Rather than hand-roll this, I lean on a well-tested resilience library. In the .NET stack that means Polly, which expresses backoff, jitter, and attempt limits declaratively. Hand-rolled retry loops accumulate subtle bugs around exception filtering and cancellation, and I would rather review a policy definition than a bespoke loop. Here is the shape of a policy I would consider production-ready for an outbound payment call:
var retryPolicy = new ResiliencePipelineBuilder<HttpResponseMessage>()
.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 4,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(200),
MaxDelay = TimeSpan.FromSeconds(30),
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.Handle<TimeoutRejectedException>()
.HandleResult(r => r.StatusCode == HttpStatusCode.ServiceUnavailable
|| r.StatusCode == HttpStatusCode.GatewayTimeout),
OnRetry = args =>
{
_logger.LogWarning(
"Retry {Attempt} after {Delay}ms for {Operation}",
args.AttemptNumber, args.RetryDelay.TotalMilliseconds, "settlement");
return default;
}
})
.Build();
A few things in that policy are deliberate. The ShouldHandle predicate is explicit about which results trigger a retry, so a 400 or 401 falls straight through without burning attempts. UseJitter is on, so Polly randomizes the exponential delays for me. And the OnRetry callback emits a structured log on every attempt, because retries that happen silently are retries you cannot reason about during an incident. I want the attempt number and the delay in my logs and metrics, every time.
Idempotency Is Not Optional
Here is the rule I will not compromise on: you may only retry an operation if repeating it is safe. For a read, that is trivially true. For a write, and especially for a write that moves money, it is true only if you have engineered it to be. A retry against a charge endpoint without idempotency protection is how you double-charge a customer, and in regulated payments that is not a bug, it is a reportable incident.
The mechanism is an idempotency key: a unique identifier the client generates for a logical operation and sends with every attempt, including retries. The server records the key alongside the result of the first successful execution. When a retry arrives carrying a key the server has already processed, it returns the stored result instead of executing the operation again. The payment moves exactly once no matter how many times the network forces us to ask.
CREATE TABLE idempotency_keys (
key VARCHAR(64) PRIMARY KEY,
request_hash CHAR(64) NOT NULL,
response_body JSONB,
status_code INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
-- First attempt inserts the key; concurrent retries collide on the
-- primary key and must wait for or read the original result.
INSERT INTO idempotency_keys (key, request_hash, expires_at)
VALUES ($1, $2, now() + INTERVAL '24 hours')
ON CONFLICT (key) DO NOTHING;
The request_hash matters: if a client reuses a key with a different payload, that is a client bug and you want to reject it loudly rather than silently return the wrong stored response. Idempotency and retry policy are two halves of the same design. I do not approve one without the other for any operation that mutates state.
Pairing Backoff with Circuit Breakers
Backoff handles the timing of individual retries, but it does not answer a higher-level question: when should we stop trying entirely and give the downstream a chance to recover? That is the job of a circuit breaker. After a threshold of consecutive failures, the breaker opens and fails fast for a cooling-off period, sending no traffic at all to the struggling dependency. When the cooldown elapses, it allows a trickle of probe requests through; if they succeed, it closes and normal traffic resumes.
The interaction between retries and circuit breakers needs thought. If you place retries inside the breaker, a burst of retries can trip it prematurely. I generally want the breaker counting logical operations, not individual retry attempts, so that a single operation exhausting its retries registers as one failure rather than four. The combination is powerful: backoff and jitter smooth out transient blips, while the breaker protects against sustained outages by cutting off traffic that has no hope of succeeding. Without the breaker, your beautifully jittered retries simply keep a dead service company.
Observability and Retry Budgets
Retries change the meaning of your metrics, and if you are not careful they hide the truth. A dashboard showing a healthy success rate can be masking a downstream that fails half the time and only succeeds on the second attempt. I insist that teams instrument retries as first-class events: count attempts separately from logical requests, track the distribution of attempts per operation, and alert when the retry rate climbs even while the eventual success rate looks fine. A rising retry rate is an early warning that a dependency is degrading before it fails outright.
The other discipline I enforce is a retry budget. Rather than letting every request retry independently, you cap the proportion of total traffic that may be retries, often somewhere around ten percent. When the budget is exhausted, additional retries are suppressed. This is a system-wide circuit breaker against the thundering herd: it guarantees that retry traffic can never balloon to several times your baseline load, no matter how widespread the failures. Combined with per-client jitter, it keeps your recovery behavior from becoming the thing that needs recovering.
Conclusion
Retry logic looks like a small piece of plumbing, but it encodes some of the most consequential decisions in a payment system: what failures you trust to be transient, how hard you are willing to lean on a struggling dependency, and whether repeating an operation can ever harm a customer. Exponential backoff with jitter, bounded attempts, idempotency keys, circuit breakers, and retry budgets are not five separate features. They are one coherent stance toward failure, and they only work in concert. Get the classification right, make every retryable write idempotent, scatter your attempts so they do not arrive in lockstep, and give your dependencies room to recover. Do that, and the small, persistent failures stay invisible to your customers, which is exactly where they belong.