Designing Webhooks That Survive Failure

Webhooks are the part of a payments platform that everyone treats as an afterthought until the night they take down a customer's order pipeline. They look trivial on a whiteboard: an event happens on our side, we fire an HTTP request to a URL the merchant gave us, and everyone moves on. In practice, webhooks are a distributed systems problem wearing the costume of a simple POST request, and the gap between those two framings is where outages, duplicate charges, and reconciliation nightmares live.

I have spent a good part of my career building and rebuilding webhook infrastructure for regulated payment flows, and I have learned that the goal is never to prevent failure. Failure is a constant: receivers go down, networks partition, deploys break TLS, and downstream systems rate-limit us at the worst possible moment. The goal is to design a system that degrades predictably and recovers without a human pulling records out of a database by hand. This post is about the concrete decisions that make that possible.

Treat Failure as the Default State

The first mental shift I ask engineers to make is to stop thinking of a delivered webhook as the normal case and a failed one as the exception. At any meaningful scale, a non-trivial fraction of deliveries will fail on the first attempt, every single day. Receivers deploy mid-request, their load balancers drop connections, certificates expire on a Friday, and DNS goes sideways. If your design assumes the happy path and bolts on error handling later, the error handling will always be undersized for reality.

Once you accept that failure is the steady state rather than an edge case, the architecture changes shape. You stop sending webhooks inline from the request that generated the event, because that couples your core transaction latency to a third party's uptime. You stop treating a 500 from the receiver as something to log and forget. Instead, every event becomes a durable record with a lifecycle, and delivery becomes a background process that you can observe, pause, and replay independently of the business logic that created it.

This framing also keeps the conversation honest with the business. When someone asks whether webhooks are reliable, the truthful answer is that individual deliveries are not, but the system as a whole guarantees eventual delivery within a bounded window. That distinction matters for how merchants build against us.

Persist the Event Before You Send It

The single most important pattern I insist on is the transactional outbox. When a payment is captured or a refund settles, the same database transaction that records that state change also writes a row into an outbox table. The webhook is not sent from inside that transaction. A separate worker reads unsent rows and attempts delivery. This decouples the correctness of our financial state from the availability of the merchant's endpoint.

The reason this matters is subtle but important: if you try to send the HTTP request inside the transaction that commits the payment, you have created a dual-write problem. The payment might commit while the webhook send fails, or the send might succeed while the transaction rolls back. With an outbox, the event and the intent to deliver it are committed atomically, and delivery becomes a problem you can retry safely.

CREATE TABLE webhook_outbox (
    id              BIGINT IDENTITY PRIMARY KEY,
    event_id        UNIQUEIDENTIFIER NOT NULL,
    endpoint_id     BIGINT NOT NULL,
    payload         NVARCHAR(MAX) NOT NULL,
    status          VARCHAR(20) NOT NULL DEFAULT 'pending',
    attempts        INT NOT NULL DEFAULT 0,
    next_attempt_at DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
    created_at      DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
    delivered_at    DATETIME2 NULL,
    CONSTRAINT uq_event_endpoint UNIQUE (event_id, endpoint_id)
);

CREATE INDEX ix_outbox_due
    ON webhook_outbox (next_attempt_at)
    WHERE status = 'pending';

Notice the unique constraint on event and endpoint. That is not decoration. It is the guardrail that stops us from enqueuing the same logical delivery twice when a worker crashes between writing the row and acknowledging it. The filtered index keeps the dispatcher's polling query cheap even when the table holds millions of historical rows.

Give Receivers the Tools to Be Idempotent

Because we retry, receivers will sometimes get the same event more than once. This is unavoidable in any at-least-once delivery system, and pretending otherwise is how you end up double-shipping orders. Rather than apologize for it, I design for it explicitly and document it loudly. Every webhook carries a stable, unique event identifier that does not change across retries, and we tell integrators in plain language to deduplicate on it.

If your webhook can be delivered once, it can be delivered twice. The only safe assumption for a receiver is that any given event may arrive more than once, and the only safe response is to make processing idempotent on the event identifier.

I also resist the temptation to make our identifiers clever. They are opaque, immutable, and never reused. We do not encode meaning into them that a receiver might come to depend on, because the moment a customer parses our IDs, those IDs become a public contract we can never change. A boring identifier that does exactly one job is worth far more than a structured one that quietly leaks our internal schema.

Design Retry Schedules That Respect the Receiver

Naive retry logic is worse than no retry logic. If a merchant's endpoint is struggling and we hammer it with immediate retries from a thousand pending events, we become a denial-of-service attack against the very customer we are trying to serve. Exponential backoff with jitter is the baseline, but the schedule needs to be tuned to the realities of how outages actually resolve.

Short outages, like a deploy or a transient gateway hiccup, resolve in seconds to minutes. Longer outages, like an expired certificate or a misconfigured firewall, can take hours because they require human intervention on the merchant's side. A good schedule front-loads a few quick attempts to recover from transient blips, then spreads the remaining attempts across a window long enough to survive a business-hours fix. Here is the kind of schedule I tend to land on:

• Attempts one through three within the first minute, to ride out transient connection resets.
• Attempts four through six over the next thirty minutes, for short deploy windows and rolling restarts.
• Remaining attempts spread across the following twenty-four to seventy-two hours, with jitter, to survive an outage that needs someone to wake up and fix a certificate.
• A hard cap after which the event moves to a dead-letter state and stops consuming worker capacity.

The jitter is not optional. Without it, every event that failed during the same outage retries in lockstep, producing synchronized thundering herds that knock the receiver down again the instant it recovers. Spreading retries across a randomized window turns a stampede into a manageable trickle.

Sign Every Payload and Make Verification Easy

A webhook endpoint is a publicly reachable URL that accepts state-changing events. If a receiver cannot verify that a request genuinely came from us, an attacker can forge a refund notification or a payment-succeeded event and trick the merchant's system into shipping goods for free. Signing is not a nice-to-have in a payments context; it is a baseline security control that auditors will and should ask about.

We sign each payload with an HMAC over the raw request body plus a timestamp, using a per-endpoint secret that the merchant can rotate. The timestamp is inside the signed content so receivers can reject stale requests and blunt replay attacks. Crucially, I push hard on documentation and sample code here, because a signing scheme that integrators implement incorrectly is a signing scheme that quietly fails open.

public bool IsValidSignature(string rawBody, string timestamp, string provided, string secret)
{
    // Reject requests older than five minutes to limit replay windows.
    if (!DateTimeOffset.TryParse(timestamp, out var sent) ||
        DateTimeOffset.UtcNow - sent > TimeSpan.FromMinutes(5))
    {
        return false;
    }

    var signedContent = $"{timestamp}.{rawBody}";
    using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
    var hash = hmac.ComputeHash(Encoding.UTF8.GetBytes(signedContent));
    var expected = Convert.ToHexString(hash);

    // Constant-time comparison to avoid leaking timing information.
    return CryptographicOperations.FixedTimeEquals(
        Encoding.UTF8.GetBytes(expected),
        Encoding.UTF8.GetBytes(provided));
}

The constant-time comparison matters more than people expect. A naive string equality check returns faster when the first byte differs, and a patient attacker can use that timing difference to recover a valid signature byte by byte. Using a fixed-time comparison closes that door for the cost of one method call.

Do Not Promise Ordering You Cannot Keep

One of the most common mistakes I see is a webhook system that implies events arrive in the order they occurred. Once you have retries, parallel workers, and an unreliable network, ordering guarantees become extraordinarily expensive to maintain and almost impossible to guarantee end to end. A payment-authorized event can easily arrive after the payment-captured event that logically followed it.

Rather than fight this, I design the payload so that receivers never need ordering to be correct. Each event carries the full current state of the resource it describes, along with a monotonic version or sequence number. A receiver compares the version against what it last saw and ignores anything older. This turns ordering from a delivery guarantee we must keep into a property the receiver can derive locally from data we already send.

This approach has a pleasant side effect. Because each event is self-describing and carries the resource's current state, a receiver that missed an event entirely can often recover simply by processing the next one. We are not forced to replay the entire history in perfect order just to get the merchant's view back in sync, which makes the whole system far more forgiving of gaps.

Make Delivery Observable to You and the Merchant

You cannot operate what you cannot see. Every delivery attempt should produce a structured log entry with the event ID, endpoint, attempt number, HTTP status, latency, and outcome. I want to answer questions like "how many events are pending for this merchant right now" and "what is the p99 time-to-delivery this week" without writing a one-off query under pressure during an incident.

Just as important, merchants need their own window into delivery. A self-service dashboard that shows recent deliveries, response codes, and a button to manually replay a specific event removes an enormous volume of support tickets. When a merchant can see that we attempted delivery eleven times and their endpoint returned a 502 each time, the conversation shifts from "your webhooks are broken" to "let me check my server logs," which is exactly where it belongs.

Internally, the metric I watch most closely is the age of the oldest pending event per endpoint. A growing backlog for a single merchant points to a problem on their side; a backlog growing across many merchants at once points squarely at us. That single dimension distinguishes a customer incident from a platform incident faster than any other signal I have found.

Build a Dead-Letter Path and a Replay Story

Eventually some events exhaust their retry budget. The wrong move is to drop them silently, because in a payments context a lost event can mean a merchant never learns that a refund settled. We move exhausted events to a dead-letter state, alert on the volume, and keep the full payload so we can replay it later. Replay is a first-class operation, not a database surgery procedure performed by a senior engineer at midnight.

A clean replay story also protects us during our own incidents. If we ship a bug that causes the dispatcher to mark deliveries as failed when they actually succeeded, the fix is to re-enqueue the affected events, not to reconstruct them from logs. Because every event is durable and idempotent on the receiver side, replaying a batch is safe by construction: receivers that already processed an event will recognize the identifier and discard the duplicate.

I treat the dead-letter queue as a quality signal, not a dumping ground. A persistent trickle of dead-lettered events for a given endpoint is a prompt to reach out to that merchant proactively, often before they have noticed the problem themselves.

Conclusion

Designing webhooks that survive failure is less about any single clever trick and more about a consistent posture: assume delivery will fail, make the event durable before you ever try to send it, retry with respect for the receiver, sign everything, and give both yourself and your merchants the visibility to understand what is happening. None of these pieces is exotic. The transactional outbox, idempotent receivers, jittered backoff, HMAC signatures, and a real replay path are well-understood patterns. The discipline is in applying all of them together and refusing to treat the easy parts as the whole problem. When you get this right, webhooks stop being the fragile edge of your platform and become one of the quiet, dependable surfaces your customers build their businesses on, which is exactly what infrastructure in a regulated payments environment is supposed to be.