Requirements for Webhooks That Survive Retries

Stripe publishes a document describing how it signs webhook payloads and how many times it will redeliver a failed one. The retry schedule runs for three days, with increasing gaps between attempts. That single number, three days, explains most of the design decisions that separate a webhook endpoint that works from one that quietly corrupts records for months.

A webhook is a POST request that a service fires at a URL the receiver controls, usually to announce that something changed. The sender knows nothing about the receiver's database, its deploy schedule, or whether its queue is backed up. It fires, waits for a response, and decides whether to fire again. Every requirement that follows comes from that asymmetry: one side holds the event, the other holds the consequences, and neither can see the other's state.

Most teams treat webhook handling as a five-line controller: parse the JSON, find the record, update it, return 200. That works in staging, where requests arrive in order and never repeat. In production, the same event shows up twice, then a third time after a deploy, and the five-line controller runs its update three times. If the update is "set status to shipped," nothing breaks. If the update is "decrement inventory," the warehouse is now short two units and nobody knows why.

Delivery guarantees start with the sender's retry policy

Before writing any handler code, the receiver needs to read the sender's retry documentation the way a contract gets read. The relevant questions are concrete. How many attempts? Over what window? Does the sender back off exponentially or at fixed intervals? Does a timeout count as a failure? Does a 4xx response stop retries while a 5xx keeps them going?

Those answers determine the worst case. A sender that retries for three days means a receiver returning 500 during a two-day outage will still get the backlog once it recovers. A sender that gives up after five attempts across twenty minutes means the backlog is gone, and the receiver needs a reconciliation job to find what it missed. Both designs are legitimate. Only one matches the handler that was actually written.

The response code matters more than most implementers expect. Returning 200 on a request that failed internally tells the sender the event was handled, so the retry never comes. Returning 500 on a request that actually succeeded triggers a duplicate. The status code is the entire communication channel back to the sender, and it needs to reflect the true outcome of processing, not the fact that the HTTP server is alive.

The endpoint has to survive being called twice

Retries are the sender's attempt to guarantee at-least-once delivery, and at-least-once means duplicates are normal traffic. The receiver's job is to make handling idempotent: processing the same event twice leaves the system in the same state as processing it once.

The cleanest mechanism is the event ID. Most providers include a unique identifier in the payload or a header, and the receiver stores it in a table with a unique constraint. The handler inserts the ID inside the same transaction that applies the event's effects. A duplicate insert violates the constraint, the transaction rolls back, and the handler returns 200 as if it had processed the event, because in the only sense that matters, it has.

Event IDs only work when the sender keeps them stable across retries. Some providers generate a fresh ID per attempt, which turns the deduplication table into noise. When that happens, the receiver has to deduplicate on business keys: the order ID plus the status transition, the payment reference plus the amount. That is harder and more fragile, and it is worth checking the provider's behavior before building on the assumption.

Idempotency also has a time dimension. A deduplication table that keeps IDs forever grows without bound. A table that drops IDs after twenty-four hours reopens the door to a retry arriving on day two. The retention window should match the sender's retry window with margin. Three days of retries means keeping deduplication records for at least a week.

Arrival order carries no information

A customer updates a shipping address, then cancels the order. The sender fires two webhooks. The address update arrives first, the cancellation second, and everything is consistent. On a busy day, the cancellation arrives first, and the receiver writes the old address over the cancelled order, or worse, resurrects a cancelled order when the address update lands.

Webhook delivery is not ordered. Providers make a best effort, and best effort fails whenever retries, parallel workers, or multiple sender nodes are involved. Any handler that assumes the latest event reflects the latest state will eventually be wrong.

The fix is to include a state version or timestamp in the payload and reject updates that are older than what the receiver already holds. When the payload has no version, the receiver can fetch the current state from the sender's API and apply that instead of the payload's snapshot, which trades ordering problems for an extra request but produces the right answer.

Sequence numbers, when the provider offers them, are worth using directly. A handler that stores the last sequence number per object and ignores anything lower is short, easy to test, and correct under the retry and reordering behavior that real providers exhibit.

Bodies cannot be trusted without verification

A webhook endpoint is a public URL that accepts POST requests and writes to the database. Without verification, anyone who learns the URL can forge events. The URL will appear in logs, in browser dev tools, in a support ticket with a screenshot, and in a misconfigured proxy somewhere.

Signed payloads are the standard defense. The provider computes an HMAC over the raw request body using a shared secret and sends the digest in a header. The receiver recomputes the digest and compares. Two implementation details cause most of the failures: the comparison must operate on the raw bytes before any JSON parsing, and the comparison must be constant-time to avoid leaking information through timing.

Timestamp checks matter as much as the signature. A valid signature on a request captured six months ago is still a valid signature. Providers that include a timestamp in the signed material expect the receiver to reject anything older than a few minutes, which closes the replay window to something small. Receivers that skip the timestamp check have a signature scheme that protects against forgery but not against replay.

The secret itself needs rotation support. A provider that lets two secrets be active at once, old and new, gives the receiver a window to deploy the new secret without dropping events signed with the old one. Receivers that hardcode a single secret end up with a manual, error-prone cutover every time the secret needs to change.

The handler should acknowledge quickly and work later

Providers wait a few seconds for a response before treating the request as failed. A handler that calls three downstream services, writes to two databases, and sends an email will occasionally exceed that window under load, and every timeout becomes a retry.

The standard shape is to verify the signature, write the raw event to a durable queue or table, and return 200. A separate worker drains the queue and does the actual processing. This decouples the acknowledgment from the work, so a slow downstream service delays processing instead of triggering retries.

The trade-off is that the queue becomes the durability boundary. If the receiver writes to an in-memory queue and crashes before the worker picks up the event, the event is lost, and the provider has already been told the event succeeded. The write to the queue has to be durable, which usually means a database insert or a message broker with persistence enabled. An in-process channel is a lossy pipe dressed as a queue.

Retries arrive during incidents, not after them

The retry backlog is worst exactly when the receiver is least able to handle it. A database failover takes the handler down for ten minutes, during which every event queues up at the provider. When the handler recovers, the backlog arrives at once, and the first thing it does is overwhelm the freshly recovered database, causing another failure and another backlog.

Rate limiting on the receiving side helps. A worker pool with a bounded concurrency, fed from the durable queue, processes the backlog at a sustainable rate instead of at the speed the provider can deliver. The queue absorbs the burst, and the workers drain it at a pace the downstream systems can tolerate.

Backoff on the sender's side interacts with this. A provider that backs off exponentially spreads the retries out, which gives the receiver room to recover. A provider that retries at fixed intervals hammers the receiver at a steady rate throughout the outage and then dumps the backlog the moment it comes back. Knowing which behavior to expect shapes the queue depth and worker count the receiver needs.

Failures need a place to go and a way back

Some events will fail permanently. A payload references an order that was hard-deleted, or a customer whose record was merged into another. Retrying forever does not help, and dropping the event silently loses whatever action it was supposed to trigger.

A dead-letter queue gives failed events somewhere to sit while a person investigates. The queue should capture the raw payload, the failure reason, the attempt count, and the timestamp, so that replaying the event later does not require reconstructing it from logs. A replay tool that reads from the dead-letter queue and re-enqueues events is the difference between a recoverable incident and a data loss event with a manual fix per record.

Alerting needs to fire on the dead-letter queue depth and on the age of the oldest event in it. A queue that grows slowly over weeks is a queue nobody is watching, and by the time someone notices, the events are too old to replay against current state. The alert threshold should be set at the point where the events are still recent enough to matter.

Reconciliation catches what webhooks miss

Every sender eventually drops an event, whether from a bug, a misconfigured retry policy, or a receiver that returned 200 during a partial failure. Webhooks are a notification mechanism, not a source of truth, and the receiver's data will drift from the sender's over time.

A periodic reconciliation job that lists the sender's objects and compares them against the receiver's records finds the drift. It runs daily or hourly depending on how much staleness the business can tolerate, and it repairs mismatches by fetching the current state and applying it. The reconciliation path also serves as a backfill mechanism when the dead-letter queue is too large to replay by hand.

Building reconciliation is unglamorous, and teams routinely skip it because the webhook path appears to work. The drift shows up months later as a support ticket about a customer whose subscription is active in one system and cancelled in another, and the investigation always ends at the same place: an event that was never delivered, and no mechanism that would have noticed.

The three-day retry window is not a promise that events arrive. It is a promise that most events arrive, most of the time, and that the ones that do not will need to be found by something else. The teams that build webhook receivers that survive are the ones that read that sentence as a design brief and wrote the reconciliation job before the first support ticket.