Webhook Idempotency: How to Actually Verify It Works

Published 2026-09-15

Every major webhook sender — Stripe, GitHub, Shopify, and the rest — delivers at least once, not exactly once. Retries aren't an edge case you might hit; they're the documented contract. If your handler isn't idempotent, you will eventually process the same event twice. The part that catches teams out isn't knowing this. It's that a handler can pass an obvious idempotency test and still fail in production.

Why retries are guaranteed, not occasional

Some senders retry automatically. Stripe and Shopify both resend on timeouts, non-2xx responses, and connection failures they can't distinguish from "the request never arrived" — from the sender's side, silence looks the same as a lost request, so the only safe default is to resend. GitHub is different: it doesn't automatically redeliver a failed webhook at all. Redelivery there only happens if you or a script triggers it from the delivery history. That still means duplicates reach you in practice — someone replays a failed delivery from the UI, or a retry script sweeps failures on a schedule — just on a human or scripted timeline instead of an automatic one.

Either way, you're handed an identifier meant for deduplication: Stripe puts a stable id on the event object, and GitHub's X-GitHub-Delivery header stays the same GUID across a redelivery of the same original delivery. The sender is assuming you'll use it. Deduplicating isn't optional hardening — it's the other half of the contract retries (automatic or not) create.

The naive implementation

Most teams get this far on the first pass: extract the delivery ID, check whether it's been seen before, skip processing if so, otherwise process and record the ID.

if (seenIds.has(deliveryId)) {
  return res.status(200).end(); // already processed, skip
}
seenIds.add(deliveryId);
applyEvent(payload);

Send the same request twice in a row with curl, and this looks correct — the second call returns early, nothing gets applied twice. That's exactly the test most people run, and it's exactly the test that doesn't find the bug.

The bug a sequential test can't find

The code above has a check-then-act race: reading seenIds and writing to it are two separate steps, not one atomic operation. If two deliveries with the same ID arrive close enough together — a real retry storm, or a sender's own retry firing before your first response lands — both requests can pass the seenIds.has(deliveryId) check before either one finishes recording it. Both proceed. You get the event applied twice — a payment credited twice, or a record inserted where there should only be one.

This only shows up under concurrent delivery, and a single sequential curl test — request, wait for response, request again — never produces concurrent delivery. It's not that the test was run carelessly; it's that the test wasn't capable of finding this class of bug at all.

A test that can actually find it

Fire two requests with the same delivery ID at the same time instead of one after another. With curl, backgrounding both and waiting is enough to create real overlap on a fast local handler:

curl -s -X POST https://your-endpoint/webhook \
  -H "Content-Type: application/json" \
  -H "X-Signature: sha256=<valid signature for BODY>" \
  -d "$BODY" &

curl -s -X POST https://your-endpoint/webhook \
  -H "Content-Type: application/json" \
  -H "X-Signature: sha256=<valid signature for BODY>" \
  -d "$BODY" &

wait

Then check the system of record, not the HTTP responses — both requests may well return 200. The question is whether the underlying effect (the row inserted, the balance changed, the email sent) happened once or twice. If what's actually running behind that endpoint is a sequential dedupe check — no database-level unique constraint, no atomic claim like INSERT ... ON CONFLICT DO NOTHING on the delivery ID, no atomic SETNX — running this a handful of times will usually surface at least one duplicate.

What this test doesn't cover

A concurrent-replay test like the one above checks one failure mode: duplicate processing under retry. It doesn't tell you whether events applied out of order get reordered correctly, or whether a stalled connection actually gets timed out instead of patiently completed. It says nothing about whether a truncated body gets rejected cleanly instead of 500ing, or whether a bad signature gets refused instead of quietly accepted. Those are separate, independently-failing checks — a handler can be correctly idempotent and still fail every one of the others.