Webhook providers retry. Events can arrive late, more than once, or out of order. A handler that sends an email, updates a record, and then writes “processed” can repeat the first two actions after a timeout.
We design the receiver around durable event identity and business state. Returning a 2xx response means the receiver has accepted responsibility for the event—not necessarily that every downstream side effect finished inside the request.
reference flow
The implementation sequence.
- 1
Verify raw request
Use the provider’s current signature method, timestamp tolerance, and secret handling.
- 2
Claim event
Insert the provider event ID or business idempotency key under a unique constraint.
- 3
Load authority
Retrieve the canonical provider object when the event alone is insufficient or unordered.
- 4
Transition state
Apply one allowed transition in a transaction and record the decision.
- 5
Run side effects
Queue notifications and external updates with their own idempotency keys.
- 6
Reconcile
Compare local pending/final states with provider truth on a schedule and repair or flag drift.
01 method
Separate event identity, object identity, and business identity.
A provider event ID answers whether this delivery was seen. A provider object ID identifies the checkout, invoice, or transaction. A business reference identifies the local order, submission, or account. We store each in its own constrained field instead of using an email address or mutable metadata as the join key.
The durable claim should be atomic. If two deliveries arrive together, one insert wins and the other can return the already-accepted result. A process-local set is not enough because it disappears on restart and cannot coordinate more than one instance.
| State | Meaning | Allowed next action |
|---|---|---|
| received | Signature valid; event claimed durably | Load or verify authoritative object |
| applied | Business transition committed | Queue independent side effects |
| completed | Required side effects confirmed | Retain audit context |
| retryable | Temporary dependency failure | Retry with bounded backoff |
| review | Conflict, unknown mapping, or non-retryable failure | Human or documented repair path |
| duplicate | Claim already exists | Return accepted result without repeating transition |
02 method
Make every side effect independently repeatable.
A payment-state update, CRM write, receipt, and internal notification are different side effects. Each should have an idempotency key tied to the business transition and a recorded result. That lets the system retry a failed CRM update without sending the receipt again.
Provider API idempotency support can reduce duplicate write risk, but it does not replace the local state machine. The integration must still decide what a completed business transition means and how conflicts are surfaced.
Implementation checklist
- Preserve the raw body required by signature verification.
- Use a unique database constraint for event claims.
- Record provider object and local business references separately.
- Model allowed state transitions and terminal states.
- Give each external side effect its own idempotency key.
- Run reconciliation for stuck, missing, and conflicting states.
03 method
Test retries, crashes, and ordering—not only valid payloads.
Our failure tests deliver the same event concurrently, replay it after restart, stop the process after the local commit but before the response, fail each side effect independently, send a later state before an earlier event, and make the provider lookup temporarily unavailable.
Vendor details control the exact verification and retry behavior. For Stripe, for example, the webhook guidance requires signature verification against the raw request body, while its API documents idempotency keys for supported requests. Current vendor documentation remains the implementation authority.
primary references
Sources used for this method.
Vendor and standards documentation can change. Confirm the current version and the requirements that apply to your organization before implementation.
related work