How Do I Recover Missed Webhook Events?
Short answer. Missed events come back from three places: the provider's resend control, a backfill call to the provider API, or your own reconciliation job. Write down the gap window from the delivery log first, then replay into an endpoint that drops duplicates by event ID. Replay without a dedupe store and you trade a gap for a double charge.
Updated 2026-10-28
A missed event is not an error in your logs. It is a gap between two records: the provider says the event happened, your system has no trace of it, or your system has a trace but the work behind it never ran. Recovery means closing that gap without creating a second copy of the work.
Why events go missing
Four causes cover most incidents. The endpoint was down for longer than the provider's retry window, so the last attempt failed and nothing was left to retry. The handler returned an error after it had already written to the database. The signature check rejected the request because the secret had been rotated. Or the endpoint did not exist yet, because the first deploy went out after the event was sent.
The first case loses the event. The other three deliver it and discard it, which is why a replay can quietly do the work twice.
Step 1: Define the gap window
Open the provider's delivery log and find two timestamps: the last delivery that reached your endpoint with a 2xx response, and the first delivery that exhausted its attempts. Use the provider's clock, not your own, because the two drift and your log may start after the event.
Write the window down before you touch anything: start, end, event types, and the count the provider reports. That count is the target for every later check. The simulated dashboard in InvokeBot's monitor tool shows the same fields you need to read in a real log: status per attempt, response code, and latency.
Step 2: Choose the recovery source
| Source | When it works | What to watch |
|---|---|---|
| Resend from the delivery log | The provider still lists the failed deliveries | One event at a time, so it is slow for a long outage |
| Backfill from the events API | The provider exposes an events list with a time filter | Pagination, rate limits, and events created after your window |
| Your own reconciliation | The provider has no resend control | You need a list endpoint on your side to compare against |
Stripe keeps failed deliveries and offers a resend action in its webhook log, and its Events API can list past events for a backfill. GitHub lets you redeliver a failed delivery from the delivery log. Both providers are described in the monitor page FAQ. When neither control exists, reconciliation is the only path.
Step 3: Make the replay safe
Every replay passes through the same handler as a live delivery, so the handler needs a duplicate check that survives a restart. Store the provider event ID before the side effect runs, and return success when the ID is already present. A short script that replays a window into the same handler looks like this:
def handle(event):
key = "webhook:seen:" + event["id"]
if store.set(key, "1", nx=True, ex=7 * 24 * 3600) is None:
return "duplicate" # already processed, stop here
apply_side_effect(event) # charge, email, ticket update
return "ok"
for event in provider.events(created_gte=window_start, created_lte=window_end):
handle(event)
The expiry on the key has to outlive the provider's retry window, otherwise a late retry arrives after the key disappears and the work runs twice. The same rule is why idempotency keys are worth using from the first deploy rather than after the first incident.
Step 4: Reconcile before you close the incident
Pull the provider's event IDs for the window into one set. Pull the IDs your system marked as processed into another. The difference is the real gap, and it is usually smaller than the number of failed deliveries suggests, because some retries did land.
Run the comparison twice: once before the replay, to size the job, and once after, to prove it is empty. A non-empty difference after a replay means the handler is rejecting events, not that the replay failed.
Step 5: Stop the next gap
- Return 2xx before the work starts, then process from a queue. A handler that answers after the database write will time out under load.
- Send events that keep failing to a dead letter queue, and alert on the queue depth rather than on individual failures.
- Alert on a drop in delivery success rate, not only on a 5xx rate. A provider-side failure looks like silence on your side.
- Keep the delivery log longer than your longest incident. A log that expires in 24 hours cannot recover a weekend outage.
Frequently Asked Questions
Can a replay deliver the same event twice?
Yes, and that is expected. Replay is a manual delivery with the same guarantees as an automatic retry, which means at-least-once. The duplicate check in the handler is what makes the second delivery harmless, so the store must be checked before the side effect and not after it.
What if the provider has no resend control?
Use the provider's events API if it lists past events with a time filter, and page through the window. If there is no list endpoint either, reconciliation is impossible from the provider side, and the only recovery is a manual export or support request. Add a local record of received event IDs so a future outage is at least measurable.
How long should I keep the processed event IDs?
Longer than the provider's retry window plus your longest replay. Providers retry for hours to days, and an incident replay can happen a week later, so a store that expires in 24 hours is too short for recovery work. Size the store from the retry window of the provider that sends the most events.