How Do I Recover Missed Webhook Events?

October 28, 2026 · 5 min read · By Michael Lip

Short answer. Missed events come back from three places: the provider's resend control, a backfill call to the provider API, or your own reconciliation job. Write down the gap window from the delivery log first, then replay into an endpoint that drops duplicates by event ID. Replay without a dedupe store and you trade a gap for a double charge.

Updated 2026-10-28

A missed event is not an error in your logs. It is a gap between two records: the provider says the event happened, your system has no trace of it, or your system has a trace but the work behind it never ran. Recovery means closing that gap without creating a second copy of the work.

Why events go missing

Four causes cover most incidents. The endpoint was down for longer than the provider's retry window, so the last attempt failed and nothing was left to retry. The handler returned an error after it had already written to the database. The signature check rejected the request because the secret had been rotated. Or the endpoint did not exist yet, because the first deploy went out after the event was sent.

The first case loses the event. The other three deliver it and discard it, which is why a replay can quietly do the work twice.

Step 1: Define the gap window

Open the provider's delivery log and find two timestamps: the last delivery that reached your endpoint with a 2xx response, and the first delivery that exhausted its attempts. Use the provider's clock, not your own, because the two drift and your log may start after the event.

Write the window down before you touch anything: start, end, event types, and the count the provider reports. That count is the target for every later check. The simulated dashboard in InvokeBot's monitor tool shows the same fields you need to read in a real log: status per attempt, response code, and latency.

Step 2: Choose the recovery source

SourceWhen it worksWhat to watch
Resend from the delivery logThe provider still lists the failed deliveriesOne event at a time, so it is slow for a long outage
Backfill from the events APIThe provider exposes an events list with a time filterPagination, rate limits, and events created after your window
Your own reconciliationThe provider has no resend controlYou need a list endpoint on your side to compare against

Stripe keeps failed deliveries and offers a resend action in its webhook log, and its Events API can list past events for a backfill. GitHub lets you redeliver a failed delivery from the delivery log. Both providers are described in the monitor page FAQ. When neither control exists, reconciliation is the only path.

Step 3: Make the replay safe

Every replay passes through the same handler as a live delivery, so the handler needs a duplicate check that survives a restart. Store the provider event ID before the side effect runs, and return success when the ID is already present. A short script that replays a window into the same handler looks like this:

def handle(event):
    key = "webhook:seen:" + event["id"]
    if store.set(key, "1", nx=True, ex=7 * 24 * 3600) is None:
        return "duplicate"          # already processed, stop here
    apply_side_effect(event)        # charge, email, ticket update
    return "ok"

for event in provider.events(created_gte=window_start, created_lte=window_end):
    handle(event)

The expiry on the key has to outlive the provider's retry window, otherwise a late retry arrives after the key disappears and the work runs twice. The same rule is why idempotency keys are worth using from the first deploy rather than after the first incident.

Step 4: Reconcile before you close the incident

Pull the provider's event IDs for the window into one set. Pull the IDs your system marked as processed into another. The difference is the real gap, and it is usually smaller than the number of failed deliveries suggests, because some retries did land.

Run the comparison twice: once before the replay, to size the job, and once after, to prove it is empty. A non-empty difference after a replay means the handler is rejecting events, not that the replay failed.

Step 5: Stop the next gap

Frequently Asked Questions

Can a replay deliver the same event twice?

Yes, and that is expected. Replay is a manual delivery with the same guarantees as an automatic retry, which means at-least-once. The duplicate check in the handler is what makes the second delivery harmless, so the store must be checked before the side effect and not after it.

What if the provider has no resend control?

Use the provider's events API if it lists past events with a time filter, and page through the window. If there is no list endpoint either, reconciliation is impossible from the provider side, and the only recovery is a manual export or support request. Add a local record of received event IDs so a future outage is at least measurable.

How long should I keep the processed event IDs?

Longer than the provider's retry window plus your longest replay. Providers retry for hours to days, and an incident replay can happen a week later, so a store that expires in 24 hours is too short for recovery work. Size the store from the retry window of the provider that sends the most events.

Frequently Asked Questions

How do I verify a webhook signature with HMAC-SHA256?

Compute HMAC-SHA256 of the raw request body using your webhook secret, then compare the hex digest to the signature header. In Node.js: const sig = crypto.createHmac('sha256', secret).update(rawBody).digest('hex'). Use crypto.timingSafeEqual() for comparison to prevent timing attacks. Always hash the raw bytes, never parsed JSON.

What are the most common webhook security vulnerabilities?

The top vulnerabilities are: missing signature verification (accepting any request), using parsed JSON instead of raw body for HMAC (breaks signatures), string comparison instead of constant-time comparison (enables timing attacks), no timestamp validation (allows replay attacks), missing TLS (payloads visible in transit), and overly permissive IP allowlists.

Should I restrict webhook endpoints by IP address?

IP allowlisting is a defense-in-depth measure but should not be your only protection. Major providers publish their webhook source IPs. IP allowlisting blocks spoofed requests before they reach your application code, but HMAC signature verification is still required because IPs can be spoofed at the network level.

How do I prevent webhook replay attacks?

Include a timestamp in the signed payload and reject requests older than 5 minutes. Store processed event IDs and reject duplicates. Together, timestamp validation and idempotency keys prevent both replay attacks and duplicate processing from legitimate retries.

What should my webhook endpoint return on failure?

Return 200 for successfully processed events, 202 for accepted-but-queued events, 400 for malformed payloads, 401 for invalid signatures, and 500 for internal processing errors. Never return detailed error messages — they leak implementation details to attackers. The status code alone tells the provider whether to retry.

Related Tools

By the same builder: GitHub — theluckystrike BeLikeNative — Grammar AI EarlyThunder — Dev Blog Bug Bounty Reality Zovo — AI Dev Tools