How Do I Rotate a Webhook Signing Secret Without Dropping Events?

October 28, 2026 · 5 min read · By Michael Lip

Short answer. Rotate in three steps: add the new secret beside the old one, deploy a verifier that accepts either signature, then remove the old secret after the provider's retry window passes. The overlap is what keeps events flowing. Swap the secret in a single step and every request signed during the changeover is rejected.

Updated 2026-10-28

A webhook secret is the key to a signature check. Replace it and the next request the provider signs uses the new key while your endpoint still checks the old one. The request looks forged, gets a 401, and the provider retries it until its attempt budget runs out. A rotation plan exists to make that window impossible.

Why a single-step swap loses events

Two clocks are running during a secret change. Your deploy takes time to reach every instance. The provider keeps signing with whichever secret is active on its side. If you change the secret in the provider dashboard and deploy the new verifier at the same moment, there is still a gap: the old instances accept only the old key while the provider signs only with the new one.

The gap is small when you run one instance and short deploys. It becomes minutes when you run several instances, a rolling deploy, or a serverless platform that keeps old code warm. Every request in that gap is signed with a key the receiving instance rejects.

The dual-secret pattern

Accept two secrets at once. Verify the signature against the primary secret first. If that fails, verify against the secondary secret. Record which one matched so you can see how much traffic still uses the old key.

def verify(raw_body, header, secrets):
    for name, secret in secrets.items():
        expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
        if hmac.compare_digest(expected, header):
            metrics.increment("webhook.signature_ok", tags={"secret": name})
            return name
    return None

secrets = {
    "new": os.environ["WEBHOOK_SECRET_NEW"],
    "old": os.environ["WEBHOOK_SECRET_OLD"],
}

Both comparisons run in constant time. The order matters only for the metric: checking the new secret first tells you when the provider has switched. Once the old secret sees no valid signatures for longer than the provider's retry window, the old value can leave the environment.

Step 1: Add the new secret without changing the check

Deploy a build that accepts both secrets while both are still valid. This deploy does not change behaviour: every request that passed before still passes. Do this before you touch the provider dashboard, because the provider cannot sign with a key your code does not know.

Keep the new secret in the same secret store as the old one. Do not put either value in a config file in the repository. The same rule applies to the signature header name: if the provider sends a version prefix such as v1=, the verifier must parse it the same way for both secrets.

Step 2: Switch the provider to the new secret

Change the secret in the provider's webhook settings. The provider starts signing new deliveries with the new key. Your dual-secret verifier accepts them, and the metric for the new secret starts to rise.

Some providers support more than one active signing secret during a rotation. If yours does, add the new secret and keep the old one active until the metric shows no old-key traffic. If yours replaces the secret in one action, the dual-secret verifier is what covers the gap.

Step 3: Watch the old-key metric, not the clock

The safe moment to remove the old secret is when no valid signature has matched it for longer than the provider's retry window. A timer alone is a guess. The retry window matters because a delivery signed with the old secret before the switch may still be retrying, and each retry is signed again with the secret active at that moment.

CheckPass condition
New-secret matchesRising, then steady at all traffic
Old-secret matchesZero for longer than the retry window
401 rateNo step change at the switch
Failed deliveriesNo new exhausted attempts in the window

Step 4: Remove the old secret and prove the change

Take the old secret out of the secret store, redeploy, and send one signed test request from the provider's test control. A 200 confirms the new secret alone is enough. A 401 means the old key is still in use somewhere, so check every instance and the environment of each deploy target before closing the change.

The webhook signature verifier recomputes a signature from a raw body and secret in the browser, which is enough to confirm the value you expect without sending a live request. The webhook security checklist covers the rest of the verification path, including the raw-body and constant-time details that a rotation can expose.

Mistakes that cost events

Frequently Asked Questions

Can I rotate a webhook secret with zero downtime?

Yes, if the verifier accepts both the old and the new secret during the changeover. Deploy the dual-secret check first, switch the provider second, and remove the old secret only after no valid signature has matched it for longer than the retry window. Without the overlap there is always a gap between the provider switch and the last old instance.

What if the provider only allows one active secret?

Keep both values in your own verifier during the changeover. The provider replaces its secret in one action, so the old key cannot be active on its side, but deliveries signed just before the switch may still be retrying. Your endpoint accepts both until those retries stop, which covers the gap the provider cannot.

How long should both secrets stay valid?

Longer than the provider's retry window, not a fixed number of hours. Watch the metric for valid signatures that match the old secret and remove it only when that count has been zero for the full window. A provider that retries for three days needs an overlap measured in days, not minutes.

Frequently Asked Questions

How do I verify a webhook signature with HMAC-SHA256?

Compute HMAC-SHA256 of the raw request body using your webhook secret, then compare the hex digest to the signature header. In Node.js: const sig = crypto.createHmac('sha256', secret).update(rawBody).digest('hex'). Use crypto.timingSafeEqual() for comparison to prevent timing attacks. Always hash the raw bytes, never parsed JSON.

What are the most common webhook security vulnerabilities?

The top vulnerabilities are: missing signature verification (accepting any request), using parsed JSON instead of raw body for HMAC (breaks signatures), string comparison instead of constant-time comparison (enables timing attacks), no timestamp validation (allows replay attacks), missing TLS (payloads visible in transit), and overly permissive IP allowlists.

Should I restrict webhook endpoints by IP address?

IP allowlisting is a defense-in-depth measure but should not be your only protection. Major providers publish their webhook source IPs. IP allowlisting blocks spoofed requests before they reach your application code, but HMAC signature verification is still required because IPs can be spoofed at the network level.

How do I prevent webhook replay attacks?

Include a timestamp in the signed payload and reject requests older than 5 minutes. Store processed event IDs and reject duplicates. Together, timestamp validation and idempotency keys prevent both replay attacks and duplicate processing from legitimate retries.

What should my webhook endpoint return on failure?

Return 200 for successfully processed events, 202 for accepted-but-queued events, 400 for malformed payloads, 401 for invalid signatures, and 500 for internal processing errors. Never return detailed error messages — they leak implementation details to attackers. The status code alone tells the provider whether to retry.

Related Tools

By the same builder: GitHub — theluckystrike BeLikeNative — Grammar AI EarlyThunder — Dev Blog Bug Bounty Reality Zovo — AI Dev Tools