How Do I Rotate a Webhook Signing Secret Without Dropping Events?
Short answer. Rotate in three steps: add the new secret beside the old one, deploy a verifier that accepts either signature, then remove the old secret after the provider's retry window passes. The overlap is what keeps events flowing. Swap the secret in a single step and every request signed during the changeover is rejected.
Updated 2026-10-28
A webhook secret is the key to a signature check. Replace it and the next request the provider signs uses the new key while your endpoint still checks the old one. The request looks forged, gets a 401, and the provider retries it until its attempt budget runs out. A rotation plan exists to make that window impossible.
Why a single-step swap loses events
Two clocks are running during a secret change. Your deploy takes time to reach every instance. The provider keeps signing with whichever secret is active on its side. If you change the secret in the provider dashboard and deploy the new verifier at the same moment, there is still a gap: the old instances accept only the old key while the provider signs only with the new one.
The gap is small when you run one instance and short deploys. It becomes minutes when you run several instances, a rolling deploy, or a serverless platform that keeps old code warm. Every request in that gap is signed with a key the receiving instance rejects.
The dual-secret pattern
Accept two secrets at once. Verify the signature against the primary secret first. If that fails, verify against the secondary secret. Record which one matched so you can see how much traffic still uses the old key.
def verify(raw_body, header, secrets):
for name, secret in secrets.items():
expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
if hmac.compare_digest(expected, header):
metrics.increment("webhook.signature_ok", tags={"secret": name})
return name
return None
secrets = {
"new": os.environ["WEBHOOK_SECRET_NEW"],
"old": os.environ["WEBHOOK_SECRET_OLD"],
}
Both comparisons run in constant time. The order matters only for the metric: checking the new secret first tells you when the provider has switched. Once the old secret sees no valid signatures for longer than the provider's retry window, the old value can leave the environment.
Step 1: Add the new secret without changing the check
Deploy a build that accepts both secrets while both are still valid. This deploy does not change behaviour: every request that passed before still passes. Do this before you touch the provider dashboard, because the provider cannot sign with a key your code does not know.
Keep the new secret in the same secret store as the old one. Do not put either value in a config file in the repository. The same rule applies to the signature header name: if the provider sends a version prefix such as v1=, the verifier must parse it the same way for both secrets.
Step 2: Switch the provider to the new secret
Change the secret in the provider's webhook settings. The provider starts signing new deliveries with the new key. Your dual-secret verifier accepts them, and the metric for the new secret starts to rise.
Some providers support more than one active signing secret during a rotation. If yours does, add the new secret and keep the old one active until the metric shows no old-key traffic. If yours replaces the secret in one action, the dual-secret verifier is what covers the gap.
Step 3: Watch the old-key metric, not the clock
The safe moment to remove the old secret is when no valid signature has matched it for longer than the provider's retry window. A timer alone is a guess. The retry window matters because a delivery signed with the old secret before the switch may still be retrying, and each retry is signed again with the secret active at that moment.
| Check | Pass condition |
|---|---|
| New-secret matches | Rising, then steady at all traffic |
| Old-secret matches | Zero for longer than the retry window |
| 401 rate | No step change at the switch |
| Failed deliveries | No new exhausted attempts in the window |
Step 4: Remove the old secret and prove the change
Take the old secret out of the secret store, redeploy, and send one signed test request from the provider's test control. A 200 confirms the new secret alone is enough. A 401 means the old key is still in use somewhere, so check every instance and the environment of each deploy target before closing the change.
The webhook signature verifier recomputes a signature from a raw body and secret in the browser, which is enough to confirm the value you expect without sending a live request. The webhook security checklist covers the rest of the verification path, including the raw-body and constant-time details that a rotation can expose.
Mistakes that cost events
- Rotating on a Friday without a rollback plan. A failed rotation is an outage that lasts until someone is awake.
- Storing the new secret in a config file. The value lands in git history and the rotation has to happen again.
- Removing the old secret as soon as the deploy finishes. Retries signed before the switch can still arrive.
- Comparing signatures with a normal string equality. A timing side channel is present in both keys, so the rotation does not remove it.
- Forgetting that some providers sign a timestamp plus the body. A new secret changes the key, not the signed string.
Frequently Asked Questions
Can I rotate a webhook secret with zero downtime?
Yes, if the verifier accepts both the old and the new secret during the changeover. Deploy the dual-secret check first, switch the provider second, and remove the old secret only after no valid signature has matched it for longer than the retry window. Without the overlap there is always a gap between the provider switch and the last old instance.
What if the provider only allows one active secret?
Keep both values in your own verifier during the changeover. The provider replaces its secret in one action, so the old key cannot be active on its side, but deliveries signed just before the switch may still be retrying. Your endpoint accepts both until those retries stop, which covers the gap the provider cannot.
How long should both secrets stay valid?
Longer than the provider's retry window, not a fixed number of hours. Watch the metric for valid signatures that match the old secret and remove it only when that count has been zero for the full window. A provider that retries for three days needs an overlap measured in days, not minutes.