Changing a key while everything is using it

Rotation is a procedure, not a value

Everybody knows keys should be rotated. Far fewer teams have done it, because the naive version (replace the value, restart, hope) takes the site down when something you had forgotten was still using the old one.

The result is that keys quietly become permanent. A credential shared with a contractor in 2023 still works. The database password is in four places, one of which is a cron job nobody has opened. And when a leak forces a rotation, it happens under pressure, badly.

The fix is to make rotation a rehearsed, dull procedure. It has one core idea: for a period, both keys are valid.

The overlap window

1. Create the new key alongside the old one. Both work.
2. Deploy the new key to everything that uses it. This is where you discover the four places, and the discovery is safe because the old key still works.
3. Watch until the old key shows no usage. Most providers expose last-used timestamps or usage-per-key metrics; if yours does not, log which key ID your own code used.
4. Disable, do not delete, the old key. Reversible for a week.
5. Delete it once nothing has broken.

Step 3 is the one people skip, and it is the only one that turns rotation from a hope into a fact.

Give your own keys an identifier

For keys you issue (session signing keys, webhook signatures, API keys for your customers) put a key ID in the artifact itself. A JWT header carries kid; a webhook header can carry a key version; your own API keys can be prefixed, as in ak_live_2_xxxxx.

With an ID, verification becomes: look up this key by ID, in a small set of currently valid keys. Rotation becomes: add a key, start signing with it, keep verifying with both, remove the old one when nothing presents it. Without an ID, you must try every key on every request, which works but tells you nothing about who is still on the old one.

The different things people mean by rotate

• A provider’s API key. Usually easy: most providers let you have two active keys precisely so you can overlap. Check before you assume.
• A database password. Create a second user with the same grants, migrate, drop the first. Changing the password on a single user in place has no overlap window and will drop live connections.
• A signing key. Needs the key ID design above, plus a verification period long enough to cover the longest-lived token you have issued.
• A cloud provider’s long-lived credentials. The real answer is to stop having them: short-lived credentials issued to a workload identity expire on their own, and rotation stops being an event. This is available on every major cloud and in GitHub Actions via OIDC, and it removes the most dangerous class of secret entirely.
• A key you have just leaked. Different procedure: no overlap, revoke first, apologise second. The leak lesson earlier in this course covers why scrubbing history is not a substitute.

Write the runbook while it is calm

One page per credential type, in the repository, listing: where the key lives, everything that consumes it, the exact commands, how to verify the new one works, and how to roll back. Ten lines each.

Then run one rotation per quarter, deliberately, on a Tuesday morning. Not because the key is compromised, but because the run finds the forgotten consumer while nothing is wrong. This is the practising-failure lesson applied to credentials, and it is the cheapest drill on that list.

What good looks like

• Every secret has an owner, an expiry date and a documented rotation procedure.
• No credential is more than a year old, and anything that touched a laptop that left the company is already rotated.
• Rotation is a fifteen-minute task somebody other than the author can perform from the runbook.
• Nothing important depends on a key that cannot be overlapped; and where one does, you know which one it is, and that outage is planned rather than discovered.