{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"rotating-a-key-without-downtime","title":"Changing a key while everything is using it","url":"https://addaly.com/learn/cloud-and-shipping/rotating-a-key-without-downtime","position":61,"of":74,"minutes":8,"answer":"Rotate by overlapping (both keys valid, deploy the new one, watch the old one go quiet.","keyIdea":"Rotate by overlapping (both keys valid, deploy the new one, watch the old one go quiet, disable before deleting) and give keys an identifier so you can tell who is still using the old one instead of guessing."},"body":{"format":"markdown","text":"## Rotation is a procedure, not a value\n\nEverybody knows keys should be rotated. Far fewer teams have done it, because the naive version (replace the value, restart, hope) takes the site down when something you had forgotten was still using the old one.\n\nThe result is that keys quietly become permanent. A credential shared with a contractor in 2023 still works. The database password is in four places, one of which is a cron job nobody has opened. And when a leak forces a rotation, it happens under pressure, badly.\n\nThe fix is to make rotation a rehearsed, dull procedure. It has one core idea: **for a period, both keys are valid.**\n\n## The overlap window\n\n1. **Create the new key** alongside the old one. Both work.\n2. **Deploy the new key** to everything that uses it. This is where you discover the four places, and the discovery is safe because the old key still works.\n3. **Watch** until the old key shows no usage. Most providers expose last-used timestamps or usage-per-key metrics; if yours does not, log which key ID your own code used.\n4. **Disable, do not delete**, the old key. Reversible for a week.\n5. **Delete it** once nothing has broken.\n\nStep 3 is the one people skip, and it is the only one that turns rotation from a hope into a fact.\n\n> Figure: Rotation with an overlap window. Steps: 1. Create the new key (alongside the old one; both work); 2. Deploy the new key everywhere (this is where you find the four places it lives); 3. Watch the old key (last-used timestamps or your own key-ID log, until it shows no usage); 4. Disable the old key (not delete: reversible for a week); 5. Delete it (once nothing has broken). For a period both keys are valid, so the forgotten cron job is discovered safely rather than by an outage. Watching the old key go quiet is the step that turns rotation from a hope into a fact.\n\n## Give your own keys an identifier\n\nFor keys you issue (session signing keys, webhook signatures, API keys for your customers) put a key ID in the artifact itself. A JWT header carries `kid`; a webhook header can carry a key version; your own API keys can be prefixed, as in `ak_live_2_xxxxx`.\n\nWith an ID, verification becomes: look up this key by ID, in a small set of currently valid keys. Rotation becomes: add a key, start signing with it, keep verifying with both, remove the old one when nothing presents it. Without an ID, you must try every key on every request, which works but tells you nothing about who is still on the old one.\n\n## The different things people mean by rotate\n\n- **A provider's API key.** Usually easy: most providers let you have two active keys precisely so you can overlap. Check before you assume.\n- **A database password.** Create a second user with the same grants, migrate, drop the first. Changing the password on a single user in place has no overlap window and will drop live connections.\n- **A signing key.** Needs the key ID design above, plus a verification period long enough to cover the longest-lived token you have issued.\n- **A cloud provider's long-lived credentials.** The real answer is to stop having them: short-lived credentials issued to a workload identity expire on their own, and rotation stops being an event. This is available on every major cloud and in GitHub Actions via OIDC, and it removes the most dangerous class of secret entirely.\n- **A key you have just leaked.** Different procedure: no overlap, revoke first, apologise second. The leak lesson earlier in this course covers why scrubbing history is not a substitute.\n\n## Write the runbook while it is calm\n\nOne page per credential type, in the repository, listing: where the key lives, everything that consumes it, the exact commands, how to verify the new one works, and how to roll back. Ten lines each.\n\nThen run one rotation per quarter, deliberately, on a Tuesday morning. Not because the key is compromised, but because the run finds the forgotten consumer while nothing is wrong. This is the practising-failure lesson applied to credentials, and it is the cheapest drill on that list.\n\n## What good looks like\n\n- Every secret has an owner, an expiry date and a documented rotation procedure.\n- No credential is more than a year old, and anything that touched a laptop that left the company is already rotated.\n- Rotation is a fifteen-minute task somebody other than the author can perform from the runbook.\n- Nothing important depends on a key that cannot be overlapped; and where one does, you know which one it is, and that outage is planned rather than discovered."},"check":{"question":"A team rotates a webhook signing key by replacing the secret in both the sender and receiver during a deploy. Some webhooks are rejected for several minutes afterwards. What explains it?","options":[{"text":"Clock skew between sender and receiver invalidated the signature timestamps","correct":false,"tempting":"Skew does break timestamped signatures, but it would affect deliveries before and after the change rather than starting precisely at the deploy."},{"text":"The receiver cached the old key and needed a restart to pick up the new one","correct":false,"tempting":"A caching bug is possible, but the described failure occurs even with a correct restart because both sides cannot change at the same instant."},{"text":"Messages signed with the old key were still in flight or being retried when the receiver stopped accepting it","correct":true},{"text":"The new key was shorter, so signatures failed length validation","correct":false,"tempting":"Key length would fail consistently rather than for a few minutes, and it would break every delivery, not some."}]},"prev":{"slug":"dependencies-and-the-supply-chain","title":"The code you did not write","url":"https://addaly.com/learn/cloud-and-shipping/dependencies-and-the-supply-chain"},"next":{"slug":"bots-scrapers-and-floods","title":"Bots, scrapers and the flood you cannot absorb","url":"https://addaly.com/learn/cloud-and-shipping/bots-scrapers-and-floods"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Changing a key while everything is using it', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/rotating-a-key-without-downtime (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Changing a key while everything is using it","courseTitle":"Shipping It","module":{"index":7,"slug":"security-after-the-short-list","title":"Security after the short list"},"answer":null,"keyIdea":"Rotate by overlapping (both keys valid, deploy the new one, watch the old one go quiet, disable before deleting) and give keys an identifier so you can tell who is still using the old one instead of guessing.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Create the new key","detail":"alongside the old one; both work"},{"label":"Deploy the new key everywhere","detail":"this is where you find the four places it lives"},{"label":"Watch the old key","detail":"last-used timestamps or your own key-ID log, until it shows no usage"},{"label":"Disable the old key","detail":"not delete: reversible for a week"},{"label":"Delete it","detail":"once nothing has broken"}],"title":"Rotation with an overlap window","caption":"For a period both keys are valid, so the forgotten cron job is discovered safely rather than by an outage. Watching the old key go quiet is the step that turns rotation from a hope into a fact."},"alt":"Figure: Rotation with an overlap window. Steps: 1. Create the new key (alongside the old one; both work); 2. Deploy the new key everywhere (this is where you find the four places it lives); 3. Watch the old key (last-used timestamps or your own key-ID log, until it shows no usage); 4. Disable the old key (not delete: reversible for a week); 5. Delete it (once nothing has broken). For a period both keys are valid, so the forgotten cron job is discovered safely rather than by an outage. Watching the old key go quiet is the step that turns rotation from a hope into a fact."}},"blocks":[{"t":"h2","id":"rotation-is-a-procedure-not-a-value","runs":[{"text":"Rotation is a procedure, not a value"}]},{"t":"p","runs":[{"text":"Everybody knows keys should be rotated. Far fewer teams have done it, because the naive version (replace the value, restart, hope) takes the site down when something you had forgotten was still using the old one."}]},{"t":"p","runs":[{"text":"The result is that keys quietly become permanent. A credential shared with a contractor in 2023 still works. The database password is in four places, one of which is a cron job nobody has opened. And when a leak forces a rotation, it happens under pressure, badly."}]},{"t":"p","runs":[{"text":"The fix is to make rotation a rehearsed, dull procedure. It has one core idea: "},{"text":"for a period, both keys are valid.","b":true}]},{"t":"h2","id":"the-overlap-window","runs":[{"text":"The overlap window"}]},{"t":"ol","start":1,"items":[[{"text":"Create the new key","b":true},{"text":" alongside the old one. Both work."}],[{"text":"Deploy the new key","b":true},{"text":" to everything that uses it. This is where you discover the four places, and the discovery is safe because the old key still works."}],[{"text":"Watch","b":true},{"text":" until the old key shows no usage. Most providers expose last-used timestamps or usage-per-key metrics; if yours does not, log which key ID your own code used."}],[{"text":"Disable, do not delete","b":true},{"text":", the old key. Reversible for a week."}],[{"text":"Delete it","b":true},{"text":" once nothing has broken."}]]},{"t":"p","runs":[{"text":"Step 3 is the one people skip, and it is the only one that turns rotation from a hope into a fact."}]},{"t":"h2","id":"give-your-own-keys-an-identifier","runs":[{"text":"Give your own keys an identifier"}]},{"t":"p","runs":[{"text":"For keys you issue (session signing keys, webhook signatures, API keys for your customers) put a key ID in the artifact itself. A JWT header carries "},{"text":"kid","code":true},{"text":"; a webhook header can carry a key version; your own API keys can be prefixed, as in "},{"text":"ak_live_2_xxxxx","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"With an ID, verification becomes: look up this key by ID, in a small set of currently valid keys. Rotation becomes: add a key, start signing with it, keep verifying with both, remove the old one when nothing presents it. Without an ID, you must try every key on every request, which works but tells you nothing about who is still on the old one."}]},{"t":"h2","id":"the-different-things-people-mean-by-rotate","runs":[{"text":"The different things people mean by rotate"}]},{"t":"ul","items":[[{"text":"A provider’s API key.","b":true},{"text":" Usually easy: most providers let you have two active keys precisely so you can overlap. Check before you assume."}],[{"text":"A database password.","b":true},{"text":" Create a second user with the same grants, migrate, drop the first. Changing the password on a single user in place has no overlap window and will drop live connections."}],[{"text":"A signing key.","b":true},{"text":" Needs the key ID design above, plus a verification period long enough to cover the longest-lived token you have issued."}],[{"text":"A cloud provider’s long-lived credentials.","b":true},{"text":" The real answer is to stop having them: short-lived credentials issued to a workload identity expire on their own, and rotation stops being an event. This is available on every major cloud and in GitHub Actions via OIDC, and it removes the most dangerous class of secret entirely."}],[{"text":"A key you have just leaked.","b":true},{"text":" Different procedure: no overlap, revoke first, apologise second. The leak lesson earlier in this course covers why scrubbing history is not a substitute."}]]},{"t":"h2","id":"write-the-runbook-while-it-is-calm","runs":[{"text":"Write the runbook while it is calm"}]},{"t":"p","runs":[{"text":"One page per credential type, in the repository, listing: where the key lives, everything that consumes it, the exact commands, how to verify the new one works, and how to roll back. Ten lines each."}]},{"t":"p","runs":[{"text":"Then run one rotation per quarter, deliberately, on a Tuesday morning. Not because the key is compromised, but because the run finds the forgotten consumer while nothing is wrong. This is the practising-failure lesson applied to credentials, and it is the cheapest drill on that list."}]},{"t":"h2","id":"what-good-looks-like","runs":[{"text":"What good looks like"}]},{"t":"ul","items":[[{"text":"Every secret has an owner, an expiry date and a documented rotation procedure."}],[{"text":"No credential is more than a year old, and anything that touched a laptop that left the company is already rotated."}],[{"text":"Rotation is a fifteen-minute task somebody other than the author can perform from the runbook."}],[{"text":"Nothing important depends on a key that cannot be overlapped; and where one does, you know which one it is, and that outage is planned rather than discovered."}]]}],"check":{"question":[{"text":"A team rotates a webhook signing key by replacing the secret in both the sender and receiver during a deploy. Some webhooks are rejected for several minutes afterwards. What explains it?"}],"options":[{"text":[{"text":"Clock skew between sender and receiver invalidated the signature timestamps"}],"correct":false,"tempting":[{"text":"Skew does break timestamped signatures, but it would affect deliveries before and after the change rather than starting precisely at the deploy."}]},{"text":[{"text":"The receiver cached the old key and needed a restart to pick up the new one"}],"correct":false,"tempting":[{"text":"A caching bug is possible, but the described failure occurs even with a correct restart because both sides cannot change at the same instant."}]},{"text":[{"text":"Messages signed with the old key were still in flight or being retried when the receiver stopped accepting it"}],"correct":true,"tempting":null},{"text":[{"text":"The new key was shorter, so signatures failed length validation"}],"correct":false,"tempting":[{"text":"Key length would fail consistently rather than for a few minutes, and it would break every delivery, not some."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"dependencies-and-the-supply-chain","title":"The code you did not write","minutes":9},"next":{"slug":"bots-scrapers-and-floods","title":"Bots, scrapers and the flood you cannot absorb","minutes":9}}}