{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"rollback-and-the-kill-switch","title":"Undo, in under two minutes","url":"https://addaly.com/learn/cloud-and-shipping/rollback-and-the-kill-switch","position":19,"of":74,"minutes":9,"answer":"Code rolls back, data does not, so keep them separate and make sure the undo you rely on is one you have actually performed while nothing was wrong.","keyIdea":"Code rolls back, data does not, so keep them separate and make sure the undo you rely on is one you have actually performed while nothing was wrong."},"body":{"format":"markdown","text":"## The only question that matters at minute five\n\nSomething is broken and you deployed twenty minutes ago. The question is not \"what is wrong\". It is \"how do I make it stop while I find out\".\n\nTeams that answer that question in two minutes have incidents measured in minutes. Teams that start debugging first have incidents measured in hours, because the fire keeps burning while they read.\n\nDecide the rule in advance and write it down: **if the cause is not clear within ten minutes, roll back and diagnose from the artifact and the logs.** The evidence does not disappear when you roll back. Your users' patience does.\n\n## What is actually reversible\n\n- **Code: yes.** A previous build artifact still exists. This is why the last module insisted on building once and promoting the artifact: rollback becomes \"run the old image\", not \"rebuild an old commit and hope the dependency tree resolves the same way\".\n- **Schema: mostly no.** Covered in the last lesson. Expand/migrate/contract exists precisely so that the *code* rollback stays safe while the schema only moves forward.\n- **Data: no.** Anything users wrote, or your job wrote, in the twenty minutes the bad version was live is now there. If the bad version wrote wrong data, rollback stops the bleeding and you still have a repair job to write.\n\nKeeping these three separate is most of the skill.\n\n> Figure: What a rollback undoes, and what it does not. Reversible: Code: redeploy the previous build artifact; A feature behind a flag: flip a boolean, no deploy; Config: separate system, check it was reverted too. Not reversible: Data users or jobs wrote while the bad version was live; A schema already written to in the new shape; A cache full of the new object shape; A queue full of new-format messages; A webhook registered or a DNS record changed. Code rolls back because the previous artifact still exists. Everything the bad version wrote, told or cached stays where it is, and each of those needs its own repair.\n\n## Practise it while nothing is wrong\n\nA rollback you have never performed is a plan, not a capability. There is always something: the button is under a menu you have not opened, the old image was garbage-collected after seven days, the command needs a flag, the deploy key has expired.\n\nDo it deliberately on a quiet afternoon. Deploy, roll back, confirm the site is on the previous version, roll forward again. Write the exact command in the runbook. The whole exercise takes fifteen minutes and it is the highest-value fifteen minutes in this course.\n\nKnow the answers to these before you need them:\n\n- What is the exact command or button, and who has permission?\n- How far back can you go? How long are old artifacts kept?\n- How long does a rollback take, measured, not estimated?\n- Does it also revert environment variables and config? Usually not: config is a separate system with its own history, and a rollback that leaves the new config in place gives you a combination that has never been tested.\n\n## The kill switch, for when a deploy is too slow\n\nA rollback takes as long as a deploy: two to five minutes at best. Sometimes that is too long, and sometimes the thing you want to disable is not the whole release.\n\nA kill switch is a check on a value you can change **without a deploy**: an environment variable the platform can update, a row in a table, a key in Redis:\n\n```js\nif (!flags.enabled(\"new-search\")) return legacySearch(q);\n```\n\nPut one around anything expensive, anything new, and anything that calls a third party. Then a partner's outage becomes a thirty-second config change instead of an emergency release. The cost is a branch in your code and the discipline to delete flags once a feature is settled: a flag nobody has thought about in six months is a path nobody has tested in six months.\n\n## Ship dark, then to 1%\n\nFeature flags also change how you deploy risk. Merge the code with the flag off; the code is in production, exercised by real traffic, doing nothing. Turn it on for your own account. Then 1% of users. Then 10%.\n\nThis gives you what a rollback cannot: a small blast radius on the way in. And the \"rollback\" for a flagged feature is a boolean, which is faster and more precise than a deploy.\n\n## The rollbacks that do not work\n\nReverting the code is not always enough, and these are the reasons why:\n\n- **A cache full of the new shape.** The old code reads a serialised object it does not understand and throws. Include a version in your cache keys so a rollback misses the cache instead of poisoning itself.\n- **A queue full of new-format messages.** Old workers cannot parse them and they go to the dead-letter queue. Version your job payloads and make consumers tolerate unknown fields.\n- **Browsers holding the new client bundle.** A single-page app already loaded in someone's tab keeps calling an API shape you just removed. Keep the old endpoint alive for a while; do not delete an API in the same release that stops using it.\n- **A third party you have already told.** A webhook registered, a DNS record changed, a payment plan created. Those do not come back on their own.\n\nNone of these makes rollback the wrong choice. They are the reasons to keep the version boundary narrow: change one thing at a time, and the thing you have to undo is one thing."},"check":{"question":"Your app caches serialised objects in Redis. A release changes the object's shape, and you roll the code back thirty minutes later. The site stays broken, with deserialisation errors, even though the running code is now identical to what worked this morning. Why?","options":[{"text":"Redis lost its connection during the rollback and is returning empty values","correct":false,"tempting":"An empty cache would produce a miss and a fresh read, which is the harmless case. The symptom here is data being read and rejected."},{"text":"The cache still holds objects written in the new shape under keys the old code reads, so the old code is deserialising data it was never written to handle","correct":true},{"text":"The rollback reverted the code but not the environment variables, so the app is pointed at the wrong Redis","correct":false,"tempting":"A genuine rollback hazard worth remembering, but a wrong Redis would be empty rather than full of unreadable objects."},{"text":"Rolling back the code also rolled back the schema, so the cached objects no longer match the database","correct":false,"tempting":"Migrations do not run backwards during a code rollback, and the error described is happening at deserialisation, before the database is involved."}]},"prev":{"slug":"migrations-without-downtime","title":"Changing the schema under a running site","url":"https://addaly.com/learn/cloud-and-shipping/migrations-without-downtime"},"next":{"slug":"logs-at-2am","title":"Reading logs at 2am","url":"https://addaly.com/learn/cloud-and-shipping/logs-at-2am"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Undo, in under two minutes', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/rollback-and-the-kill-switch (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Undo, in under two minutes","courseTitle":"Shipping It","module":{"index":2,"slug":"making-a-deploy-boring","title":"Making a deploy boring"},"answer":null,"keyIdea":"Code rolls back, data does not, so keep them separate and make sure the undo you rely on is one you have actually performed while nothing was wrong.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Reversible","items":["Code: redeploy the previous build artifact","A feature behind a flag: flip a boolean, no deploy","Config: separate system, check it was reverted too"]},"type":"compare","right":{"title":"Not reversible","items":["Data users or jobs wrote while the bad version was live","A schema already written to in the new shape","A cache full of the new object shape","A queue full of new-format messages","A webhook registered or a DNS record changed"]},"title":"What a rollback undoes, and what it does not","caption":"Code rolls back because the previous artifact still exists. Everything the bad version wrote, told or cached stays where it is, and each of those needs its own repair."},"alt":"Figure: What a rollback undoes, and what it does not. Reversible: Code: redeploy the previous build artifact; A feature behind a flag: flip a boolean, no deploy; Config: separate system, check it was reverted too. Not reversible: Data users or jobs wrote while the bad version was live; A schema already written to in the new shape; A cache full of the new object shape; A queue full of new-format messages; A webhook registered or a DNS record changed. Code rolls back because the previous artifact still exists. Everything the bad version wrote, told or cached stays where it is, and each of those needs its own repair."}},"blocks":[{"t":"h2","id":"the-only-question-that-matters-at-minute-five","runs":[{"text":"The only question that matters at minute five"}]},{"t":"p","runs":[{"text":"Something is broken and you deployed twenty minutes ago. The question is not “what is wrong”. It is “how do I make it stop while I find out”."}]},{"t":"p","runs":[{"text":"Teams that answer that question in two minutes have incidents measured in minutes. Teams that start debugging first have incidents measured in hours, because the fire keeps burning while they read."}]},{"t":"p","runs":[{"text":"Decide the rule in advance and write it down: "},{"text":"if the cause is not clear within ten minutes, roll back and diagnose from the artifact and the logs.","b":true},{"text":" The evidence does not disappear when you roll back. Your users’ patience does."}]},{"t":"h2","id":"what-is-actually-reversible","runs":[{"text":"What is actually reversible"}]},{"t":"ul","items":[[{"text":"Code: yes.","b":true},{"text":" A previous build artifact still exists. This is why the last module insisted on building once and promoting the artifact: rollback becomes “run the old image”, not “rebuild an old commit and hope the dependency tree resolves the same way”."}],[{"text":"Schema: mostly no.","b":true},{"text":" Covered in the last lesson. Expand/migrate/contract exists precisely so that the "},{"text":"code","i":true},{"text":" rollback stays safe while the schema only moves forward."}],[{"text":"Data: no.","b":true},{"text":" Anything users wrote, or your job wrote, in the twenty minutes the bad version was live is now there. If the bad version wrote wrong data, rollback stops the bleeding and you still have a repair job to write."}]]},{"t":"p","runs":[{"text":"Keeping these three separate is most of the skill."}]},{"t":"h2","id":"practise-it-while-nothing-is-wrong","runs":[{"text":"Practise it while nothing is wrong"}]},{"t":"p","runs":[{"text":"A rollback you have never performed is a plan, not a capability. There is always something: the button is under a menu you have not opened, the old image was garbage-collected after seven days, the command needs a flag, the deploy key has expired."}]},{"t":"p","runs":[{"text":"Do it deliberately on a quiet afternoon. Deploy, roll back, confirm the site is on the previous version, roll forward again. Write the exact command in the runbook. The whole exercise takes fifteen minutes and it is the highest-value fifteen minutes in this course."}]},{"t":"p","runs":[{"text":"Know the answers to these before you need them:"}]},{"t":"ul","items":[[{"text":"What is the exact command or button, and who has permission?"}],[{"text":"How far back can you go? How long are old artifacts kept?"}],[{"text":"How long does a rollback take, measured, not estimated?"}],[{"text":"Does it also revert environment variables and config? Usually not: config is a separate system with its own history, and a rollback that leaves the new config in place gives you a combination that has never been tested."}]]},{"t":"h2","id":"the-kill-switch-for-when-a-deploy-is-too-slow","runs":[{"text":"The kill switch, for when a deploy is too slow"}]},{"t":"p","runs":[{"text":"A rollback takes as long as a deploy: two to five minutes at best. Sometimes that is too long, and sometimes the thing you want to disable is not the whole release."}]},{"t":"p","runs":[{"text":"A kill switch is a check on a value you can change "},{"text":"without a deploy","b":true},{"text":": an environment variable the platform can update, a row in a table, a key in Redis:"}]},{"t":"code","lang":"js","label":"JavaScript","text":"if (!flags.enabled(\"new-search\")) return legacySearch(q);"},{"t":"p","runs":[{"text":"Put one around anything expensive, anything new, and anything that calls a third party. Then a partner’s outage becomes a thirty-second config change instead of an emergency release. The cost is a branch in your code and the discipline to delete flags once a feature is settled: a flag nobody has thought about in six months is a path nobody has tested in six months."}]},{"t":"h2","id":"ship-dark-then-to-1","runs":[{"text":"Ship dark, then to 1%"}]},{"t":"p","runs":[{"text":"Feature flags also change how you deploy risk. Merge the code with the flag off; the code is in production, exercised by real traffic, doing nothing. Turn it on for your own account. Then 1% of users. Then 10%."}]},{"t":"p","runs":[{"text":"This gives you what a rollback cannot: a small blast radius on the way in. And the “rollback” for a flagged feature is a boolean, which is faster and more precise than a deploy."}]},{"t":"h2","id":"the-rollbacks-that-do-not-work","runs":[{"text":"The rollbacks that do not work"}]},{"t":"p","runs":[{"text":"Reverting the code is not always enough, and these are the reasons why:"}]},{"t":"ul","items":[[{"text":"A cache full of the new shape.","b":true},{"text":" The old code reads a serialised object it does not understand and throws. Include a version in your cache keys so a rollback misses the cache instead of poisoning itself."}],[{"text":"A queue full of new-format messages.","b":true},{"text":" Old workers cannot parse them and they go to the dead-letter queue. Version your job payloads and make consumers tolerate unknown fields."}],[{"text":"Browsers holding the new client bundle.","b":true},{"text":" A single-page app already loaded in someone’s tab keeps calling an API shape you just removed. Keep the old endpoint alive for a while; do not delete an API in the same release that stops using it."}],[{"text":"A third party you have already told.","b":true},{"text":" A webhook registered, a DNS record changed, a payment plan created. Those do not come back on their own."}]]},{"t":"p","runs":[{"text":"None of these makes rollback the wrong choice. They are the reasons to keep the version boundary narrow: change one thing at a time, and the thing you have to undo is one thing."}]}],"check":{"question":[{"text":"Your app caches serialised objects in Redis. A release changes the object’s shape, and you roll the code back thirty minutes later. The site stays broken, with deserialisation errors, even though the running code is now identical to what worked this morning. Why?"}],"options":[{"text":[{"text":"Redis lost its connection during the rollback and is returning empty values"}],"correct":false,"tempting":[{"text":"An empty cache would produce a miss and a fresh read, which is the harmless case. The symptom here is data being read and rejected."}]},{"text":[{"text":"The cache still holds objects written in the new shape under keys the old code reads, so the old code is deserialising data it was never written to handle"}],"correct":true,"tempting":null},{"text":[{"text":"The rollback reverted the code but not the environment variables, so the app is pointed at the wrong Redis"}],"correct":false,"tempting":[{"text":"A genuine rollback hazard worth remembering, but a wrong Redis would be empty rather than full of unreadable objects."}]},{"text":[{"text":"Rolling back the code also rolled back the schema, so the cached objects no longer match the database"}],"correct":false,"tempting":[{"text":"Migrations do not run backwards during a code rollback, and the error described is happening at deserialisation, before the database is involved."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"migrations-without-downtime","title":"Changing the schema under a running site","minutes":11},"next":{"slug":"logs-at-2am","title":"Reading logs at 2am","minutes":7}}}