Undo, in under two minutes

The only question that matters at minute five

Something is broken and you deployed twenty minutes ago. The question is not “what is wrong”. It is “how do I make it stop while I find out”.

Teams that answer that question in two minutes have incidents measured in minutes. Teams that start debugging first have incidents measured in hours, because the fire keeps burning while they read.

Decide the rule in advance and write it down: if the cause is not clear within ten minutes, roll back and diagnose from the artifact and the logs. The evidence does not disappear when you roll back. Your users’ patience does.

What is actually reversible

• Code: yes. A previous build artifact still exists. This is why the last module insisted on building once and promoting the artifact: rollback becomes “run the old image”, not “rebuild an old commit and hope the dependency tree resolves the same way”.
• Schema: mostly no. Covered in the last lesson. Expand/migrate/contract exists precisely so that the code rollback stays safe while the schema only moves forward.
• Data: no. Anything users wrote, or your job wrote, in the twenty minutes the bad version was live is now there. If the bad version wrote wrong data, rollback stops the bleeding and you still have a repair job to write.

Keeping these three separate is most of the skill.

Practise it while nothing is wrong

A rollback you have never performed is a plan, not a capability. There is always something: the button is under a menu you have not opened, the old image was garbage-collected after seven days, the command needs a flag, the deploy key has expired.

Do it deliberately on a quiet afternoon. Deploy, roll back, confirm the site is on the previous version, roll forward again. Write the exact command in the runbook. The whole exercise takes fifteen minutes and it is the highest-value fifteen minutes in this course.

Know the answers to these before you need them:

• What is the exact command or button, and who has permission?
• How far back can you go? How long are old artifacts kept?
• How long does a rollback take, measured, not estimated?
• Does it also revert environment variables and config? Usually not: config is a separate system with its own history, and a rollback that leaves the new config in place gives you a combination that has never been tested.

The kill switch, for when a deploy is too slow

A rollback takes as long as a deploy: two to five minutes at best. Sometimes that is too long, and sometimes the thing you want to disable is not the whole release.

A kill switch is a check on a value you can change without a deploy: an environment variable the platform can update, a row in a table, a key in Redis:

if (!flags.enabled("new-search")) return legacySearch(q);

Put one around anything expensive, anything new, and anything that calls a third party. Then a partner’s outage becomes a thirty-second config change instead of an emergency release. The cost is a branch in your code and the discipline to delete flags once a feature is settled: a flag nobody has thought about in six months is a path nobody has tested in six months.

Ship dark, then to 1%

Feature flags also change how you deploy risk. Merge the code with the flag off; the code is in production, exercised by real traffic, doing nothing. Turn it on for your own account. Then 1% of users. Then 10%.

This gives you what a rollback cannot: a small blast radius on the way in. And the “rollback” for a flagged feature is a boolean, which is faster and more precise than a deploy.

The rollbacks that do not work

Reverting the code is not always enough, and these are the reasons why:

• A cache full of the new shape. The old code reads a serialised object it does not understand and throws. Include a version in your cache keys so a rollback misses the cache instead of poisoning itself.
• A queue full of new-format messages. Old workers cannot parse them and they go to the dead-letter queue. Version your job payloads and make consumers tolerate unknown fields.
• Browsers holding the new client bundle. A single-page app already loaded in someone’s tab keeps calling an API shape you just removed. Keep the old endpoint alive for a while; do not delete an API in the same release that stops using it.
• A third party you have already told. A webhook registered, a DNS record changed, a payment plan created. Those do not come back on their own.

None of these makes rollback the wrong choice. They are the reasons to keep the version boundary narrow: change one thing at a time, and the thing you have to undo is one thing.