{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"backups-and-restore","title":"The restore you have never tested","url":"https://addaly.com/learn/cloud-and-shipping/backups-and-restore","position":39,"of":74,"minutes":8,"answer":"An untested backup is a guess; the drill is the only thing that turns it into a plan.","keyIdea":"An untested backup is a guess; the drill is the only thing that turns it into a plan."},"body":{"format":"markdown","text":"## A backup you have not restored is a hypothesis\n\nThat is the whole lesson, and the rest is detail. You do not have backups. You have files you believe are backups, and belief is untested until the day it is tested for you.\n\n## What makes a copy a real backup\n\nThe usual advice is 3-2-1: three copies, two kinds of media, one off-site. For a small project the part that actually matters is narrower:\n\n**At least one copy must live outside the blast radius of the thing it protects.** If the same credentials that can drop your database can also delete the bucket holding its backups, then one compromised key, or one confident `terraform destroy`, ends the company. A different account, ideally a different provider, with write-only or append-only access from the app side.\n\n## Two words worth knowing\n\n**RPO**: how much data you can afford to lose. A nightly dump means your RPO is 24 hours. If the disaster lands at 15:30 and the dump ran at 02:00, thirteen and a half hours are gone.\n\n**RTO**: how long you can be down. A 40GB dump does not restore in five minutes. It can take one to three hours, plus index rebuilds, plus the time spent finding the file and remembering the command.\n\nWrite both numbers down. If your honest RPO is \"an hour\" and you have nightly dumps, you have not made a decision; you have avoided looking.\n\n> Figure: One bad afternoon with a nightly dump. Timeline: 02:00, Nightly pg_dump runs and uploads to the bucket; 15:30, A migration drops the wrong column. Thirteen and a half hours of data is lost: the RPO; 15:50, Somebody finds the backup file and the restore command; 16:10, Restore starts into a fresh, empty database; 17:40, Restore finishes, indexes rebuild; 17:55, App pointed at it, rows checked. Two hours twenty-five minutes down: the RTO. Everything written between the dump and the disaster is gone: that gap is the RPO. The time from disaster to a working site is the RTO, and it is nearly always two to five times what was guessed, because finding the file and remembering the command are part of it.\n\n**Point-in-time recovery** is the alternative: the database keeps its write-ahead log, so you can restore to 15:29:58, one second before the bad migration. Managed Postgres usually offers around 7 days of PITR on paid plans and often none on free ones. Go and check today which you have. \"The provider handles it\" is not an answer; the retention number is.\n\n## The drill\n\nPut it on a calendar. Quarterly is plenty for a small project.\n\n1. Take the real backup file, from where it lives.\n2. Restore into a fresh, empty database.\n3. Point a copy of the app at it and open a page.\n4. Check specific rows: your own account, the newest row in the busiest table.\n5. Time the whole thing and write the number down. That is your RTO, and it is usually two to five times what you guessed.\n\nEvery team that does this for the first time finds at least one of these:\n\n- The dump has been zero bytes since a flag changed in March.\n- The cron ran fine, the upload failed silently, and nobody checked the exit status.\n- `pg_dump` from version 16 refuses to restore into version 15.\n- The backup is encrypted and the key lived on a server that no longer exists.\n- Nobody actually knows the restore command.\n\n## What is not a backup\n\n- **A read replica.** It faithfully replicates your `DELETE FROM users` in under a second.\n- **Provider redundancy or RAID.** Protects against a disk dying. Offers nothing against you.\n- **Soft deletes.** Protects against a user clicking the wrong thing. Offers nothing against a migration.\n\nAll three protect against hardware. Almost nothing that destroys a small project's data is hardware.\n\n## Enough, and cheap\n\nA nightly `pg_dump | gzip` to a bucket at a different provider, thirty days of retention, and (this is the part people skip) **an alert when today's file is missing or smaller than a plausible threshold.** \"Backup job succeeded\" logs lie; file size does not. Add the quarterly restore drill. Storage costs a few cents to a few dollars a month.\n\nThat is a real backup system, and it fits in about fifteen lines of shell."},"check":{"question":"A team runs a nightly `pg_dump` to object storage. Monitoring confirms the job exits 0 every night and a file of roughly the expected size appears. On a Tuesday afternoon a bad migration wipes a table. What is the most likely way this still goes badly?","options":[{"text":"Nobody has ever performed a restore, so the first attempt happens under pressure and whether the dump is complete and loadable is still unknown","correct":true},{"text":"The exit code is unreliable, because a shell pipeline reports the status of the last command and can hide a `pg_dump` failure","correct":false,"tempting":"A genuine bug worth fixing with pipefail, but a file of plausible size appearing nightly is evidence against it, and it is not the deeper problem."},{"text":"They lose up to a day of data because the dump is nightly, and that is the core failure","correct":false,"tempting":"Correctly names the RPO cost, but that is a trade-off they chose knowingly, not the thing that turns an incident into a disaster."},{"text":"Object storage versioning will cause the restore to pull an older copy than intended","correct":false,"tempting":"Invents a specific mechanism for the failure instead of the plain one: a procedure that has never been run."}]},"prev":{"slug":"email-that-arrives","title":"Why your email went to spam","url":"https://addaly.com/learn/cloud-and-shipping/email-that-arrives"},"next":{"slug":"the-security-you-actually-need","title":"The short list that actually matters","url":"https://addaly.com/learn/cloud-and-shipping/the-security-you-actually-need"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The restore you have never tested', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/backups-and-restore (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The restore you have never tested","courseTitle":"Shipping It","module":{"index":5,"slug":"staying-alive","title":"Staying alive: the bad day and the day after"},"answer":null,"keyIdea":"An untested backup is a guess; the drill is the only thing that turns it into a plan.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"type":"timeline","title":"One bad afternoon with a nightly dump","events":[{"when":"02:00","what":"Nightly pg_dump runs and uploads to the bucket"},{"when":"15:30","what":"A migration drops the wrong column. Thirteen and a half hours of data is lost: the RPO"},{"when":"15:50","what":"Somebody finds the backup file and the restore command"},{"when":"16:10","what":"Restore starts into a fresh, empty database"},{"when":"17:40","what":"Restore finishes, indexes rebuild"},{"when":"17:55","what":"App pointed at it, rows checked. Two hours twenty-five minutes down: the RTO"}],"caption":"Everything written between the dump and the disaster is gone: that gap is the RPO. The time from disaster to a working site is the RTO, and it is nearly always two to five times what was guessed, because finding the file and remembering the command are part of it."},"alt":"Figure: One bad afternoon with a nightly dump. Timeline: 02:00, Nightly pg_dump runs and uploads to the bucket; 15:30, A migration drops the wrong column. Thirteen and a half hours of data is lost: the RPO; 15:50, Somebody finds the backup file and the restore command; 16:10, Restore starts into a fresh, empty database; 17:40, Restore finishes, indexes rebuild; 17:55, App pointed at it, rows checked. Two hours twenty-five minutes down: the RTO. Everything written between the dump and the disaster is gone: that gap is the RPO. The time from disaster to a working site is the RTO, and it is nearly always two to five times what was guessed, because finding the file and remembering the command are part of it."}},"blocks":[{"t":"h2","id":"a-backup-you-have-not-restored-is-a-hypothesis","runs":[{"text":"A backup you have not restored is a hypothesis"}]},{"t":"p","runs":[{"text":"That is the whole lesson, and the rest is detail. You do not have backups. You have files you believe are backups, and belief is untested until the day it is tested for you."}]},{"t":"h2","id":"what-makes-a-copy-a-real-backup","runs":[{"text":"What makes a copy a real backup"}]},{"t":"p","runs":[{"text":"The usual advice is 3-2-1: three copies, two kinds of media, one off-site. For a small project the part that actually matters is narrower:"}]},{"t":"p","runs":[{"text":"At least one copy must live outside the blast radius of the thing it protects.","b":true},{"text":" If the same credentials that can drop your database can also delete the bucket holding its backups, then one compromised key, or one confident "},{"text":"terraform destroy","code":true},{"text":", ends the company. A different account, ideally a different provider, with write-only or append-only access from the app side."}]},{"t":"h2","id":"two-words-worth-knowing","runs":[{"text":"Two words worth knowing"}]},{"t":"p","runs":[{"text":"RPO","b":true},{"text":": how much data you can afford to lose. A nightly dump means your RPO is 24 hours. If the disaster lands at 15:30 and the dump ran at 02:00, thirteen and a half hours are gone."}]},{"t":"p","runs":[{"text":"RTO","b":true},{"text":": how long you can be down. A 40GB dump does not restore in five minutes. It can take one to three hours, plus index rebuilds, plus the time spent finding the file and remembering the command."}]},{"t":"p","runs":[{"text":"Write both numbers down. If your honest RPO is “an hour” and you have nightly dumps, you have not made a decision; you have avoided looking."}]},{"t":"p","runs":[{"text":"Point-in-time recovery","b":true},{"text":" is the alternative: the database keeps its write-ahead log, so you can restore to 15:29:58, one second before the bad migration. Managed Postgres usually offers around 7 days of PITR on paid plans and often none on free ones. Go and check today which you have. “The provider handles it” is not an answer; the retention number is."}]},{"t":"h2","id":"the-drill","runs":[{"text":"The drill"}]},{"t":"p","runs":[{"text":"Put it on a calendar. Quarterly is plenty for a small project."}]},{"t":"ol","start":1,"items":[[{"text":"Take the real backup file, from where it lives."}],[{"text":"Restore into a fresh, empty database."}],[{"text":"Point a copy of the app at it and open a page."}],[{"text":"Check specific rows: your own account, the newest row in the busiest table."}],[{"text":"Time the whole thing and write the number down. That is your RTO, and it is usually two to five times what you guessed."}]]},{"t":"p","runs":[{"text":"Every team that does this for the first time finds at least one of these:"}]},{"t":"ul","items":[[{"text":"The dump has been zero bytes since a flag changed in March."}],[{"text":"The cron ran fine, the upload failed silently, and nobody checked the exit status."}],[{"text":"pg_dump","code":true},{"text":" from version 16 refuses to restore into version 15."}],[{"text":"The backup is encrypted and the key lived on a server that no longer exists."}],[{"text":"Nobody actually knows the restore command."}]]},{"t":"h2","id":"what-is-not-a-backup","runs":[{"text":"What is not a backup"}]},{"t":"ul","items":[[{"text":"A read replica.","b":true},{"text":" It faithfully replicates your "},{"text":"DELETE FROM users","code":true},{"text":" in under a second."}],[{"text":"Provider redundancy or RAID.","b":true},{"text":" Protects against a disk dying. Offers nothing against you."}],[{"text":"Soft deletes.","b":true},{"text":" Protects against a user clicking the wrong thing. Offers nothing against a migration."}]]},{"t":"p","runs":[{"text":"All three protect against hardware. Almost nothing that destroys a small project’s data is hardware."}]},{"t":"h2","id":"enough-and-cheap","runs":[{"text":"Enough, and cheap"}]},{"t":"p","runs":[{"text":"A nightly "},{"text":"pg_dump | gzip","code":true},{"text":" to a bucket at a different provider, thirty days of retention, and (this is the part people skip) "},{"text":"an alert when today’s file is missing or smaller than a plausible threshold.","b":true},{"text":" “Backup job succeeded” logs lie; file size does not. Add the quarterly restore drill. Storage costs a few cents to a few dollars a month."}]},{"t":"p","runs":[{"text":"That is a real backup system, and it fits in about fifteen lines of shell."}]}],"check":{"question":[{"text":"A team runs a nightly "},{"text":"pg_dump","code":true},{"text":" to object storage. Monitoring confirms the job exits 0 every night and a file of roughly the expected size appears. On a Tuesday afternoon a bad migration wipes a table. What is the most likely way this still goes badly?"}],"options":[{"text":[{"text":"Nobody has ever performed a restore, so the first attempt happens under pressure and whether the dump is complete and loadable is still unknown"}],"correct":true,"tempting":null},{"text":[{"text":"The exit code is unreliable, because a shell pipeline reports the status of the last command and can hide a "},{"text":"pg_dump","code":true},{"text":" failure"}],"correct":false,"tempting":[{"text":"A genuine bug worth fixing with pipefail, but a file of plausible size appearing nightly is evidence against it, and it is not the deeper problem."}]},{"text":[{"text":"They lose up to a day of data because the dump is nightly, and that is the core failure"}],"correct":false,"tempting":[{"text":"Correctly names the RPO cost, but that is a trade-off they chose knowingly, not the thing that turns an incident into a disaster."}]},{"text":[{"text":"Object storage versioning will cause the restore to pull an older copy than intended"}],"correct":false,"tempting":[{"text":"Invents a specific mechanism for the failure instead of the plain one: a procedure that has never been run."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"email-that-arrives","title":"Why your email went to spam","minutes":9},"next":{"slug":"the-security-you-actually-need","title":"The short list that actually matters","minutes":10}}}