The 3-2-1 rule
- 3 copies: 1 production + 2 backups;
- 2 media: not all on one disk or one cloud;
- 1 offsite / offline: against ransomware and datacenter failure.
RPO and RTO
RPO (recovery point objective) = how much data loss you tolerate; RTO (recovery time objective) = how long downtime you tolerate. They decide backup frequency and recovery design — align with the business first.
Verify backups
Hash the backup when written and hash again on restore to confirm no corruption. Keep encryption keys separate from the backup, or the backup is unreadable.
Most common backup failures
- Backup but never restore: the first real restore happens in a crisis and then it doesn't work;
- No monitoring: days of silent failures mean you weren't backed up at all;
- Same-source backup: backup and live data in one account/cloud, deleted together by ransomware.
Restore drills
Do a real restore periodically (even one file), record the time, and push RTO down to an acceptable range. See Verifying file integrity with SHA-256.
Try it
Hash your backup files: File hash.
Real-world cases: three times "we have backups" but could not restore
- Backups on the same host: when the disk dies, data and backups disappear together. Keep copies off-site, with at least one offline or immutable.
- Never rehearsing a restore: only during an incident do you discover the backup script has failed silently for six months. Run restore drills and verify integrity and readability.
- Backing up data but not configuration: data comes back while keys, certificates and orchestration config are lost, so the service still will not start. Cover code, configuration, secrets and identity.
FAQ
How often should I back up? Work back from your tolerable data loss (RPO); critical systems may need continuous backup. How long to retain? By compliance and business need — commonly dense for 7 days, daily for 30, then monthly. Should backups be encrypted? Yes; they are often overlooked and must be encrypted with tightly controlled access. How thorough should a drill be? At least one full restore into an isolated environment, recording actual duration and blockers to validate your RTO.
Tiering your backup strategy
Different data has very different recovery needs, and one policy for everything satisfies nobody.
- Tier by recovery objective: core transactional data may need minute-level RPO, logs and analytics tolerate days, archives only quarterly. One frequency for all is both expensive and unnecessary.
- Full plus incremental: take a periodic full backup with incremental or differential in between to control cost while keeping restore steps short.
- Isolate backup credentials: the backup system should not share production credentials, and production should not be able to delete backups. Ransomware deletes backups before encrypting, so isolation is the last defence.
- Verify automatically: after each run, check readability, plausible size and checksum, and feed the result into monitoring — unreadable files stored for months are worthless.
- Write executable restore docs: include exact commands, ordering and required credentials rather than "restore from backup". Nobody wants to read something vague during an outage.
Running the drill
Set a clear target: how fast and to what state counts as success. Include someone unfamiliar with the system, because the original owner may not be on call. Record blockers and track follow-ups, or the next drill hits the same walls.
Cloud-specific cautions
In the cloud, snapshots and backups are often enabled by default and never verified. Confirm whether they cross availability zones or regions, whether deletion protection is on, and whether a compromised account could delete them too. Add these to routine checks rather than regretting later.