"We have backups"
Every product had a backup. CronJobs dumped the databases and shipped them to a bucket, and that was enough to answer "do you have backups?". But nobody could answer the questions that matter in a disaster: does the backup restore? How long does it take to come back? How much data can we lose? And if the backup stops working, does anyone find out?
The only honest way to answer was to restore.
Restoring for real
I provisioned an isolated restore environment and, right at the start, found a flaw in the Terraform: the 600 GB data disk was never formatted or mounted. I fixed it, isolated the disk from another test running in parallel, and restored the production database by streaming straight from the bucket, without going through any local machine. That was 249 GB and 85 tables in 2h41, with integrity validated.
The exercise revealed what no dashboard showed. Production backups had been failing for more than a day, silently. There was a referential integrity inconsistency already in production that would have gone unnoticed. And the bucket had piled up about 92 TB with no lifecycle policy, roughly US$ 2.3 thousand a month.
I repeated the method on a product running MongoDB. The dump came from a secondary node, off peak, with at most 15 seconds of replication lag and no impact on production. That was 425 million messages restored with zero failures and an identical checksum. The measured recovery time was about 5h20, because rebuilding the indexes took longer than loading the data.
Why the backup failed
The retention routine had a deadlock: old dumps were only deleted after a successful backup. When the disk filled up, the backup failed before cleaning, and the cycle never recovered on its own. The volume reached 96%, full of partial dumps. I freed space by deleting only copies I had already validated in the bucket and fixed the cause: the dump now goes straight to the bucket, with no local copy.
On a third product, I found a silent failure before it happened. The backup job had no maximum run time, didn't allow concurrent runs and sent its notification from inside itself. A stuck job would block every backup after it, and the warning would depend on the very thing that was stuck. I proposed detecting failure from outside, by the age of the last success.
Making the backup speak up, and protecting it
I created alerts for failure and for a backup stalled longer than expected, plus start, success and failure messages in the team channel. I validated the rules with promtool before applying them.
The dump was also uploaded in plain text: a bucket leak would expose the whole database, personal data included. I evaluated three ways to hold the keys and chose envelope encryption with the cloud KMS, with a new data key per file, a master key that never leaves the KMS and integrity checking by HMAC. The extra cost was zero. I split privileges into two identities: one that can only encrypt and write, another that can only decrypt and read. The bucket credential moved from a ConfigMap to a Secret, and the manifests that only existed in the cluster went into Git, checked field by field against production.
Turning it into policy
With measured times in hand, I wrote the disaster recovery policy, with RPO and RTO targets based on what actually happened in the tests, a record of the drills, frequency and success criteria, aligned with ISO 27001 backup and continuity controls. So confidence wouldn't depend on someone remembering to test, I built an automatic drill: a job that takes the latest encrypted backup, decrypts it, restores it to a test database, validates it and notifies the team.
Along the way, I found and documented a production database with no backup at all, with a plan to set one up.
What changed
The main products' backups stopped merely "existing" and became restored and measured. RTO and RPO became numbers in a policy, not estimates. Backups that failed silently now alert the team, and dumps are now encrypted, with keys out of reach of anyone who only has bucket access.
What I take from it
The only way to know whether a backup works is to restore it. Every test revealed something no dashboard showed. And the real recovery time (5h20, not "a couple of hours") only shows up when you measure it.