The idea
A command-line tool that takes a backup you already produce — a Postgres dump in S3, a restic snapshot, a MySQL logical export — and restores it into a disposable container on a schedule. Once the restore finishes it runs a short list of assertions you wrote in YAML: row counts within a sane band, the newest order no more than 26 hours old, a known user row present, schema version matching the running app. It then writes a dated report with the restore duration, the assertions that passed, and the exact bytes it pulled.
Why build this
Most teams have backups. Very few have evidence that any of them restore. The failure mode is quiet and specific: the nightly job has been dumping a database that dropped a role six months ago, or the encryption key rotated, or the dump is 40 bytes because pg_dump exited non-zero into a pipe that swallowed the error. Nobody finds out until the day it matters.
The reason nobody tests restores is not ignorance, it is that a restore test is annoying to build. You need a scratch machine, credentials, enough disk, and someone to look at the result. Containers and cheap object storage have made the mechanics trivial; what is missing is a small opinionated runner that turns it into a cron line. Compliance pressure helps too — SOC 2 and ISO 27001 both want restore evidence, and most companies produce it by writing a paragraph.
Stack sketch
- Go for the CLI, distributed as a single static binary plus a container image.
- Docker or Podman via the local socket to spin up the restore target;
testcontainers-gohandles lifecycle and port wiring. - Restore drivers for Postgres (
pg_restore,psql), MySQL (mysql), MongoDB (mongorestore), restic, and plain tarballs. - Backup sources over S3-compatible storage with the AWS SDK, plus local paths and SFTP.
- Assertions in YAML: a
sql:query with anexpect:expression evaluated byexpr-lang/expr. - Reports as JSON plus a rendered Markdown summary; optional signing with
cosignso the artifact is tamper-evident. - Notifications through a generic webhook, with Slack and PagerDuty payload presets.
Scope for v1
- In: Postgres and restic sources, S3 and local backends, container-based restore, SQL assertions, JSON + Markdown report, non-zero exit on failure so any scheduler catches it.
- In:
backuptest initthat inspects a live database and generates a starter assertion file. - Out: backup creation. The tool never writes to your production database and never produces backups — it only consumes them.
- Out: a web dashboard, a hosted service, multi-tenant anything. The report is a file.
- Out: point-in-time recovery and WAL replay. That is a second milestone, not a first one.
Where it could go
The obvious next step is partial-restore timing. Once you are restoring nightly anyway, you have a real, measured RTO instead of a number someone put in a spreadsheet — and you can graph it as the dataset grows and quietly crosses the point where your recovery plan stops being credible. Publishing that trend line is the feature a CTO actually wants.
After that, differential assertions get interesting: compare last night's restore to the previous one and flag tables that stopped growing, which catches a broken ETL long before anyone notices in a dashboard. A third path is the compliance artifact — a signed, append-only log of every restore attempt, exportable as the evidence an auditor asks for, which is the version a company will pay for.
Watch out for
Restore tests need read access to production backups, which makes the runner a high-value target and a data-residency problem — keep it inside the same account and network boundary as the backups themselves. Also budget for size: a naive nightly restore of a 2 TB database will cost more in egress and disk than the outage it is insuring against, so sampling and staging-sized fixtures need to be a first-class option, not an afterthought.