The idea
A small service that treats "can we restore?" as a test that runs on a schedule. Every night it pulls the most recent backup artifact from wherever it lives, restores it into a disposable container, runs a set of assertions against the restored state, then destroys everything and records a pass or fail. You get a dashboard of green squares per backup source and an alert the first time one goes red — which is usually months before the day you actually need the restore.
Why build this
Almost everyone running their own infrastructure has backups. Very few have evidence those backups restore. The common failure modes are quiet: pg_dump running against a stale replica, a --exclude pattern that silently grew to cover the uploads directory, a cron job whose container was rebuilt without the credentials, an S3 lifecycle rule that expired objects at 30 days. Every one of these produces a backup file of plausible size that lands in the bucket on schedule. The monitoring you already have says the job exited zero.
The gap is that verifying a restore is genuinely annoying to do by hand — you need somewhere to put it, a way to check it, and the discipline to do it again next quarter. Automating it is the only version that survives. And the tooling is finally cheap: ephemeral containers make the "somewhere to put it" free, and most backup formats have a restore command that is one line.
Stack sketch
- Runner: Go binary that shells out to the Docker Engine API (
github.com/docker/docker/client) to create, run, and tear down restore sandboxes. One container per drill, always withAutoRemoveand a hard timeout. - Config: a
drills.yamldescribing each source — where the artifact lives, which image restores it, and the assertions to run. Keep it declarative and diffable, no UI-created state. - Fetchers:
rcloneas a subprocess covers S3, B2, Backblaze, SFTP, and local paths without you writing per-provider code. - Restore recipes: shipped defaults for Postgres (
pg_restoreintopostgres:17-alpine), MySQL, SQLite, plain tarballs, and Restic/Borg repositories. Everything else is "run this image with the artifact mounted at/artifact". - Assertions: SQL queries with expected bounds (
select count(*) from users> 0,max(created_at)within 48 hours), file existence and minimum-size checks, and an optional arbitrary exit-code script. - State + UI: SQLite for run history, plus a server-rendered Go template dashboard. No SPA.
- Alerting: a webhook per drill, plus a Prometheus
/metricsendpoint exposing last-success timestamp and restore duration.
Scope for v1
In: the YAML config, S3 and local-filesystem fetchers, Postgres and tarball recipes, SQL and file assertions, nightly cron scheduling, the run-history dashboard, and webhook alerts on transition to failure.
Out: agents on remote hosts, taking the backups themselves (assume something already does), multi-tenancy, RBAC, and any hosted version. Deliberately out: restoring to a real environment. The sandbox is disposable and the runner never has write credentials to anything that matters.
Where it could go
The natural next step is measuring restore time, not just restore success. Once you have nightly timing data you can chart recovery objective drift — the moment your database gets big enough that a restore takes four hours instead of twenty minutes, that shows up as a slope on a graph rather than a discovery during an outage.
After that, drills become artifacts you can hand to someone else. A signed monthly report showing "every backup source restored and passed assertions on these dates" is the exact evidence SOC 2 and ISO 27001 auditors ask for, and it's the version of the answer that's actually true. That's also the wedge into paid: the runner stays open source, the attestation and reporting layer is the product.
Watch out for
The restored data is real production data sitting in a container, so the sandbox needs to be genuinely isolated — no shared network, no outbound access, and logs scrubbed of query output. Getting that wrong turns a safety tool into a breach.