A change feed for your own infrastructure

The idea

A small self-hosted service that polls the things your infrastructure actually depends on — container image digests, DNS records, TLS certificates, open ports, package versions, environment files, cron entries — and writes every observed difference into a single chronological feed. When something breaks at 2am, you open one page, scroll back to the hour it started, and see what moved. No log aggregation, no metrics, no tracing: just a diary of state changes, with before-and-after diffs.

Why build this

Every solo operator and small team has the same debugging loop: something is broken, and the first real question is "what changed?" Answering it means checking shell history on three boxes, scrolling a CI run list, remembering whether Watchtower pulled overnight, and guessing whether the registry moved :latest under you. The information exists; it's just scattered across systems that each hold one slice of it and none of which are ordered by time.

Big shops solve this with change management tooling that assumes every change comes through a pipeline. Small setups don't work that way — a change is as likely to be a manual docker compose pull, a registrar edit, or a package upgrade from unattended-upgrades. Polling observable state catches all of those, because it doesn't care how the change was made.

The timing is good: nearly everything worth watching now has a cheap read-only interface. Registries expose manifest digests over the Docker Registry HTTP API v2, DNS is a dig away, certificate expiry and fingerprint come free from a TLS handshake, and Docker's own socket answers "what is running, from which digest, with which env keys" in one call.

Stack sketch

• Collector: Go single binary, one goroutine per source, each returning a normalized {source, key, value, observed_at} set on a schedule.
• Storage: SQLite with a snapshots table and a changes table; a change row is written only when a key's hash differs from the last snapshot. Months of history fits in a few hundred MB.
• Sources for v1: Docker socket (/var/run/docker.sock) for running containers and image digests, registry manifest HEAD requests for upstream digest drift, miekg/dns for A/AAAA/MX/TXT records, crypto/tls dial for cert fingerprint and expiry, dpkg-query/rpm -qa over SSH for package sets, and file hashes for a configured list of paths.
• UI: server-rendered HTML with html/template, one reverse-chronological page, filter by source, expandable unified diffs via sergi/go-diff.
• Notifications: outbound webhook per change with a rules file — quiet for routine cert renewals, loud for a DNS record change.
• Deploy: one container behind your existing reverse proxy; secrets via environment, no agent on remote hosts beyond SSH key access.

Scope for v1

In: the six sources above, a fixed 5-minute poll, the feed page, per-source filtering, webhook notifications, and a YAML config listing hosts, domains, and file paths. Out: agents, push-based collection, RBAC, multi-tenant anything, alerting logic beyond match-and-fire, and any attempt to correlate changes with incidents automatically. The value in v1 is entirely "one ordered list, complete enough to trust."

Where it could go

The obvious next step is annotation: let a human mark a range of the feed as an incident, attach a note, and keep it. After a few months you have a searchable history of what actually caused outages on your own stack — the thing postmortem docs are supposed to be but rarely are, because nobody writes them for a two-person setup.

From there, two directions. One is prediction: the feed already knows every cert expiry and every image that has drifted from upstream, so it can surface "these three things will bite you in the next 30 days" without any new collection. The other is packaging it for the managed-services crowd — a consultant running twenty client stacks wants exactly this feed, scoped per client, as the artifact they show when a client asks what they've been doing.

Watch out for

Polling intervals hide changes: anything that flips and flips back inside the window is invisible, and shortening the interval multiplies registry API calls into rate-limit territory fast. Be explicit in the UI about what the resolution is, so nobody reads a gap in the feed as proof that nothing happened.