Fix the link rot in your back catalog

A CLI that audits every outbound link in your published writing, classifies how it died, and proposes an archived replacement.

The idea

Point a command at the directory where your posts live. It extracts every outbound link, checks each one, and sorts the failures into categories that actually mean something: moved permanently, gone, domain expired, paywalled since you linked it, blocked-to-bots but alive, or quietly replaced by a parked page that still returns 200. For everything genuinely dead, it queries the Wayback Machine for the snapshot closest to the post's publication date and writes a patch that rewrites the link. You review the diff and commit.

Why build this

Anyone who has written online for more than three years is sitting on a slow leak. The rough industry figure is that a quarter of links break within five years, and the older the post, the worse it gets — which is backwards, because the old posts are the ones still pulling search traffic. Nobody notices because nobody rereads their own archive.

Link checkers already exist and they solve the easy half. lychee will tell you a URL returned 404 in about ten seconds. What it will not tell you is that the link still resolves but now points at a domain squatter, that the 403 is Cloudflare and the page is fine, that this particular link has been failing for eight months rather than since Tuesday, or what you should put there instead. The interesting work is all in classification and repair, and that is where a tool can earn its place.

The timing is good because the raw material is free and well documented. The Wayback CDX API lets you ask for the snapshot nearest a given timestamp, and the Save Page Now API lets you archive a link at publish time so the fallback exists before you need it.

Stack sketch

Language: Go, distributed as a single binary. Concurrency is the whole workload and errgroup with a semaphore per host is the right shape.
Extraction: goldmark with its AST walker for Markdown, plus goquery for rendered HTML if you point it at a built dist/. Links carry their file, line, and the post's front-matter date.
Checking: plain net/http with a shared client, HEAD first and a ranged GET fallback for servers that mishandle HEAD. Per-host rate limiting with golang.org/x/time/rate.
Soft-404 detection: fetch a deliberately nonsense path on the same host, then compare the suspect page's normalized text to that control using a SimHash. High similarity plus a 200 means the page is gone and the server is lying about it.
Archive lookup: the Wayback CDX server (timestamp, original, statuscode fields), filtered to statuscode:200 and sorted by distance from the post date.
State: SQLite via modernc.org/sqlite — no cgo. One row per (URL, check run) so you get a rot history rather than a snapshot.
Output: a unified diff to stdout, --write to apply, and a JSON report for CI.

Scope for v1

In:

Markdown directories with YAML front matter, which covers Hugo, Astro, Eleventy, and Jekyll.
The six failure classes above, each with a confidence score.
Wayback replacement suggestions with the snapshot date shown inline.
A rot history table and a --since flag so a scheduled run only reports what changed.

Deliberately out:

A headless browser. JS-rendered link targets are a real gap; accept it and mark those links unchecked rather than guessing.
Auto-opening pull requests. Write the patch, let the human run git apply.
Image and asset checking.
Any hosted component.

Where it could go

The strongest expansion is prevention rather than repair: a publish hook that submits every new outbound link to Save Page Now the moment a post ships, so three years later the snapshot you want already exists and is dated correctly. That turns the tool from a cleanup job into insurance, and it is maybe two hundred lines on top of what v1 already does.

After that, the obvious product is the scheduled version — a small hosted service that runs the same binary weekly against a repo and opens one batched PR a month, plus a dashboard showing rot rate over time per post. Publishers with a real archive will pay for that in a way solo bloggers will not, so build the CLI for credibility and sell the schedule. A third direction is adjacent audiences with the same problem and stricter needs: academic writing, legal citations, and documentation teams whose links point at vendor docs that get reorganized every release.

Watch out for

Be polite to archive.org or you will get throttled into uselessness — cache CDX responses aggressively, stay well under the Save Page Now rate limits, and never parallelize against a single host. The subtler risk is false positives: bot-blocked pages that look dead will destroy trust in the tool faster than missing a real 404, so default to "blocked, verify manually" whenever you are not sure.