Which CI jobs have ever caught a bug

The idea

A command-line tool that reads your CI provider's run history and builds a ledger for every job in every workflow: how many times it ran, how many minutes it consumed, how often it failed, and — the part nobody measures — how often it was the only job that failed. A job that has run nine thousand times and never once been the sole red check is pure tax. A job that fails rarely but always alone is the one earning its keep. Output is a ranked table plus a --suggest mode that proposes concrete changes: move this job to nightly, make that one required, delete the third.

Why build this

Every CI pipeline accretes. Someone adds a matrix entry for an old runtime, a second lint pass, a smoke test that duplicates a unit test, and nothing is ever removed because removal needs an argument and nobody has the data to make one. The result is a pipeline where PRs wait eleven minutes and the team quietly stops reading the check list.

Teams already feel the cost in two places: the invoice, and the pull request latency that turns reviews into context-switching. What they lack is the other half of the equation — which jobs have ever told them something they didn't already know. Providers expose plenty of run metadata through APIs now, and the correlation is cheap to compute, so the analysis is entirely a tooling gap rather than a data one.

Stack sketch

• Go, distributed as a single static binary (ciaudit) via Homebrew and go install — it has to run in CI itself, where installing a Python toolchain is friction.
• Providers behind one interface, starting with GitHub Actions (/repos/{o}/{r}/actions/runs plus per-run /jobs, which carries started_at/completed_at and conclusion), then GitLab CI and Buildkite.
• Local cache in SQLite via modernc.org/sqlite (pure Go, no cgo). Incremental sync keyed on run ID, so the first pull is slow and every later one is seconds.
• Cost model as a YAML file: per-minute rates by runner label, including self-hosted estimates. Defaults shipped for GitHub's public rates, overridable.
• Rendering with lipgloss for the terminal table, plus --json and --markdown so it can post itself into a PR comment.
• Correlation is plain SQL over the cached runs — no stats library needed. Per job: run count, failure count, failures where it was the only failing job, median duration, p95 duration, and estimated spend.

Scope for v1

In:

• ciaudit sync and ciaudit report against GitHub Actions for one repo.
• The core table: job name, runs, minutes, estimated cost, failure rate, sole-failure count.
• Flagging three patterns: never-failed jobs, always-fails-with-others jobs (redundant signal), and jobs whose p95 duration sets the critical path.
• A --since 90d window, because a job's value changes after a refactor.

Out:

• Writing changes to workflow YAML. Report only; humans edit the pipeline.
• Flaky-test attribution at the individual test level — that's a different tool with a different data source.
• A hosted dashboard, auth, or multi-repo rollups.
• Any attempt to price self-hosted runners accurately; take the rate from config and say so.

Where it could go

The obvious next step is the critical-path view: given each job's duration distribution and the needs graph, show what PR wall-clock time would be if a given job moved off the required path, and what it would be if the slowest matrix leg were sharded. That turns the report from an audit into a planning tool, and it's the version a platform team will actually schedule work against.

After that, two directions. One is multi-repo, which is where this becomes a weekly artifact for an engineering org rather than a one-off cleanup — same binary, a config listing repos, output as a committed Markdown file so the trend lives in git. The other is the inverse question: coverage gaps. If no job has ever failed on a given package, either that package is perfect or nothing tests it, and crossing the job ledger with per-path change frequency tells you which.

Watch out for

Sole-failure count is a proxy, not proof: a job that never fails alone may still be the one preventing a class of bug from ever being written, and deleting it won't show up as regret for months. Present the numbers as an argument to be made by a human, and keep the window configurable so a single bad quarter doesn't condemn a job.