Reproduce flaky tests on purpose

The idea

A command-line tool that hunts flaky tests instead of waiting for them. You point it at your test suite and it reruns the suite many times under deliberate perturbations — shuffled test order, varied parallelism, skewed system clock, injected filesystem and network latency, tightened timeouts — and records which tests fail under which conditions. The output is not "this test is flaky", but "test_invoice_totals fails 40% of the time when it runs after test_currency_cache with --jobs 8", plus the exact command to reproduce it.

Why build this

Every team past a certain size has a handful of tests that fail perhaps one run in thirty. Nobody can reproduce them locally, so they get retried, then retried automatically, then eventually skipped. The cost is not the CI minutes; it's that a retry policy quietly converts real intermittent bugs — order dependence, unawaited promises, shared fixtures, timezone assumptions — into invisible ones. Teams either ignore flakes or throw them at whoever is unlucky enough to be on build duty.

The reason to build this now is that the pieces are cheap. Modern runners all emit machine-readable results (JUnit XML or JSON), all support seeded random ordering, and container runtimes make it trivial to run the same suite dozens of times with different environmental knobs. What's missing is the thing that orchestrates the search and narrows a failure down to a minimal reproducer, which is a well-defined engineering problem rather than a research one.

Stack sketch

• Go for the CLI — a single static binary, easy to drop into a CI image, with good process and container control.
• Result parsing from JUnit XML plus native JSON reporters (pytest --json-report, go test -json, Jest --json, cargo nextest --message-format). JUnit XML is the fallback for anything else.
• SQLite for the local flake ledger: one row per (test, run, seed, config, outcome, duration). Everything downstream is a query.
• Docker or Podman for isolated reruns, with --cpus and --memory to reproduce the "CI box is slower than my laptop" class of failure.
• libfaketime for clock skew; tc netem inside the container for latency and packet loss.
• Delta-debugging over the test ordering — a ddmin-style bisection that shrinks a 900-test failing order down to the two or three tests that actually interact.
• GitHub Actions and GitLab CI reporters that post a summary comment with the top offenders and their reproduction commands.

Scope for v1

In:

• flakehunt run --times 50 -- pytest — repeat the suite, capture results, write to SQLite.
• Order shuffling with recorded seeds, plus parallelism sweeps.
• A flake score per test: failure rate, plus whether failures cluster under a specific configuration.
• Order-dependence bisection for the single worst test.
• flakehunt report — a terminal table and a Markdown export.

Out:

• Language-specific instrumentation or coverage analysis.
• Any attempt to fix the test.
• A hosted dashboard or account system.
• Distributing reruns across machines.

Where it could go

The obvious next step is history. Once the ledger has a few months of runs, flakiness becomes a trend you can regress on: which commit introduced the flake, which tests are getting worse, which are stable enough to un-quarantine. That's the argument for a lightweight server that ingests ledgers from CI runs and keeps the long view, while the CLI stays the thing that does the work.

Beyond that, the interesting direction is root-cause classification rather than detection. The perturbation that reproduces a failure is itself a strong signal: fails only under clock skew means a timezone or TTL bug, fails only under high parallelism means shared state, fails only when it runs after a specific test means fixture leakage. A small classifier over that signal — even a rules table before any model gets involved — turns the report from a list of suspects into a list of likely causes. There's also a natural pairing with quarantine policy: a tool that can prove a test is order-dependent has earned the right to auto-open an issue with the reproducer attached.

Watch out for

The rerun budget is the whole product — fifty runs of a slow suite is hours, so you need smart sampling (rerun only the tests that have ever failed, bisect lazily) or the tool is too expensive to leave on. Also, some "flakes" are genuine product bugs under load; be careful that the report frames them as findings, not as tests to suppress.