Meta puts LLM-generated bug-catching tests into production
Meta says its Catching JiTTests, LLM-generated tests built to fail on buggy diffs, now run in production with fewer false positives.
Meta says its LLM-generated "Catching JiTTests" have moved from research prototype to production infrastructure, according to Meta's engineering blog (October 29, 2026). An AI reviewer reads a diff. This system instead generates executable tests designed to fail when a change alters behaviour. Teams shipping AI-written code can borrow the pattern.
Meta says the system runs overnight on spare capacity against the previous day's diffs that were scored as risky. Two workflows generate the tests:
- An intent-aware workflow infers what the diff is meant to do, lists the risks and builds mutants from them.
- A "dodgy diff" workflow treats the diff itself as a mutant.
Failing tests become "weak catches." An ensemble of LLM judges and a rule-based assessor called RubFake rank them. Engineers are contacted only when a candidate ranks highly, which Meta calls human-on-the-loop. Meta says the approach takes more compute than LLM review but catches bugs that review can miss.
Meta reports that 589 engineers (64% of its outreach) engaged with the system and reported a 60% true-positive rate. It also says that in the first half of 2026 the system played a key role in catching 74 serious privacy bugs before they landed, and privacy SEVs fell 20%. Sam on AI notes these are Meta's own deployment figures, not an independent estimate. It also points out that a failing test proves behaviour changed, but a human still decides whether that change is a bug.
Meta's post doesn't name the models it uses, give a per-diff cost or share code. It points to its FSE 2026 paper for details. For an off-the-shelf option, QA.tech reports that Greptile's TREX beta writes and runs targeted tests in a sandbox.