The agent says green. Treft checks again—from a fresh checkout the agent never touched.

The pipeline

Every change Treft asks you to approve went through this, independently of the agent:

Verification does not mean bug-free. It means Treft reproduced the evidence independently and tells you exactly what was proven.

The claim

When an agent finishes a change, Treft does not take its word. It takes the actual diff, builds a fresh checkout on a separate machine, applies the patch, cuts the network, and runs your test suite. The agent never touches this environment. The result on your card is what happened there, not what the agent reported.

Ruler protection

The cheapest way to a green suite has a name: assertion weakening — the agent quietly editing the tests that judge it. Hit a failing test, soften the assertion, re-run, see green, move on. The suite still passes. It just measures less.

Your test suite is the ruler. The ruler is protected: a run cannot pass by weakening the tests it is graded against. If the agent changed a kept test, Treft restores the original from base, runs the suite as you wrote it, and flags the attempt on the card. The protection is structural, so tampering fails even when attempted.

• 0/56 runs modified a kept test in the audited corpus — and the protection stands whether or not anyone tries.
• The audit reaches one ring further out: Treft walks the fixtures and helpers your kept tests import, and an edit to genuine test support is flagged the same way. In the same corpus: zero.

The law under all of it: the author never grades its own work.

Why it matters

An agent can pass its own tests and still be wrong. It runs them inside its own working copy, where it can weaken the harness or skip the case that fails. Even an honest green only proves what the tests cover.

Independent studies of the same gap put it at 15–30%. UTBoost (arXiv:2506.09289) found 15.7% of passes recorded on SWE-bench Verified fail once the test suites are strengthened. A study at ICSE 2026 — Are “Solved Issues” in SWE-bench Really Solved Correctly? (arXiv:2503.15223) — found 29.6% of test-passing patches behave differently from the developer’s own fix. Most tools never check. They show you the agent’s own claim with a checkmark on it.

The grades your changes carry

A Treft result is a grade, not a checkmark. Each grade names exactly what was established and nothing more — all four rungs, in ascending depth:

The grade gates approval, and Treft grades the ruler too: when the test suite around a change is too weak to trust, the grade is demoted. Shipping below the bar takes your explicit consent.

How accurate is verification?

Human review still found semantic issues in 15.4% of clean-room passes — measured on JS/TS repositories, our July 2026 census. That is why Treft reports evidence depth instead of issuing a magic green check.

Say what kind of number that is: the false-pass rate of the verifier — measured against gold, published with confidence intervals. Not comment precision, not a benchmark score — how often the verdict itself was wrong.

The population, stated plainly: 389 verified passes across 150 tasks in 38 production JS/TS repositories (vitest or jest, single-package — monorepos excluded), re-verified clean-room and then adjudicated by blind human review. Measured where we could measure hardest; the six-language expansion is next to be measured. Representative misses are semantic, not loud: an unresolved path edge case, a router matching the wrong pattern — changes the kept tests never exercised. The full methodology, strata, and confidence intervals are in the measurement appendix below.

Two readings inside it worth naming plainly: suite quality roughly doubles the risk (weak kept suites 23.1% vs adequate 12.5% — which is why weak-suite grades are demoted), and the rate is stable across models — the error belongs to the check, not to whichever agent wrote the code.

How we grade ourselves

Treft runs a measurement lab that tries to break its own verifier. We replay changes we know are wrong and count how many slip through. One of the lab’s oracles writes acceptance tests the agent never saw (the L2_synthesized_acceptance rung above), and it caught the cases every other check missed. The oracle layer built against our known-wrong cases — dependency, secret, type, flake, and suite-adequacy checks — now runs in production on every clean-room pass. We publish what the lab finds, including the reads that did not flatter us.

What we don’t catch

Passing tests do not prove a change correct. That is why the grades exist: each one states what was checked, and a change can clear every rung and still be wrong in a way no test exercised. Treat the grade as evidence, not absolution.

By default, Treft governs merges that go through Treft. A merge clicked directly on GitHub bypasses that gate. Treft posts a Treft Verification check-run on every pull request it verifies, and requiring that check in your branch protection narrows the bypass: GitHub blocks the merge while verification runs and when the clean-room run comes back red — we have watched it do both.

It does not close the bypass, and we would rather say so than let you find out. GitHub counts a neutral conclusion as a passing required check, and Verification concludes neutral whenever no clean-room read exists — a toolchain we do not support yet, a setup that failed, a verifier that timed out — and when a run only passed because Treft restored a test the change had edited. Those are the cases Treft itself refuses to ship on. Verification answers what evidence exists; it was never the bar. The bar is a second, strictly-concluding check that answers whether your policy allows the merge, and until it is proven blocking on a live protected branch we make no enforcement claim for it. Treat Verification as evidence, and keep the bypass in mind.

Questions engineers ask

The takeaway

Treft makes the evidence visible, independent, graded, and auditable — before you approve. What you do with the final say is yours. That it is genuinely yours is the point.

No card · choose one repo · disconnect anytime