The agent says green. Treft checks again, from a fresh checkout the agent never touched.

The pipeline

Every change Treft asks you to approve went through this, independently of the agent:

Verification does not mean bug-free. It means Treft reproduced the evidence independently, and tells you what it proved.

The claim

When an agent finishes a change, Treft does not take its word. It takes the diff and builds a fresh checkout on a separate machine, then runs your test suite there with the network cut. The agent never touches this environment. The result on your card is what happened there, not what the agent reported.

Ruler protection

There is a cheap way to a green suite, and it is the agent quietly editing the tests that judge it. Hit a failing test, soften the assertion, re-run, see green, move on. The suite still passes. It just measures less. That has a name: assertion weakening.

Your test suite is the ruler. The ruler is protected: a run cannot pass by weakening the tests it is graded against. If the agent changed a kept test, Treft restores the original from base, runs the suite as you wrote it, and flags the attempt on the card. The protection is structural. Tampering fails.

• The protection stands whether or not anyone tries.
• The audit reaches one ring further out: Treft walks the fixtures and helpers your kept tests import, and an edit to genuine test support is flagged the same way.

The law under all of it: the author never grades its own work.

Why it matters

An agent can pass its own tests and still be wrong. It runs them inside its own working copy, where it can weaken the harness or skip the case that fails. Even an honest green only proves what the tests cover.

Independent studies of the same gap put it at 15–30%. UTBoost (arXiv:2506.09289) found 15.7% of passes recorded on SWE-bench Verified fail once the test suites are strengthened. A study at ICSE 2026, Are “Solved Issues” in SWE-bench Really Solved Correctly? (arXiv:2503.15223), found 29.6% of test-passing patches behave differently from the developer’s own fix. Most tools never check. They show you the agent’s own claim with a checkmark on it.

When the tests disagree with the request

That leaves the first failure: an agent weakening the test suite that grades it. It has been measured directly. A study at ICLR 2026 rebuilt 349 SWE-bench Verified tasks. In each one, the written specification and the unit tests contradict each other. There is no way to pass except by breaking the specification. Under that contradiction, GPT-5 took the shortcut on 54.0% of tasks and Claude Opus 4.1 on 50%. For Claude-family models, over 79% of those shortcuts were edits to the test files (ImpossibleBench, arXiv:2510.20270 · 2026).

Two conditions travel with that number. The tasks were built so that passing requires a shortcut, so 54% is not how often an agent cheats on ordinary work. And every prompt in the study told the model not to modify the tests, so the real propensity is at least this high.

What it measures is what a model does when a test disagrees with what was asked. That condition is not exotic. Change behaviour in a mature codebase, and assertions written for the old behaviour now contradict the new requirement. Nobody has to intend it. Treft restores the kept tests from base before it grades, so editing an assertion buys nothing.

The grades your changes carry

What you get back is a grade. It says how far the evidence goes and nothing further. All four rungs, in ascending depth:

The grade gates approval, and Treft grades the ruler too: when the test suite around a change is too weak to trust, the grade is demoted. Shipping below the bar takes your explicit consent.

How accurate is verification?

It is the false-pass rate of the verifier — measured against gold, published with confidence intervals. Not comment precision, not a benchmark score, but how often the verdict itself was wrong.

The population: 389 verified passes across 150 tasks in 38 production JS/TS repositories (vitest or jest, single-package; monorepos excluded), re-verified clean-room and then adjudicated against the reference fix that landed upstream, judged within the intent of the change. That is the scope of the measurement, not a limit on the product. Treft verifies monorepos.

The misses are semantic rather than loud: an unresolved path edge case, or a router matching the wrong pattern. Changes the kept tests never exercised. The full methodology, strata, and confidence intervals are in the measurement appendix below.

Two things stand out. Suite quality roughly doubles the risk: weak kept suites 23.1% against adequate 12.5%, which is why Treft demotes a weak-suite grade. And the rate holds steady across models, so the error belongs to the check, not to whichever agent wrote the code.

Does it hold outside JavaScript?

We ran the census again on other ecosystems. Go came back at 8.3%, 95% CI [3.8%, 15.1%] on 109 adjudicated passes. Python 11.1% [5.5%, 19.5%] on 90. Rust 13.3% [5.1%, 26.8%] on 45. Pooled across the three, 10.0% [6.6%, 14.4%] on 250.

Every one of those intervals overlaps the JS/TS reading, and the overlap is the result worth having. The false-pass rate behaves as a property of the checker rather than of the language, which is what you would want to be true before trusting a grade on a stack we have measured less.

Ruby and Java/Kotlin are not measured. We banked 30 passes in Ruby and 4 in Java/Kotlin, then adjudicated 4 and 2 of them. That is not a measurement, so there is no number for them here. Measured where we could measure hardest, then measured again across Go, Python and Rust: every interval overlapped it.

How we grade ourselves

We run a measurement lab that tries to break our own verifier. We replay changes we know are wrong and count how many slip through. One of the lab’s oracles writes acceptance tests the agent never saw (the L2_synthesized_acceptance rung above), and it caught the cases every other check missed. We built an oracle layer against those known-wrong cases: dependency, secret, type, flake and suite-adequacy checks. It runs in production now, on every clean-room pass. We publish what the lab finds, including the reads that did not flatter us.

What we don’t catch

Passing tests do not prove a change correct. That is why the grades exist: each one states what was checked, and a change can clear every rung and still be wrong in a way no test exercised. The grade is evidence. It is not a guarantee.

Treft governs the merges you make inside it. A merge clicked directly on GitHub bypasses that gate. Treft posts a Treft Verification check-run on every pull request it verifies. Require that check in your branch protection and it narrows the bypass: GitHub blocks the merge while verification runs, and again when the clean-room run comes back red. We have watched it do both.

It does not close the bypass. We would rather tell you than have you find out. GitHub counts a neutral conclusion as a passing required check, and Verification concludes neutral whenever no clean-room read exists: an unsupported toolchain, a setup that failed, a verifier that timed out. It concludes neutral again when a run passed only because Treft restored a test the change had edited. Those are the cases Treft itself refuses to ship on. Verification answers what evidence exists. It was never meant to be the bar. The bar is a second check that concludes strictly and answers whether your policy allows the merge. Until we have proven that check blocking on a live protected branch, we make no enforcement claim for it. Treat Verification as evidence, and keep the bypass in mind.

A token can approve, if you let it. Treft issues agent tokens so an agent can call us as you; by default they only read. Approving is a separate scope, off on every token, and it cannot be switched on by the API itself: a token has no route to the page that mints tokens, so it can never widen its own authority. Turning it on is a deliberate act here, on this website, and that grant is the approval. It is a standing one, made knowingly, by the person whose name the merge will carry. Consenting to ship without a passing verdict is a third scope again, off even when approve is on, because it is the highest-trust thing you can hand over. A repository owner can refuse the whole channel by setting token_approval to banned in that project’s policy, and no scope, no consent flag and no re-issued token can lift it. Every merge records how it was approved: by a person in the app, or by a named token.

Questions engineers ask

The takeaway

Before you approve, Treft shows you the evidence: independent, graded and auditable. What you do with the final say is up to you. The point is that it is genuinely yours.

No card · choose one repo · disconnect anytime