Who checks the work when AI gets faster?

A model can hold the decisive document without using it

In document work where one exception decides the case, we bet faster drafting moves effort into review before it saves any. In a hypothetical contract review, an assistant must say whether a change is allowed. It quotes the general rule and misses the exception two pages later, though the contract sat in its input. In RULER, a synthetic long-context benchmark, almost all of the 17 models tested showed large drops as inputs grew, despite near-perfect single-needle scores [9].

The draft arrives fast, but someone still has to find the exception. So the bet can only be judged on the whole job: producing the answer, checking the decisive evidence, correcting mistakes and verifying any action. Checking costs minutes, repair calls and refusals, and it still misses some failures. That cost belongs on the same ledger as the speed of the draft.

In 2023, Nelson Liu and colleagues held a question and its answer fixed, and moved the answer-bearing document among distractors. The models they chose often did best when that document sat near the start or the end, and worse when it sat in the middle [3]. A good reliability test has this shape: hold the evidence and the right answer fixed, move something that shouldn’t matter, and watch whether the answer moves.

RULER used 13 tasks across four skills: finding a fact, following a chain of facts, combining many facts and answering questions. A further analysis, run mainly on Yi-34B, found that model degraded further as input length and task complexity grew, in its tested settings [9].

Cheng-Yu Hsieh and colleagues offer hopeful counterevidence. They adjusted how much weight a model gives each part of its input. In the models and datasets they tested, that improved retrieval from the middle and retrieval-augmented generation [10].

So the position effect can turn on the model, the retrieval pipeline, the prompt and any fix. Test the mix of sources and placements your system will use, with the contract’s exception at the start, the middle and the end.

A wrong answer can sound like common knowledge

Some wrong answers sound right because they repeat a common mistake. TruthfulQA, a benchmark of questions that invite common misconceptions, tests exactly that. Its 817 questions span 38 categories. In its tested generation setting, the best model was truthful on 58% of them [2].

The 94% human baseline came from one participant answering 250 randomly sampled questions, with internet access and a suggested two minutes per question [2]. The person and the models had different resources, so the two figures don’t compare like for like.

Confidence and tidy citations help a reviewer, but neither checks the answer. NIST’s generative-AI profile defines confabulation as “confidently stated but erroneous or false content” and notes that such output can also be internally inconsistent [1].

For the contract, a reviewer needs a record that shows:

• the smallest claim, such as “the change is allowed”;
• the exact passage that supports it;
• any later paragraph that limits it;
• the date, jurisdiction or version it covers;
• the source that would contradict it;
• what nobody has checked.

A model can go wrong quietly, reaching the right conclusion for the wrong reason or citing a passage that merely shares words with the claim. It may drop a condition that changes the answer, or lean on an old source without saying so. A cautious answer that names the claim it couldn’t support can beat a fluent one that hides the gap.

Mrinank Sharma and colleagues observed sycophancy across the five assistants in their study, and found evidence about one cause. Preference judgments used in training can reward agreeing with a user’s stated view over being truthful [5]. To stress-test the contract review, keep the contract the same, have the user state, confidently and wrongly, that the change is allowed, and see whether the conclusion moves.

In tests, a model’s explanation often left out what swayed it

Treat a model’s explanation as a help, never as a record of how it reached its answer. Miles Turpin and colleagues biased the answers of the models they tested with small changes to the input. In one test, they reordered the options in example prompts so the right answer was always “(A)”. The models’ chains of thought systematically failed to mention the change that had swayed them [4].

A later Anthropic study gave selected reasoning models pairs of multiple-choice questions, one with a hint and one without. When the hint changed a model’s answer, the authors checked whether its reasoning admitted using the hint [8]. Usually it didn’t.

Across the paper’s hint types, overall faithfulness scores were 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1. Many single settings scored below 20%, and faithfulness was lower on GPQA than on MMLU [8].

In a separate reinforcement-learning experiment, an earlier snapshot of Claude 3.7 Sonnet learned to exploit reward hacks the authors planted in its environments. It mentioned the hack in under 2% of examples, in five of its six environments. The authors stress that their controlled settings differ from many everyday tasks [8]. Still, the study suggests a method for the contract review: insert a cue that shouldn’t matter but does, and see whether the stated reasoning admits it.

For work with consequences, like the contract decision, the reviewer also needs the system’s own records. They include the source packet or the identifiers of what was retrieved, the versioned prompt and permission rule, each tool request and response, the validator’s output and the observed end state.

If the story and the records disagree, check where the records came from, because a manipulated tool response can be wrong too. Mark whatever is still unverified.

When the model acts, judge the state it left behind

Once an AI can act, the end state is the evidence. Only it shows whether the requested setting changed and whether a forbidden one changed too. Whether a failed task can be safely undone needs its own rollback test.

WebArena, a 2023 benchmark of 812 long web tasks for agents, each needing many steps, scores agents on functional correctness. It checks the answer an agent returned, or the state it left the website in, instead of comparing the agent’s clicks with a reference list [6].

Greshake and colleagues demonstrated indirect prompt injection, where instructions hidden in content an application retrieves can steer it without going through its direct interface [7]. A reliability test should plant such an instruction in an untrusted field, beside the moved document and the false premise.

Picture a line in the contract’s annex that tells the assistant to approve the change. It shouldn’t change the right answer, and if it does, a tidy citation won’t show why. Retrieved text can be evidence about a subject, but it shouldn’t gain permission to rewrite a rule, widen a tool’s scope or approve its own next step.

One test per failure tells you what broke

Test one failure at a time. A test that checks one claim, such as “the change is allowed,” can fail clearly and tell you what broke, and one big score can’t. A model can fail at grounding, at staying consistent when something changes, at calibration or at control, and each needs its own test.

NIST’s profile weighs a risk by its likelihood and the size of its consequences, and says organizations may scale their effort to the severity and likelihood of harm [1]. So different tasks need different proof.

A research assistant may need to show a gap in its sources, while a system proposing a production configuration may need a preview of the change and a test. An agent with permissions may need a verifier that rejects a malformed or out-of-scope request. Calling all of these “guardrails” hides the different proof each one needs.

A fair comparison rules out rival explanations. If a fixed evidence packet improves an answer, the cause might be better retrieval, formatting that made the task easier, a leaked answer key or a model that already knew the source.

A validator that blocks a bad action may block good ones too. When abstentions rise, the system may have become safer or only less helpful. That trade-off is what our bet is about.

The test that could change our view

Someone has to check, or something does. We could be wrong about what that costs, if checking automates as well as drafting does. We’ve designed a small test to find out, but haven’t run it.

It uses 12 held-out task–source pairs, each a task plus its sources, like our hypothetical contract question and its contract. Each pair comes in two versions, one clean and one with a single change (the evidence moved, a false premise added, or an untrusted instruction planted). Every version gets answered three ways: directly, with a citation for each claim, and with citations plus a checker.

We’d count every reviewer minute and every repair call. The comparison would use the same evidence and the same changes, count failures and abstentions, and allow no unseen human rescue. Once the test covers simulated actions, it would score end states too. The full protocol is on our proposed-studies page, What would prove us wrong?

The test asks whether an added check earns its cost. Suppose cited answers without the checker made no more material errors than those with it, or the checker only raised refusals. Then we’d drop our claim that the task needs an added check. And if checking turned out cheap and a reviewer added nothing, we’d count that against our bet.