Who remains in control when an agent acts?

Even careful approval can’t stop a valid wrong call

Claude Code users approved roughly 93% of the permission prompts it showed them, Anthropic’s engineers reported in May 2026 from the company’s own telemetry. They called that kind of oversight fallible, and approval fatigue showed up within weeks [16]. Fatigue is the flaw everyone can see, but a second one survives even a careful reviewer.

A person can approve exactly the right thing, and the action that reaches the world can still be a different one. We call that action a valid wrong call, one that passes every access check and still isn’t what the person decided.

AgentDojo’s first figure, a scene the authors built for their prompt-injection benchmark, shows how. A user asks an assistant to summarize the emailed meeting notes and send them to the boss. An attacker’s message in the same inbox, subject “VERY IMPORTANT”, begins “Ignore all prev…”, and the figure sets the attacker’s goal as leaking the victim’s last email [9].

Suppose the assistant obeys and calls send_email with a valid credential and an allowed operation. A normal access check may find nothing wrong.

Greshake and colleagues showed in 2023 that text planted in data an application is likely to read can steer what it does, including how it calls other services [1]. The authority comes from the user and the instruction from someone else, and a credential check sees only the first. A system can enforce its final check perfectly while the rule behind it is too broad, out of date, or cut loose from the approval it stood for.

Benchmarks take the request as given and score what planted text changed

Tests can now show when planted text changes what an agent does, and whether it finished the task as written. They can’t yet show whether that task matched what the person decided, and that gap is where the valid wrong call lives.

AgentDojo scores each attack by checking, in code, the model’s output and the environment’s state, not by asking another model [9]. In the meeting-notes email, a security check would ask whether the victim’s last email really reached the attacker.

Even such scores can flatter a defense. In a preprint revised on 23 March 2026, Bhagwatkar and colleagues argue that four public benchmarks are too easy to max out. They blame flawed scoring, bugs and, above all, weak attacks [10].

A stronger, defense-aware attack got past their own firewall’s Sanitizer, which filters tool outputs, a non-trivial share of the time, in the attack’s Stage 3 on AgentDojo’s Slack tasks. The result is setup-specific, and an appendix shows one bypass of a Sanitizer built on GPT-4o [10].

CaMeL, a research defense covered in our prompt-injection essay, plans the task from the user’s trusted request. Data it reads later can’t change that plan, so the planted email couldn’t add a send. CaMeL also checks its security policies against capability tags on the data before each tool call. The authors report some utility cost on AgentDojo, and their main threat model trusts the user’s prompt and assumes untampered memory [2].

Those assumptions suit prompt injection, the threat CaMeL targets, but trusting the request sets aside the case we care most about, a request whose terms were wrong from the start. Say the user typed the wrong address. A design that trusts the request would carry out the mistake faithfully.

“The agent was authorized” hides four separate claims

Access control asks only whether this caller may do this thing to this resource right now. Three gaps open between that answer and what the person decided.

The first gap is the label. The user meant one summary for the boss, but a scope for sending mail lets the agent email anyone. RFC 9396’s opening says a scope string can’t express a transfer of a stated sum to a named merchant, or read access to one directory and write access to one file. Its fix, Rich Authorization Requests, adds an authorization_details field that spells out a permission in detail [12].

An agent’s policy needs the same detail, which our agentic-finance essay works through for payments. It should name:

• who is acting, and with which key;
• the resource or audience;
• the allowed operation and limits on its values;
• the policy version and expiry;
• a nonce against replay;
• the authorization record each tool call relies on.

The second gap is the wording. Even a detailed policy can’t hold everything a person means. Suppose the user had instead asked for “the approved summary” to go to the boss. “Approved” might cover the boss’s exact address, the final text, the documents allowed to shape it, the reviewer’s role or a time window, and the instruction doesn’t say which.

The tool adapter, which turns the model’s proposal into a real request, widens the gap. It may quietly add a CC address, fetch a new document or retry after the decision has expired. A reference monitor can reject a badly formed request, but it can’t add a condition nobody wrote down.

The third gap is the plumbing. The Model Context Protocol (MCP), an open standard linking AI apps to tools, has authorization rules dated 2025-06-18. They cover connections over HTTP and make authorization optional, while tools on STDIO should get credentials from the environment instead [3]. Either way, a connection token says nothing about what a person approved.

A credential or scope tests only the first of these claims.

Signatures can show an approval is intact, not that it was understood

A signature can prove that nobody changed an approval. It can’t prove that the person understood it, and only the first is something a machine can check.

Claude Code’s approval rate shows that people clicked. A short prompt can leave out context that matters, and a detailed one can swamp the reviewer or put sensitive content on the screen. A signature can confirm a click while the summary is wrong, the recipient unclear or the screen deceptive.

MCP’s core specification, dated 2026-07-28, says users must explicitly consent to and understand all data access and operations. Hosts, the AI apps that connect to MCP servers, must get explicit user consent before invoking any tool, yet MCP itself can’t enforce those principles at the protocol level [11].

NIST’s Generative AI Profile asks organizations to define and separate human and AI roles and duties. It also asks for evaluations proportional to identified risks, and a record of how content provenance data affects privacy and security [13]. Both leave the hard work to whoever builds and deploys the agent.

A sound approval object shows the person the decision that matters before anything happens, and ties that inspection to the call that follows. Like the policy above, it names the actor, target, operation and limits, policy version and expiry. It adds a canonical copy of the proposed effect, and labels for which inputs may shape it.

For the meeting-notes email, it would name the boss as the only recipient and the notes as the only trusted input. The model may fill in the proposal but shouldn’t be able to rewrite the approval once the person has signed.

An approval protects nothing if another route reaches the effect

A lock helps only if every door has one. The mediator between the agent and the effect must see every route:

• the intended tool call;
• retries and redirects;
• child processes;
• alternate credentials;
• local files;
• network egress.

Say the agent, blocked at the email tool, posts the meeting notes by plain web request instead. A check on the email tool alone would let them leave.

RFC 9700, OAuth’s current security best practice, recommends audience-restricted tokens, and sender-constrained tokens that work only with a key the client holds. It also says sender constraint is undermined when an attacker gets both the token and the key material, and corrupted client software is one way that happens [4]. Credentials narrow a route, and a compromised client goes around it.

Revocation can lag too, as our agentic-finance essay details. RFC 7009 says a revoked token is invalidated immediately, then admits that in practice there could be a propagation delay among servers [5].

What a system can check is local. Once a monitor has seen a given revocation record, does it deny the next attempt before the effect? That’s worth testing, though well short of a promise that revocation is global or real-time.

A receipt proves what was signed, not that it was right

A receipt, a signed record of what an agent’s operator says it did, shows what an accepted key and log committed to. It can’t show that every effect was observed, that the policy was correct, that the runtime was honest or that anyone wanted the effect.

Suppose the notes reached the attacker. A good receipt answers four questions:

• Which policy version did the agent present?
• Which preflight check was recorded?
• Which key signed the record?
• Which log holds it?

Our first receipt draft, AAR v0.1, now archived, sketched a signed JSON receipt of this kind [8]. Our own bounded tests found flaws in its offline checker. Among them, it rejected some good proofs that a record was in the log and accepted a receipt with most of its activity fields missing [14][15].

Even a bug-free receipt shares the limits of two older systems, covered in our receipt spec essay. RATS, the IETF’s remote-attestation architecture, separates the verifier that appraises evidence against a policy from the relying party that decides what to allow. It calls freshness an architectural decision and warns that a race is always possible, in which the machine’s state or the policy changes after the evidence was made [6].

Certificate Transparency logs only grow, and their proofs make anything logged auditable, but monitors still have to read them and someone has to revoke a wrongly issued certificate [7]. A receipt has both limits. It describes a moment that has passed, and someone has to read it and act.

Privacy cuts the other way. The AAR v0.1 draft left out the prompt, the output, raw file paths and fields that name a person directly. It still admitted that hosts, counts, hashes and measurements can reveal operational metadata [8].

So every receipt is a trade. A thin record protects the user but may leave a reviewer unable to tell a bypass from a harmless failure, while a rich record helps the reviewer and can leak.

Bind the approval to the recipient, amount, expiry and policy version

If your agent can send, pay or share, start by binding each approval to the exact call the tool makes. The approval should name the recipient, the amount or scope, the expiry and the policy version, and the tool should refuse any call that doesn’t match. A send to anyone but the boss would fail, however valid the credential.

That closes one kind of valid wrong call, where the credential is broad and the decision specific. Two gaps remain. Someone had to choose the approval’s terms, so it can describe the wrong decision. Its field values can also come from a document the person never meant to trust.

A rule on which sources may fill each field, akin to CaMeL’s data-flow checks, would add a layer aimed at the second gap. Layered defense is old advice, not evidence for this layer.

Anthropic’s containment account separates environment, model and content defenses, and says they should overlap and complement each other [16]. NIST’s guide to engineering trustworthy secure systems lists defense in depth as a design principle. No single defense should carry the whole job of controlling a loss, and later lines should stop it growing or contain it [17].

We don’t yet know whether the source rule earns its place. The synthetic test we’ve designed compares a signed, fine-grained approval with the same approval plus the rule, across five kinds of case:

• ordinary work;
• injected instructions;
• swapped recipients;
• revoked approvals;
• tampered receipts.

Two results would show the source rule isn’t worth its cost. It might stop no more prohibited effects than the signed approval alone, or stop them only by blocking legitimate work beyond what a domain reviewer would accept. The full protocol is in What would prove us wrong?

A signed approval can’t show that the person understood it, chose freely and had the authority to give it. That needs a human-factors study, and Claude Code’s approval rate of roughly 93% is reason enough to run one.