Can agents read hostile information without following its instructions?

Agents should quote a hostile line, not obey it

An agent should be able to quote a hostile instruction without obeying it. We call the failure promotion, when a sentence that should have stayed evidence is treated as an instruction with authority. Our line sits inside the trusted task, which sets a template for the actions it permits. Source content may fill its checked data slots or pick a branch it already allows, but may never invent a new action, recipient, privilege or branch.

In early 2023, Kai Greshake and colleagues hid instructions in HTML comments on local test pages, so no public site carried the injection. Bing Chat, running on GPT-4 in Microsoft Edge’s sidebar, read the page the user had open. One hidden instruction told it to persuade the user to reveal their real name, a goal it could pursue for the rest of the conversation [1].

Greshake’s team called this indirect prompt injection. The attacker needs no direct line to the model, only a place in the data it’s likely to read [1]. Those comments carried nothing a user needed, so refusing them would have cost nothing.

The harder case is a vendor report, the hypothetical we follow from here, which an assistant reads to draft a response for a reviewer. One line tells the model to ignore the task, fetch another file or email an archive elsewhere. The reviewer may need that line quoted, and the report’s date may belong in the case note. A defense that refuses the whole report loses the date, and one that obeys the line sends the archive.

To a model, an order and a quotation look alike

Both arrive as text. A marker such as UNTRUSTED gives the model a hint, but it doesn’t turn the prompt into a programming language, where a parser strictly enforces what each piece may be.

What matters is how far a string reaches. It may touch content only, or reach control: the order of calls, the targets, the parameters or the privileges. Quoting the report’s hostile line touches content, and emailing the archive where it says touches control.

Zverev and colleagues formally defined instruction-data separation for single-turn models, built a measure for it and released the SEP dataset. The models they tested fell short of high separation, and the standard fixes, rewording the prompt or fine-tuning, either barely helped or cost utility. The exception was GPT-3.5 and GPT-4, which separated well after prompt engineering. The authors caution that GPT-4 may have had an unfair advantage, since the same model helped create the dataset [2].

BIPIA’s authors, who built a benchmark for indirect prompt injection, found the models they evaluated universally vulnerable. They suspect two root causes, the first that models can’t tell information to read from instructions to act on. Boundary markers and explicit reminders helped, but they conclude that such black-box defenses can’t on their own make the models resist indirect injection [3]. Both papers stop at what the model writes, before any tool call turns a misread sentence into an effect.

Evidence may shape what the agent writes, and reach an action only through a checked slot or a branch the task allows.

Untrusted text can lose its source label at every handoff

Untrusted text moves, and each step can strip the label that marked it untrusted. That’s one way promotion happens. Retrieval is only the first step:

• a search result can be inserted into a prompt;
• a tool can return prose that changes the next call;
• a tool description can steer which tool gets selected;
• an agent can write a note that becomes context later.

So the design question is whether provenance survives each step. Labeling the report when it arrives doesn’t help if its summary is later pasted into a trusted planning prompt. Memory adds delay, since the hostile line could wait in a neutral-looking saved note until it shapes a later plan.

The Model Context Protocol (MCP), an open standard linking AI apps to tools, leaves builders to decide which sources to trust. Its core specification, dated 2026-07-28, describes resources as context and data and tools as functions for a model to execute, and gives three warnings [9].

• Tools amount to arbitrary code execution and need careful handling.
• Annotations and other descriptions of what a tool does should count as untrusted unless a trusted server supplied them.
• MCP itself can’t enforce its user-consent and tool-safety principles at the protocol level.

Labels bring failures of their own. They can be wrong, maliciously issued, stale or too coarse for the decision. Even a correct label vouches only for where a sentence came from, never for whether it’s true.

Training can help a model resist planted orders but can’t prove provenance

Wallace and colleagues built an instruction hierarchy, which ranks system messages first, user messages next and third-party content such as tool outputs last. They fine-tuned GPT-3.5 Turbo to ignore lower-ranked instructions that conflict with higher-ranked ones. In the authors’ GPT-3.5 setting, the trained model resisted their evaluated attacks better, even attack types left out of training. They also report some over-refusal regressions, with the model sometimes ignoring or refusing harmless requests [4].

That result doesn’t make other treatments equivalent. An API role is a field in an interface, and a provider’s policy is run-time behavior. A provenance marker or a system prompt announcing a hierarchy is just more prompt text. Hierarchy training changes the model’s weights, so marking sources in the prompt of a model trained some other way tests presentation alone.

Better model behavior can still cut harmful proposals, and clear source labels can limit which plans get accepted. An effect monitor, a check on each real action, can reject a call the task doesn’t allow, such as emailing the archive elsewhere. But neither training nor labels authenticate where a message came from, and a monitor guards only the actions routed through it. So the separation that matters most has to sit outside the model, where effects happen.

CaMeL moves the separation outside the model, at a cost

CaMeL, a research defense, plans the task from the user’s trusted query alone, working out which steps will run and where each piece of data may go. It then applies capabilities at tool calls, and under CaMeL’s design, untrusted data the agent retrieves can’t change the protected program flow [6]. The report’s hostile line could be quoted but couldn’t add a step.

That safety can cost utility. In the paper’s June 2025 revision, with its stated security property, CaMeL solved 77% of the attack-free tasks on AgentDojo, a benchmark of agent tasks and planted attacks. The same system with no defense solved 84% [6].

CaMeL’s main threat model trusts the user’s prompt and assumes uncompromised memory. It sets some attacks aside, among them text-to-text manipulation of the answer with no forbidden data flow, such as a slanted summary. The authors find CaMeL vulnerable to side channels and report “data requires action” failures, where a task can’t be done without letting data choose the next step [6]. Its code release warns that the research interpreter likely contains bugs and might not be fully secure [11].

Wu, Cecchetti and Xiao offer a second, system-level design that labels information by trust, builds a structured plan and runs a security monitor. Its formal model assumes trusted facilities, secure communication channels between them, and the inputs and labels it specifies. Its batch security test uses a benchmark whose tool outputs are simulated, not real [10].

OpenAI says, in a March 2026 engineering post, that the most effective injections seen in practice increasingly resemble social engineering more than simple prompt overrides. Its answer is to design systems so manipulation’s impact stays limited even when some attacks succeed [12].

We’d expect that to be the hardest kind of promotion to catch. A sharper version of the report’s hostile line would skip “ignore previous instructions” and present an unauthorized step as necessary to finish the task.

A score may not hold against a smarter attacker or on another model

A benchmark score tells you how a defense did on one test, against one set of attacks. A score for promotion should count separately whether an untrusted source, such as the report, added a forbidden step or swapped an argument, say a recipient. A third failure needs its own count: a simulated effect, a harmful action in the test world. A run that finishes the task can still contain all three.

AgentDojo has 97 tasks and 629 security test cases, and checks tool outcomes against the state of its simulated world. Its authors stress that strong models fail many ordinary tasks with no attack [5]. State checks are stronger evidence than an output string, though the result still depends on the benchmark’s tasks, attacks, models and tools.

Bhagwatkar and colleagues’ preprint on an agent firewall, revised on 23 March 2026, calls its results “perfect security with high utility” across four public benchmarks. It also argues that those benchmarks are too weak [7].

Their own tables tell a messier story. On ASB, the Sanitizer, which filters tool output, still let some attacks through. InjecAgent has no utility metric, so it can’t show ordinary tasks still got done. When the authors attacked with the defense in view, in a Stage 3 cascade on AgentDojo’s Slack tasks, some attacks got past the Sanitizer, a setup-specific result [7].

Hofer and colleagues found a black-box optimizer substantially better than a white-box gradient method, across 80 task pairs in four AgentDojo domains. Injections tuned across tasks still worked on a held-out domain against Qwen3-4B, but not on that domain against GPT-5. The gradient method’s strings, tuned on Qwen3-4B, showed very low measured transfer to GPT-5 in the tested set-ups, which used one source model [8]. So an attack’s apparent strength depends on who built it, and against which model.

Three results from our unrun test would sink the checked route

We can’t yet say how often promotion happens in deployed systems. Every result cited here is tied to its own models, tasks and date, and none estimates that rate. Nor do we know whether the line holds at an acceptable cost.

We’ve designed a test to find out. It applies established information-flow controls, offers no new defense or proof, and claims no advantage over CaMeL or Wu, Cecchetti and Xiao’s design. What it adds is a way to say where a source’s influence should stop. It runs ordinary and hostile sources through four paths:

• a document;
• a tool result;
• a tool description;
• a saved note.

Each path runs under three set-ups:

• plain serialization, the text as it is;
• source-marked serialization, each piece tagged with its source;
• a checked route, letting a source’s values into approved slots only.

Any of three results would sink the case for the checked route. Source marking alone might do as well at lower cost. The checked route might win only by refusing data the task needs, or an adaptive attacker might get a forbidden step through. The full design is in What would prove us wrong?