Can AI defend systems without becoming another way in?

The attacker only has to choose the target

In 2007 Daniel Cid opened a raw SSH connection. In place of a version string, he sent a line dressed as a log message, with a made-up address in it [8]. The server logged it, with his real address at the end of the line.

DenyHosts, a tool that bans addresses it finds in that log, banned the made-up one. Then Cid put the word all where the address had been. DenyHosts wrote sshd: all to /etc/hosts.deny and locked every SSH client out.

DenyHosts used the one action it was built for, a ban. It still did the attacker’s work. Cid only had to plant a host where its loose pattern would find it. We call this the planted target, a permitted action aimed at a target the attacker planted in the evidence.

DenyHosts was not alone. Cid, who disclosed that he wrote a rival tool, reported the same kind of weakness in BlockHosts and fail2ban [8]. The National Vulnerability Database record for fail2ban, CVE-2007-4321, describes the flaw [7]. The appendix gives its details.

Now give a similar job to an AI agent. Suppose planted log lines make your own payment gateway look like the source of an attack. An agent built to block attackers would block the gateway, with a tool it’s allowed to use.

We think an AI defender that takes its targets from alerts inherits the DenyHosts flaw. We also think permitted mistakes like this one are the hardest for an AI defender to stop.

We expect the two usual guards to fail in opposite directions

Three guards could stand between the agent and that block. Two are the usual ones. The third, the sensor check, is ours.

Every verdict on these guards, here and in the next section, is our own reasoning about two hypothetical blocks. Neither we nor any source we reviewed has tested either one. Two hypotheses behind these verdicts are unmeasured too. One is that in defense a task’s own tools usually suffice for the attack. The other is that a defender’s targets often come from alerts.

The allowlist would fail open. The data-flow rule would fail closed. The sensor check might pass both cases.

A tool allowlist would let the planted block through

A tool allowlist limits the tools an agent may use on a task. AgentDojo, a benchmark for tool-using agents, tests one. The model picks the list for each task before it reads untrusted data [3]. The planted block would go through, since blocking is the agent’s job.

AgentDojo’s tasks are all general assistant work. Its authors say their filter fails when the task’s own tools suffice for the attack, as in 17% of AgentDojo’s test cases. It also fails, they say, when nobody can plan the list in advance. They credit part of its success to cases where the task only reads data but the attack must change something [3]. Containment always changes something, so we don’t expect that read-only advantage to help a defender.

A strict data-flow rule would also hold routine blocks whose targets came from alerts

A strict data-flow rule tracks where each value came from. CaMeL is one such design. It keeps the agent’s plan apart from the data it reads. It checks each value’s permissions when the agent calls a tool [4]. This guard would hold the planted block for a person, since the address came from untrusted log lines. It would also hold a routine block whose target came from an alert.

CaMeL’s design already holds some benign actions. In its own example, the agent is about to email a document to an address from untrusted meeting notes. If the recipient can’t already read the document, the agent asks the user first. It asks even without an attacker [4].

If a defender’s targets often come from alerts, a strict data-flow rule sends routine work to a person, and containment becomes a queue.

A check against the sensor’s own record might pass both cases

The sensor check is a rule outside the model. Before the agent blocks an address from an alert, the rule looks for it in the network sensor’s own records of the event, such as flow records. If the address isn’t there, the rule refuses the block or sends it to a person, however convincing the alert.

Here is an illustration, with made-up addresses. Suppose planted log lines say the attack came from 10.0.0.5, your payment gateway. Suppose the sensor’s flow records of the attack traffic show 203.0.113.7 as its source and don’t name 10.0.0.5. A block of 10.0.0.5 would find no match in those records, so the rule would hold it. A block of 203.0.113.7 would find a match, so it could go ahead.

Cid’s proposed fix for DenyHosts offers an analogy. It anchors one pattern to the end of the line, where sshd wrote the real address [8]. Like the sensor check, it relies on a part of the line the attacker can’t write. Alerts have no fixed end of line. A field the sensor filled in itself is the nearest thing. That’s the design we’d bet on.

The check has gaps. An attacker who can forge a packet’s source address reaches the sensor’s record too. So does one who controls a client-supplied field, such as a forwarded address. A rule strict enough to guard key assets would sometimes refuse to block a compromised one. And PACT’s authors, whose design tracks where each tool input came from, name inferring a value’s origin as an open problem, one our own check shares [10].

Human review of every block wouldn’t fix the planted target either. The reviewer would see the same planted evidence, in a polished recommendation.

Judge any guard on the work it stops

Whichever guard a team picks, judge it on the work it stops as well as the attacks it stops. A rule that refused every block would never block the gateway. It would never block a real attacker either.

AgentDojo shows how big that cost can be. It tests a detector for injected instructions, a kind of defense that is not one of our three guards. In its Table 5, with GPT-4o against the authors’ strongest attack, the detector cut attack success from 57.69% with no defense to 7.95%. The share of normal tasks the agent finished fell from 69.00% to 41.49% [3]. That’s about two-fifths fewer finished tasks.

A defender can’t simply do without a guard, though. Attackers write some of what it reads, such as a hostname in an alert or a web page quoted in a ticket. In one benchmark of indirect prompt injection, every model tested sometimes followed instructions planted in the content it read. Its authors suspect models struggle to tell data from instructions [2].

No source we reviewed measures the trade-off that matters here. A defender needs two numbers. One is how often planted evidence aims an allowed block at the wrong host. The other is how often a stricter rule refuses a correct target because it looks unusual. The closest study feeds hostile log text to a security assistant, but it scores only labels, summaries and advice, never tool calls [11].

One experiment could prove us wrong

The experiment would test our central claim, that the sensor check beats both usual guards. Seed containment jobs with correct targets that look unusual and planted targets that look routine. Give the same jobs to a tool allowlist, a strict data-flow rule and the sensor check, and change nothing else. Score each guard on fresh cases it wasn’t tuned on.

We’d rather see four numbers than one. They are work done, forbidden changes made, false blocks and the cost of review. Our proposed studies page has the full design.

Two results would count against us.

• If either usual guard does as well at similar cost, our check adds little.
• If human reviewers supply all the benefit, the rule does less than we claim.

We think any AI defender that blocks whatever its alerts name is open to the kind of attack that beat DenyHosts in 2007. Treat every target an alert hands your agent as text an attacker may have written.