Can AI work with sensitive information without exposing it?

Giving the assistant less can make the legal answer wrong

Give an AI assistant less of a confidential agreement, and less can leak; give it too little, and the legal answer can go wrong. A clause extractor given one paragraph may miss a definition two pages earlier. In our hypothetical, counsel asks an assistant for the termination obligations in a confidential supplier agreement. We map that one task as five handovers, each needing its own answer on what crosses, who gets it, why, and what would show a mistake.

To answer well, the assistant may need the relevant clauses, their exceptions and the effective date. It probably doesn’t need every negotiation comment, the unrelated schedules or the bank details. We call that the smallest useful slice, the least of the matter that still lets each step do its job well. Our provisional view is that the assistant should get that slice, and every other system in the workflow no more than its own job needs.

One task can pass the agreement through five hands

Any handover can carry part of the agreement across what our essay on cheap inference calls the boundary, the point where data first becomes readable to someone else. The table follows the agreement from source to reader. Not every product has all five steps, and the steps are categories for analysis.

Each row gives the question a handover must answer, and the leak a storage claim would miss.

Our step-by-step map draws on two NIST documents. The Privacy Framework treats privacy risk as part of a firm’s wider risk management, and says managing it isn’t suited to one-size-fits-all solutions [6]. The zero-trust architecture grants no trust just because of where a request sits on the network or who owns the device, and focuses on protecting resources [7].

The Privacy Framework deals with problems people could face, and the agreement raises commercial confidentiality on top of any personal data in it. So we borrow the framework’s attention to each processing step, without claiming it covers commercial secrets. Our map lets a team test its own promise. If it can’t name the model provider, retrieval index, tool endpoint, memory layer and diagnostic path this task uses, it hasn’t yet found every place the promise could fail.

“We do not retain prompts” says nothing about who saw them

“We do not retain prompts” and “the information was not exposed” are claims about two different moments. Retention asks whether someone keeps data after a run, for how long and where. Disclosure asks who or what got the data in the first place, and a system can keep nothing and still show a secret to the wrong party.

While the agreement is read, a model service may process a prompt briefly, a tool may get a parameter and an error collector a copy. A reader allowed to see the answer may still see a sensitive inference in it. An empty store, checked later, can’t speak for any of those moments. We keep four things apart: what someone saw being sent, what a provider says it keeps, what someone confirmed was deleted, and the paths nobody observed.

Limits on keeping data still matter, but they need a clear scope. The Internet Architecture Board wrote privacy guidance as an RFC, and the distinct threats it names include disclosure, correlation, identification and secondary use. It lists data minimization as one of three kinds of mitigation, beside user participation and security [10].

The RFC writes about individuals, but the same threats apply by analogy to a business. A host name can identify a supplier, and a faithful summary can give away the commercially important part of the agreement.

In MCP, the open standard linking AI apps to tools and data, hosts are the AI apps that connect to MCP servers. Its dated specification’s key principles say hosts must obtain explicit user consent before exposing user data to servers, and must not send resource data elsewhere without it. The same specification says MCP itself can’t enforce those principles at the protocol level [8]. Which messages go out is up to the host, provider, tool adapter, access-control system and operator.

A file can leak in at least four ways

A file can leak because the model memorized it, the model repeated retrieved documents, untrusted text steered the system’s calls, or a tool held more permission than the task needed. Each path needs its own test, and treating model access, retrieval and tools as one privacy risk hides most of them.

Start with memorization. In 2019 Nicholas Carlini and colleagues introduced an exposure metric for rare training sequences, and showed that generative models can memorize secret values without anyone intending it. Of the defenses they tested, only differentially private training eliminated the measured issue, at some cost in utility [1].

Their 2021 USENIX Security paper used only black-box queries to extract hundreds of verbatim sequences from GPT-2’s training data, public personally identifiable information among them. It also found that memorization doesn’t need overfitting. Some worst-case training examples were memorized, although on average training examples had barely lower loss than test examples [2]. These results hold for the models, corpora and attacks the authors chose.

In that paper’s Figure 1, a short prefix goes into GPT-2, and out comes a real person’s full name, email address, phone number, fax number and physical address. The paper prints them partly masked, with the person’s permission. They had been posted as a software project’s contact details, and appear in just a few documents online [2].

The authors call that reproduction, outside the setting where the details were shared, a breach of contextual integrity. They warn that a chatbot could likewise give one user’s phone number in answer to another user’s question [2].

That leak came from memorization in training, and the agreement reaches the model as input at work, so memorization isn’t the mechanism we study. The point about context still carries over, since each handover of the agreement is a new context.

The second path, a model that repeats retrieved documents, moves where the risk begins. In a v2 preprint of 30 March 2025, Yuefeng Peng and colleagues describe what amounts to a supply-chain attack on RAG. Poisoned fine-tuning data plants a backdoor in the generator model, and once triggered, the system reproduces documents from its retrieval database, verbatim or paraphrased. They report results on four datasets and three models [4].

The attacker must first poison the generator’s fine-tuning data, and the main threat model assumes a benign retriever and knowledge database [4]. For the agreement, we infer that a store the model reads is only as safe as that model’s fine-tuning history.

Indirect prompt injection, the third path, opens another way out. Kai Greshake and colleagues planted text where a system would likely fetch it. That text could change how an LLM-integrated app behaves, including whether and how it calls other APIs [3]. Every such call is a chance for part of the agreement to leave.

CaMeL, from Edoardo Debenedetti and colleagues, answers with design. It works out control and data flow from the trusted user query, so untrusted retrieved data can’t change the program’s flow. It uses capabilities, machine-checked permissions, to enforce policies on tool calls and block unauthorized data flows. In the authors’ comparison on AgentDojo, a benchmark of agent tasks under attack, that security cost some utility [5].

The cited June 2025 v2 assumes trusted user instructions and uncompromised memory in its main threat model. It leaves out text-to-text attacks that cause no prohibited data flow, such as a distorted summary [5]. The paper supports routing data only where a policy allows, under those assumptions.

Sending less can drop the fact the answer depends on

“Send less, keep less, grant less” sounds simple until it harms the work. Every cut to what an assistant sees can also cut what it gets right.

The clause extractor in our opening is one case. A tool given a hash instead of the supplier’s name may be unable to check a public filing. A system that keeps no lasting task state may repeat work, lose its record of fixes or fail to give the continuity a user asked for. An answer stripped of all sensitive detail may be too vague for the decision it was meant to support.

So the unit of minimization is the task. For counsel’s question, search may need a few clauses instead of the whole agreement, and a drafting tool a structured fact or citation instead of the original. A diagnostic record may need an error type instead of the prompt.

Other controls follow the same rule: retrieval limited by field and by matter, separate handling for trusted instructions and untrusted retrieved text, and allowlisted tools whose parameters are limited to their purpose. Credentials should be short-lived, and memory rules should tell continuity a user asked for apart from incidental chat history. Each control changes one row of the map, and has to be tested there.

Sometimes the text the task needs is too sensitive for the system at hand. Then the right outcome may be human review, a different technical arrangement, a precomputed representation, or no automated processing at all. A good privacy design makes that stopping point visible, and doesn’t stretch a weak control into a broad promise. To find it, a team has to say plainly how the agreement could leak.

Write the threat model, then six answers for every system

Before the next confidential matter goes into an AI assistant, write a short threat model. For the agreement, the sensitive material may include clause text, counterparties, the matter’s identity and the negotiations. The forbidden effects may include sending a clause to an unapproved host, writing it into memory shared across matters, putting it in a diagnostic payload or showing it to another user.

The cause could be a curious legitimate user, a compromised tool, a malicious document author, a provider operator with service access or a misconfigured component. The model shouldn’t be the only check between retrieved text and a call that matters. Review and test the parts in between, which usually include access controls, the retrieval filter, model client, policy decision point, tool adapter, credential store, effect interceptor and trace collector.

Don’t stop at a tool’s documentation. Our own static review of MCP server packages read their code without running it. It found third-party endpoints in package code that a check of the packages’ own documentation didn’t find [9].

Even a clean review may miss the path that matters, such as a retry queue, an SDK trace or a human support path. “Not observed” has to stay visible as a status of its own, and must never quietly become zero.

Then write down six answers for each system that will touch the matter.

• Which part of the matter does it need?
• Which person or system receives it?
• Which task justifies sending it there?
• What keeps it from being reused elsewhere?
• What is retained?
• What evidence would reveal a mistake?

The first answer is that system’s smallest useful slice. Together the answers separate an assistant preparing a note for counsel from an unrelated tool, log or future conversation. If a line stays blank, that system shouldn’t get the matter yet.

We haven’t measured what tighter limits cost the answer. The test we’ve designed but not run would try the same synthetic legal task three ways:

• with broad context;
• with task-minimized retrieval, memory and tool payloads, cut to what the task needs;
• with that minimized handling plus a check before each simulated effect.

It would score leaked facts and complete answers together, and its schedule and scoring rules are in our proposed studies. Three findings would change our position:

• the tighter versions don’t reduce unwanted disclosure;
• they work only by blocking correct answers;
• other evidence shows them leaking through paths the test can’t observe.