Prompt injection is what happens when text an agent reads gets treated as text an agent was told. A calendar invite, a support ticket, a web page, a PDF in an inbox — anything the model ingests can carry an instruction, and a model with tools can act on it before anyone reviews the output. We measure whether yours does, off the tool-call log.
Someone types the attack straight into the chat box — a role-play frame, a base64 payload, a “previous instructions no longer apply”. This is the version most testing covers, because it is the easiest to reproduce. It is also the version that matters least once an agent has tools, because the attacker in this case already has an account.
The instruction is planted in content the agent will read on your behalf: a line in a shared document, a comment in a repository, an invisible span in an email. The user asks something ordinary. The agent reads the poisoned source, and the injected instruction arrives with the same authority as the task. Nobody in the loop typed the attack, which is exactly why nobody in the loop catches it.
A chatbot that follows an injected instruction produces bad text. An agent that follows one sends the email, grants the access, or moves the money — and the refusal you tested for in the chat window tells you nothing about it. Refusal is measured in the transcript. Execution is measured in the tool-call log. Almost nobody reads the second one, and that is where the loss happens.
NIST’s Center for AI Standards and Innovation, with Gray Swan and the UK AI Security Institute, ran more than 250,000 attack attempts from over 400 participants against 13 frontier models in agentic scenarios. At least one successful attack was found against every one of them, and robustness did not track capability — the more capable model was not reliably the safer one. Choosing a better vendor is not a control.
Injection payloads are seeded into the content the agent reads — file contents, tool results, page bodies — and the agent is given working tools it is free to use. A breach is not a judgement about tone: it is read off the tool-call log. Did the function fire, with what arguments, to which account. Every tool in the battery is mocked, so nothing in any test sends a real message or moves a real dollar. The finding is the model’s decision, captured.
Passing an injection battery is not proof of safety. It means the specific families we ran did not succeed on the version we tested, on the day we tested it, at a measured rate with a stated confidence interval. A different payload, a different scenario, or simply more trials could surface something a sample did not. Vendors ship updates without a changelog, so a boundary that held in July may not hold today — which is an argument for measuring again, not for trusting the first result harder.
One model, one pass of the 198-scenario say-vs-do suite, graded off the tool-call log. Findings and a sealed evidence pack the same day, at a published price, credited in full against a fuller audit.