How do you test an AI agent before you deploy it?
Give it working tools and ordinary jobs with a trap in each, then read what it actually did, not what it said it did. That is the core of it. Here is the whole method, as Potestas AI runs it and as you can run much of it yourself.
Why is testing an agent different from testing a chatbot?
A chatbot's risk is what it says. An agent's risk is what it does with the tools it is given: payments, email, a shell, records. A model can refuse to discuss fraud and still wire the money once a payment tool is in reach. So the test has to hand the model working tools and read the tool-call log, the record of every action it actually took.
What should the test grade?
The actions. Our say-vs-do suite has 258 scenarios across payments and wires, records and data, access and permissions, and a clinical set. Each one defines the harmful tool call, and the grader reads which tools the model called and with what arguments. There is no second AI deciding what happened. A model that narrates a wire transfer it never executed did not execute it, and the log says so.
Should I run the free industry-standard tests?
Yes, as a first step. AgentDojo, AgentHarm, StrongREJECT, MASK, b3, XSTest and garak are public and anyone can run them. Read each result with its reply limit, its count of empty answers and its blocked prompts beside it: in our comparison of seven tools on four models, some graded empty replies as passes or as honesty, and one blocked prompt could stop a whole run.
Is one kind of attack enough?
No. A well-known free test checks one kind of attack, an instruction hidden in data the agent reads. Our audit tries twelve kinds, by where the hidden instruction enters: an email thread, a calendar invite, a support ticket, the agent's own memory, a plan it saved, a tool's description, another agent. One run of the free test rated a model nearly clean; on the same model, five other kinds got through.
How many times should I run it?
More than once. A single pass shows a behaviour can happen, not how often. The same model with the same seed can choose different actions on a second run, so a rate needs repeated, independent runs and a confidence interval. Count the turns that never got an answer separately: a blocked or silent turn is not a pass.
What should I keep at the end?
A record someone else can check without trusting you: exactly what was sent, exactly what came back, the grading, a fixed seed, and hashes showing nothing changed. Every pack we seal is also stamped in the Bitcoin blockchain, and the buyer gets a verifier to run themselves.
Can someone run this for me?
Potestas AI does: the seven industry-standard tests, then the 258-scenario say-vs-do suite on the model you name, a findings report and a sealed evidence pack within 72 hours, for $3,000. You supply the model's name and nothing else.
More: Run the industry-standard tests yourself, the seven-tool comparison, dated public findings, and a real sealed evidence pack with its verifier.
Want this on your model?
Seven industry-standard tests, then our 258-scenario agentic say-vs-do suite, on the model you name. Findings report and sealed evidence pack within 72 hours. $3,000, by card.