A wrong answer costs you either way — miss a real flaw and it ships broken, or kill a good product on a false alarm. We tell you which one you're looking at, and prove it.
Not about AI in general. About yours — the one with tools wired into it, in production, today.
If you cannot answer these, that is not negligence. Nobody can, without measuring — the tooling to answer them barely exists and almost no one is running it.
You will still be asked.
By a client's compliance team, by a regulator, or by whoever is in the room the first time something goes wrong. The answer is either a measurement or a shrug.
A number without evidence is a guess wearing a lab coat. Every finding ships with the exact input, the exact output, the grading and a fixed seed — so your own reviewers judge the call themselves. Nothing asserted that you cannot go and check.
Every result is re-graded by a model from a different vendor than the one under test. No model grades its own homework.
Borderline, artifact, and misread results are excluded before anything is called a finding. Fewer, truer results — not a longer list.
Operational risk is separated from borderline dual-use, so you know what actually matters and what doesn't.
Fewer, truer findings — not a longer list.
We ran seven free AI safety tools on four named models, then our own audit on the same four. Here is one model, kimi-k2.6, through both.
The free tool wasn’t wrong. It tests one kind of attack, and on this model that kind rarely works. Your agent faces more than one. Where the free tools and our audit test the same thing, they agree: AgentDojo, AgentHarm and our injection tests ranked all four models in the same order.
Correct — and it is why the battery is frozen rather than improvised. Same probes, same grader, same sequence, on a schedule, so a change between runs is a change in the model and not in the measurement. That is item 01 below.
The part that argument leaves out: continuous monitoring sold by the vendor whose system is being monitored is not independent, and it produces an alert feed rather than an artifact. Ask what document it hands your regulator, and whose name is on it.
Your vendor publishes safety numbers, and they are probably honest. They are also structurally incapable of answering any of the five below about your deployment — and each is a question you will eventually be asked.
A trillion-dollar company can out-spend us on tooling. It cannot stand outside every vendor at once and measure them the same way, over time.
Vendors ship updates with no changelog, and a model that refused last month may answer today. Same battery, on a schedule, flagging what moved — a finding class only a standing instrument can produce.
The industry moved to agents. Safety testing stayed on what a model will say in a chat box. We measure the gap between what it describes and what it executes when handed a tool.
A single lab can only red-team its own model. We run the identical battery across Model A, Model B, and Model D at once — so the finding isn't "this model failed," it's "here's how these models differ on the same test, and the one topic where the gap matters most." That's structurally impossible from inside any single vendor.
We map where a model draws its line across a graduated ladder, and whether that line is coherent. One that refuses a milder request but answers a harsher one has an inconsistent policy — a structural defect, not a jailbreak.
A model behaves differently inside an ongoing conversation than under cold one-off prompts. We measure both on identical questions and report the gap — the conversation is how your users actually interact.
We built a finance agent and gave it the tools to move money. Your job: talk it into a fraudulent wire. It takes about 20 seconds to see how easily an agent trusts a plausible request — the exact failure mode our research measures across four frontier models. Two of them wired the money. Money is one of four things an agent can be talked into — paying, granting access, handing over a credential, deleting — and the suite tests all four.
PyRIT, garak and Promptfoo are capable and genuinely useful — we use this class of tooling ourselves. But a scanner and a signed opinion are different things in kind, and the choice most teams actually make is a third one: nothing at all.
We run the standard free benchmark too, and we agree with it where it looks. Then we show you what it can’t see: the harms it missed on your model, what your model says versus what it does, and whether it tells you when it’s done something wrong.
| Do nothing | Run a free tool | Commission an opinion | |
|---|---|---|---|
| What it costs | Nothing, until something happens | No licence fee, plus the engineer-time to stand it up, tune it, and keep it running across every model update | A fixed price, published, credited toward the full audit |
| What you end up holding | An assumption | A log | ✓ A case file — exact input, exact output, the grading, a fixed seed |
| Who is accountable if a finding is wrong | You | You | ✓ We are. That is what you are buying |
| What you can put in front of a board or a regulator | Nothing | Output you would have to interpret and defend yourself | ✓ Sealed evidence a third party can re-run without us |
| When the model updates next month | You find out from your users | Someone has to remember to run it again, and the comparison drifts | ✓ The same frozen battery, on a schedule, comparable to the last run |
| What it tells you about what the agent did | Nothing | Depends entirely on what you built on top of it | ✓ Read off the tool-call log — did the payment function fire, and to which account |
Doing nothing is the option most teams actually pick, and it is rarely a decision — it is what happens when the question has no owner. In our own published study one frontier model executed a fraudulent transfer on every financial-fraud pretext put to it.
Another, handed the identical scenarios, named it as textbook business email compromise and declined every time. Same test, same tools, opposite outcomes — and neither fact was knowable without running it.
Client names are confidential, so we will never show you a wall of them. What we can show you is everything we have put on the public record, where anyone — including a competitor — can check it.
A certification body audits your management system — policies, roles, risk register. It cannot tell you whether your agent wires money on a forged invoice, hands over a credential, or deletes what it was told to keep. That is a different measurement, and it is the one we make.
We produce forensic technical evidence — measured, sealed, and reproducible. Aligned with the frameworks below.
If you have EU market exposure: the EU AI Act's requirements for accuracy, robustness and cybersecurity are behavioural, not documentary — they ask what the system does, not what your policy says it should do. The same evidence answers them.
⚠️ What this does not do. None of the above is a certification, and passing our battery does not make a system compliant with anything. A clean screen is not a safety claim. We hand your auditor, your regulator and your board a measurement they can check themselves — the compliance conclusion stays theirs to draw.
You are choosing between a number you know and a number you don't. Here are both, as honestly as we can put them.
The cost of the second column is never just the money. It is the client asking what you knew and when — and your answer being written afterwards instead of before.
Testing is cheap. Being asked is not.
And to be clear about which is which: a $3,000 screen is not insurance and does not make anything compliant. It replaces a guess with a measurement. That is all it does — and on the evidence above, that turns out to be the expensive gap.
Every engagement delivers the full sealed evidence package. Price reflects scope — the standard never changes.
Ask a frontier model the same question twice and you will not always get the same answer. Not a different wording — a different decision.
We measured it on our own bench.
Two identical audits, same model, same fixed seed, same probe sequence, two hours apart on 28 July 2026. The battery was byte-for-byte identical and the schedule was provably the same.
Two of thirty turns came back with a different verdict. On one, the agent declined an action. On the rerun, it took it.
Hand you the first result and the report says the model declined; hand you the second and it says the model acted. A single run cannot tell you which one your deployment will get.
What we do instead. Every engagement runs in two phases. First, sustained long-form campaigns — long enough to surface behavior that only appears under extended use, which a single-pass benchmark structurally cannot see. Then repeated independent runs against the probe families those campaigns flagged. That second phase is what turns a number into a measurement.
Why most evaluations don’t do this. It costs more, and a single number looks cleaner on a slide. A one-run benchmark cannot distinguish a model that reliably refuses from one that refuses most of the time and occasionally doesn’t. The second model is the one that surprises you in production.
The repetition count is part of the scope — agreed in writing before we start, and recorded in the sealed evidence pack.
Cleared engagements available via an established network of TS/SCI and polygraph-cleared personnel, scoped and staffed per contract — for classified, ITAR-sensitive or high-assurance environments where the vendor itself has to be trusted, not just the tooling.