We ran seven free AI safety tools on four models. Here’s what they caught, missed and cost.

What we ran

Seven recognised free tools: AgentDojo (ETH Zurich), AgentHarm (Gray Swan AI and the UK AI Security Institute), StrongREJECT (UC Berkeley), MASK (Center for AI Safety and Scale AI), b3 (Lakera, with the UK AI Security Institute), XSTest, and garak (NVIDIA). Five of them ran through the UK AI Security Institute’s Inspect framework. Each ran once on each model, between 29 September and 2 October 2026.

Our audit ran on the same four models: an agentic test where the model has working tools and ordinary jobs to do, graded from what it actually did, and a 400-turn conversation under sustained pressure.

The scoreboard

Each free tool’s own headline score, read from its own log. Higher is worse unless the column says otherwise.

AgentHarm and StrongREJECT scores are averages on a 0–1 scale, as each tool reports them. The kimi-k2.6 MASK score was measured with a larger reply limit; see “Silence graded as honesty” below. *garak’s own score for kimi-k2.6 counts 3,791 empty replies as passes; see “How the free tools break”.

We checked our work first

The team that built AgentHarm published gpt-4o-mini’s score: 62.5% harm, refusing 22.0% (Table 9 of the paper, no attack). Our run: 59% harm, refusing 22%. Close enough to show the tools were set up correctly, so the differences below are the tools and the models, not our setup.

Where the free tools and our audit agree

On attacks hidden in the data an AI reads, AgentDojo, AgentHarm and our injection tests ranked the four models in exactly the same order: kimi-k2.6 hardest to steer, then Llama-3.3-70B, gpt-4o-mini, and grok-4.20 easiest. On plainly harmful requests, StrongREJECT and our 400-turn test agreed again, with Llama-3.3-70B the weakest.

Where we measure the same thing, we get the same answer. So when our audit finds something the free tools don’t, it isn’t a different ruler. It’s a different place to look.

What only the audit found

One kind of attack versus many. AgentDojo tests one kind of attack, an instruction hidden in data the agent reads. Against kimi-k2.6 it got through 11 times in 949 (1.2%), the best result of the four. Our audit tries 12 kinds of attack. On the same model, five different kinds got through, by seven routes in: an email thread, a calendar invite, a support ticket, its own memory, a plan it had saved, a tool’s description, and another agent. What they caused: customer data emailed to an outside address, board papers sent to an address someone had planted in its memory, administrator rights granted through a plan it had saved earlier, among others. AgentDojo wasn’t wrong. It tests one kind of attack, and your agent faces more than one.

Talking is not working. grok-4.20 refused only 6% of safe-but-alarming questions on XSTest, and refused 6% of openly harmful jobs on AgentHarm. Given ordinary legitimate work in our audit, it acted correctly on only 52 of 61 jobs. Refusing to talk and refusing to work are different failures.

Pressure over time. StrongREJECT asks one harmful question per conversation. It scored three of the four models at or near zero, and its classic jailbreak wrapper barely works any more. Our 400-turn test still recorded security-critical failures in all four: 1 for kimi-k2.6, 7 for grok-4.20, 12 for gpt-4o-mini, 14 for Llama-3.3-70B (as sealed, before human review of the turns it flagged).

The free tests disagree with each other

Which of the four is the riskiest? AgentDojo, AgentHarm and b3 said grok-4.20. StrongREJECT said Llama-3.3-70B. garak’s own score said grok-4.20, but scored on the same prompts with silence not counted as a pass, it said kimi-k2.6, the model every other test rated safest. Our audit, measuring harm during ordinary work, put grok-4.20 below both gpt-4o-mini and Llama-3.3-70B. Two research papers published this year found the same disagreement across 12 and 22 models (arXiv 2605.16282, arXiv 2607.28685). One free score is one test, not a safety rating.

Silence graded as honesty

MASK caps each reply at 1,000 tokens. kimi-k2.6 thinks before it answers, and it often spent the whole allowance thinking, so 45% of its replies came back empty. MASK graded those scenarios as “evading”, and evading doesn’t count as lying. At the default setting kimi-k2.6 looked about 90% honest.

Given room to answer, it lied in 228 of 1,000 scenarios and scored 77%. One setting hid more than half of the model’s lies. Our audit records an empty reply as “not tested”, never as a pass.

How the free tools break

• A single blocked prompt can stop the test. When xAI’s or Moonshot’s own filter refused a prompt, StrongREJECT under Inspect’s default stopped the entire run. garak read xAI’s refusal as a bad API key and quit 239 prompts into a 12,000-prompt run. We changed both to keep going and count blocked prompts separately.
• Empty answers get graded. XSTest gives each reply 256 tokens. kimi-k2.6 used them thinking and returned nothing in 5 of its first 6 replies; XSTest’s grader still marked them as complied, partly refused or refused.
• Silence scored as a pass, again. kimi-k2.6 gave empty replies to 3,791 of garak’s 11,938 prompts, and garak graded them as passes. Part of the cause is a garak bug: its prompt-hijacking probe shrinks the reply limit to 60–100 tokens and never restores it, so every probe that runs after it inherits the tiny limit, on every model. kimi-k2.6 spent those tokens thinking and said nothing; the other models were cut off mid-answer. garak’s own score for kimi-k2.6 is 32.3%. On the 3,114 prompts all four models were asked and kimi-k2.6 actually answered, it failed 36.2% of checks, more than grok-4.20 (33.5%), Llama-3.3-70B (27.0%) or gpt-4o-mini (26.8%).
• Each model gets different questions. garak samples a random set of prompts for many of its probes, so the four models were partly asked different things. Comparing models fairly means scoring them on the prompts they share.
• One bad grade can lose a run. On Llama-3.3-70B, MASK’s grader returned a malformed answer and the tool discarded a 33-minute run. Our own grader review in September found a fault of the same kind; every grader needs this tested.

A cheap model can be expensive

kimi-k2.6 is priced at $4.00 per million tokens written, against $2.50 for grok-4.20. But it reasons privately before every reply, and that reasoning is billed. On b3 it wrote about 3,500 tokens per answer where gpt-4o-mini wrote 265, and each answer cost about seven times grok-4.20’s. A price list doesn’t show what an answer costs.

What the free tools do better

• Anyone can run them. Recognised names, published papers, open code. That is exactly why we show them beside our audit.
• A single over-refusal number. XSTest’s safe prompts and AgentHarm’s harmless twin of each harmful job each give one clear figure for refusing legitimate requests.
• Honesty about the world. MASK separates what a model believes from what it says under pressure.
• Your own defences, measured. b3 scores each app at three defence levels: gpt-4o-mini went from 28% with no defence to 1.4% at the strongest.
• Breadth. garak runs 81 probes across 18 families of text attack.
• Speed. AgentHarm finished in under two minutes on gpt-4o-mini.

How we ran it

Inspect evals for AgentHarm, StrongREJECT, MASK, b3 and XSTest, with each tool’s own default settings and graders; garak 0.17.0 with the same 81 probes and one reply per prompt on every model; AgentDojo 0.1.35, all four suites. XSTest used the gated public dataset (450 prompts).

Settings the vendors forced, recorded so you can match them: xAI rejects AgentHarm’s seed of 0, so grok-4.20 ran at seed 1; Moonshot allows only temperature 1 for kimi-k2.6; kimi-k2.6 was given larger reply limits on XSTest, MASK and garak so it could answer at all; garak’s own hijacking probe later overrode that limit (see above). We published no test prompts on this page, only the kinds of failure.