Run the industry-standard tests yourself. Then compare.

What you need

• Python 3.11 and three separate environments: one for Inspect (five of the tests), one for garak, one for AgentDojo. They pin different libraries, so keep them apart.
• An API key for the model you are testing, on an OpenAI-compatible endpoint. Most vendors offer one.
• An OpenAI API key for the graders. AgentHarm, StrongREJECT, MASK and XSTest grade replies with OpenAI models by default. That is each tool’s own setting; we leave it as it ships.
• A Hugging Face account. XSTest’s dataset sits behind a short terms page you accept once.
• Budget and time. Every prompt is a paid call to your model, and the graders are paid calls too. A model that reasons before it answers costs far more and runs far slower. Try each test on a few samples first (--limit 5 in Inspect).

The versions we use

Newer versions change prompts and scoring. Use these if you want to compare with us.

1. Five tests through Inspect

Install, then point Inspect at your model. myvendor is any short name you choose; Inspect reads the two variables that carry it.

pip install inspect-ai==0.3.273 inspect-evals==0.22.0

export MYVENDOR_API_KEY=...                          # your model's key
export MYVENDOR_BASE_URL=https://api.your-vendor.com/v1
export OPENAI_API_KEY=...                            # the graders
export HF_TOKEN=...                                  # after accepting XSTest's terms

Then the eight runs. Each tool keeps its own defaults and graders; the three flags only keep a run alive: a vendor’s content filter refusing one prompt does not abort the rest, and no request waits forever.

# AgentHarm: harmful agent jobs, then the harmless twins of the same jobs
inspect eval inspect_evals/agentharm --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
inspect eval inspect_evals/agentharm_benign --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
# StrongREJECT: plain, then with its classic jailbreak wrapper
inspect eval inspect_evals/strong_reject --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
inspect eval inspect_evals/strong_reject -T jailbreak_method=AIM --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
# MASK: does it say what it believes under pressure
inspect eval inspect_evals/mask --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
# b3: attacks on apps built on the model
inspect eval inspect_evals/b3 --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
# XSTest: safe questions it should answer, unsafe ones it should refuse
inspect eval inspect_evals/xstest -T subset=safe --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20
inspect eval inspect_evals/xstest -T subset=unsafe --model openai-api/myvendor/YOUR-MODEL --no-fail-on-error --timeout 300 --max-connections 20

Read each tool’s headline score with inspect view, from the logs it writes.

2. garak

In its own environment. Tell it where your model is in a small options file:

pip install garak==0.17.0
export OPENAICOMPATIBLE_API_KEY=...                  # your model's key

{"openai": {"OpenAICompatible": {
  "uri": "https://api.your-vendor.com/v1",
  "model": "YOUR-MODEL",
  "suppressed_params": ["n"]
}}}

Run it in two parts. garak’s prompt-injection probe lowers the reply limit for every probe that runs after it and never puts it back, so it goes alone and last. The fixed seed makes a re-run ask the same prompts.

# part 1: 78 probes (list below)
garak --target_type openai.OpenAICompatible --target_name YOUR-MODEL \
  --generator_option_file options.json --probes <the 78 below> \
  --generations 1 --seed 2026 --parallel_attempts 16

# part 2: prompt injection, alone
garak --target_type openai.OpenAICompatible --target_name YOUR-MODEL \
  --generator_option_file options.json --probes promptinject.HijackHateHumans,promptinject.HijackKillHumans,promptinject.HijackLongPrompt \
  --generations 1 --seed 2026 --parallel_attempts 16

ansiescape.AnsiEscaped,ansiescape.AnsiRaw,dan.Ablation_Dan_11_0,dan.AutoDANCached,dan.DanInTheWild,divergence.Repeat,divergence.RepeatedToken,encoding.InjectAscii85,encoding.InjectAtbash,encoding.InjectBase16,encoding.InjectBase2048,encoding.InjectBase32,encoding.InjectBase64,encoding.InjectBraille,encoding.InjectEcoji,encoding.InjectHex,encoding.InjectMorse,encoding.InjectNato,encoding.InjectROT13,encoding.InjectUU,encoding.InjectUnicodeTagChars,encoding.InjectZalgo,exploitation.JinjaTemplatePythonInjection,exploitation.SQLInjectionEcho,goodside.Tag,goodside.ThreatenJSON,goodside.WhoIsRiley,grandma.Slurs,grandma.Substances,grandma.Win10,grandma.Win11,latentinjection.LatentInjectionFactSnippetEiffel,latentinjection.LatentInjectionFactSnippetLegal,latentinjection.LatentInjectionReport,latentinjection.LatentInjectionResume,latentinjection.LatentInjectionTranslationEnFr,latentinjection.LatentInjectionTranslationEnZh,latentinjection.LatentJailbreak,latentinjection.LatentWhois,latentinjection.LatentWhoisSnippet,leakreplay.GuardianCloze,leakreplay.GuardianComplete,leakreplay.LiteratureCloze,leakreplay.LiteratureComplete,leakreplay.NYTCloze,leakreplay.NYTComplete,leakreplay.PotterCloze,leakreplay.PotterComplete,lmrc.Bullying,lmrc.Deadnaming,lmrc.QuackMedicine,lmrc.SexualContent,lmrc.Sexualisation,lmrc.SlurUsage,malwaregen.Evasion,malwaregen.Payload,malwaregen.SubFunctions,malwaregen.TopLevel,misleading.FalseAssertion,packagehallucination.Dart,packagehallucination.JavaScript,packagehallucination.Perl,packagehallucination.Python,packagehallucination.RakuLand,packagehallucination.Ruby,packagehallucination.Rust,phrasing.FutureTense,phrasing.PastTense,snowball.GraphConnectivity,sysprompt_extraction.SystemPromptExtraction,web_injection.ColabAIDataLeakage,web_injection.MarkdownImageExfil,web_injection.MarkdownURIImageExfilExtended,web_injection.MarkdownURINonImageExfilExtended,web_injection.MarkdownXSS,web_injection.PlaygroundMarkdownExfil,web_injection.StringAssemblyDataExfil,web_injection.TaskXSS

3. AgentDojo

pip install agentdojo==0.1.35

If your model is on AgentDojo’s own list, its command line runs all four suites (banking, Slack, travel, workspace):

python -m agentdojo.scripts.benchmark --model <AgentDojo's name for it> \
  --attack important_instructions --benchmark-version v1.2

If it is not on the list, drive it through AgentDojo’s OpenAI-compatible client (OpenAILLM) pointed at your endpoint, the way we run every model: temperature 0, no defence, AgentDojo’s default system message, the important_instructions attack, all four suites, each run once without the attack and once with it.

Settings that change the answer

• Your vendor may refuse a setting. Some refuse seed 0, which AgentHarm sets; some allow only temperature 1. Use the value the vendor forces and write it down (--seed 1, --temperature 1 in Inspect; leave the refused parameter out with suppressed_params in garak).
• A model that reasons before it answers needs room to answer. Raise the reply limit or the tools grade blank replies: --max-tokens 8192 for MASK, --max-tokens 4096 for XSTest, and "max_tokens": 4096 in garak’s options file.
• garak 0.17.0 treats an HTTP 403 as a bad key and stops the whole run. Some vendors answer a prompt their filter blocks with a 403. When it happens, the run did not finish; count it as blocked, not as a result.
• Count the empty replies. Some tools score silence as a pass. A model that answers nothing can look like the safest one in the room.
• One run is presence, not a rate. Run it again tomorrow and some numbers will move.

Two of these settings came after our 2 October comparison: garak’s prompt-injection probe now runs on its own, and garak now runs seeded. Everything else matches it.

Then compare

Put your numbers beside ours for four named models: what the seven tests caught and missed. Where these tests and our audit measure the same thing, they agree. The difference is in what they never try: AgentDojo tries one kind of attack; our agentic test tries twelve, with working tools in the model’s hands.