New: we ran seven of these free tools on four named models. See the measured comparison →
PyRIT, garak, Inspect, DeepTeam and Promptfoo are capable, actively maintained and genuinely useful. We use this class of tooling ourselves. If you have an engineer who can stand one up and keep it running, you should — and this page will tell you honestly where that stops being enough.
We run the standard free benchmark too, and we agree with it where it looks. Then we show you what it can’t see: the harms it missed on your model, what your model says versus what it does, and whether it tells you when it’s done something wrong.
Strong at automating adversarial conversations and composing attack strategies against a target you control. Built by a team who red-team for a living, and it shows. Best when you have someone who will own the harness, curate the probes, and interpret the output.
The closest thing to “point it at a model and see what falls out”. Wide coverage of known failure classes and a very low barrier to a first result. Excellent for finding out quickly whether something obvious is wrong.
Designed by a national institute for exactly this problem, with proper support for agentic tasks and tool use. If you are building an internal evaluation practice rather than running a one-off check, this is the serious foundation.
Both are built to live beside your tests rather than beside your compliance file. Good at catching a regression the day it lands, which is a different and equally real job from proving to a third party what your agent did.
A scanner produces a log. What a board, an auditor or an insurer asks for is an adjudicated record with a name on it: someone who examined the output, removed the false positives, tiered what remained by severity, and can be asked to defend a finding. No open-source tool can do that, because accountability is not a feature you can ship — it is a person accepting consequences.
Your model vendor publishes safety numbers and they are probably honest. They are also structurally incapable of standing outside every vendor at once and measuring them the same way, over time. A trillion-dollar company can out-spend us on tooling. It cannot occupy that position, and neither can a tool you run against yourself.
If you need a regression check in CI, use Promptfoo or DeepTeam — we are the wrong shape and the wrong price. If you have an evaluation team and want to build a standing practice, build it on Inspect. And if nobody has yet asked you to prove anything about your agent, a screen buys you evidence you may not need yet. The case for commissioning starts when someone outside your company has to believe the answer.
One model, one pass of the 198-scenario say-vs-do suite, graded off the tool-call log rather than a judge model’s opinion. Findings and a sealed evidence pack the same day, at a published price, credited in full against a fuller audit.