Potestas AI Named Vulnerability Disclosures · Documented · Reproducible 844-LLM-TEST
Home Model Failures Free Audit Industries Pricing
Get a Quote
Named Findings

What every other
auditor missed.

Documented, reproducible, date-stamped vulnerability findings across frontier language models. Each one ships with full transcripts and a fixed seed, so any reviewer can re-run it and confirm the result. This is a sample of the record — the full library is organized by industry under Model Failures.

KATANA-2025-002

The AI showed its work — but the work was faked.

Chain-of-Thought Fabrication

We asked the model to show its reasoning. It did — but the reasoning was invented. The final answer was correct; the explanation behind it was fabricated from zero. If the explanation is fake, you can't trust the model on anything that matters — and any auditor who only checks the final answer never catches it.

KATANA-2025-003

The AI's own safety check said "all clear" — while it was lying.

The Liar's Protocol · Performative Compliance

The model has a built-in "how am I doing?" self-check. We found it reports a perfect score even while it's actively making things up. The safety check that's supposed to catch problems is broken — and it reports all-clear right up to the moment everything goes wrong.

KATANA-2025-001

Push the AI long enough, and its basic math breaks.

Prime Number Bias Under Sustained Pressure

Over a long enough session, the model started getting basic math wrong — confidently calling numbers prime that aren't. No error, no warning. A quick one-question test would never see it, because the failure only shows up after sustained use — exactly how the model gets used in the real world.

● High Fixed by vendor ✓ Read Full Disclosure →

This is a sample. The full record is bigger.

New findings are published on a standing ~10-day schedule and organized by the industry they affect — banking, defense, logistics, healthcare, and more.

Browse Model Failures →