Potestas AI Model Failures · Documented · Reproducible · Updated on a Standing Schedule 844-LLM-TEST
Home Model Failures Free Audit Industries Pricing
Get a Quote
Model Failures · The Standing Record

Where frontier AI
actually breaks.

Documented, reproducible findings across frontier models — organized by the industry they affect, so you see what matters to you without wading through everything else. Every finding ships with full transcripts and a fixed seed. Nothing here is asserted that can't be re-run.

Current Intel

The state of play,
right now.

One team, one identical battery, every frontier model — scored the same way, on a standing schedule. No vendor can produce this comparison about itself. That's the point.

Get the full briefing — every model, every industry — in your inbox.
Joined by CISOs, procurement officers, and safety teams. No spam. Unsubscribe anytime.
Internally, we call this the AI COP — the Common Operating Picture on how frontier models are actually behaving. Old habits.
Integrity Rankings● Current Cycle
GPT-4oA · 91.2%
Claude Sonnet 4A · 89.7%
Gemini 2.5 ProB · 82.4%
Grok-4B · 78.1%
Llama 405BC · 71.6%
* Scores reflect the full sustained-pressure protocol. Wrapped deployments scored separately.
Pick Your Industry
Select an industry — each finding re-summarizes for your world.
KATANA-RR-2026-01 · Flagship Study Say-vs-Do · Agentic Testing

We tried to socially engineer four frontier AI agents. Two of them wired the money.

4-Model Comparative Study · 9 Agentic Scenarios · Published June 2026

Read the Full Study → Try the Attack Yourself → ● Flagship
KATANA-2025-002 · Critical Finding Reasoning-Integrity Validation

The AI faked its own reasoning — and still got the right answer.

Chain-of-Thought Fabrication · Grok-4 observed · other models under test

Read Full Disclosure → ● Critical
KATANA-2025-003 · Critical Finding Self-Report vs. Actual Behavior

The AI's own safety check said "all clear" — while it was lying.

The Liar's Protocol · Performative Compliance · Grok-4 primary · other models confirmed

Read Full Disclosure → ● Critical
KATANA-2025-001 · High Severity Sustained-Use · Conversational Testing

Push the AI long enough, and its basic math quietly breaks.

Prime Number Bias Under Sustained Pressure · Grok-4 observed

Read Full Disclosure → ● High Fixed by vendor ✓
No published findings for this industry yet — new findings land on a standing ~10-day schedule. Check back, or get the briefing above.
This record grows on a standing schedule. New stress tests run continuously against every frontier model — including every new release — and findings publish roughly every ten days. One-shot tools give you a snapshot. This is the only continuously-updated integrity record of how these models change over time.

The research behind the method.

Our findings don't stand alone. The peer-reviewed evidence base — NDSS, ICSE, IEEE S&P, NIST, OWASP — maps the field to the case for forensic auditing, every citation verified.

Open the Evidence Base →