The Evidence Base

Nine methodological pitfalls endemic to LLM security research — data leakage, context truncation, surrogate-model generalization, evaluation fragility — appear in 100% of the 72 peer-reviewed papers surveyed at leading security and software engineering venues between 2023 and 2024. Every paper contains at least one pitfall.

If every published paper on LLM security contains a methodological flaw, unpublished vendor benchmark reports carry even less evidentiary weight. A defensible audit requires explicit documentation that these pitfalls were controlled for — which a chain-of-custody evidence pack provides and a summary score cannot.

Researchers attempted to replicate 18 LLM-centric studies from ICSE 2024 and ASE 2024. Of the five studies with sufficient artifacts to even attempt replication, none could be fully reproduced. Only two were partially reproducible. No characteristic of LLM studies — including official reproducibility badges — predicted whether results could be replicated.

Reproducibility is not a nicety — it is the difference between a finding and an observation. If published, peer-reviewed, artifact-badged results cannot be reproduced, a private benchmark report certainly cannot. Every Katana audit delivers the full run artifact: prompt, response, grader output, seed, and hash — so any third party can independently verify every finding.

A classifier built on basic code metrics performs on par with state-of-the-art LLMs for vulnerability discovery. Root-cause analysis reveals that LLMs are responding to shallow syntactic features rather than genuine security reasoning — which explains why their benchmark performance does not translate to real-world detection.

High benchmark scores on vulnerability detection may reflect pattern-matching to superficial features, not actual safety capability. Behavioral auditing under adversarial pressure — not static code scanning — is the only way to distinguish genuine robustness from calibrated surface performance.

Across 228 code scenarios and eight leading LLMs, the SecLLMHolmes framework found non-deterministic responses, incorrect reasoning, and poor real-world performance. Changing only function or variable names caused incorrect answers in 26% of cases for leading models — demonstrating fundamental non-robustness under trivial perturbation.

Non-robustness to surface-level perturbation is exactly the failure mode an adversary will exploit. A vendor claiming their model "passed" a vulnerability benchmark has not shown robustness — they have shown performance on one test, under one framing. Sustained adversarial campaign testing is the only method that reveals whether robustness is real or a measurement artifact.

Across 110 explicitly malicious multi-step agent tasks (440 with augmentations) covering 11 harm categories including fraud, cybercrime, and harassment, frontier LLM agents demonstrated significant rates of task compliance — even without jailbreaking. LLM agents acting with tools pose substantially greater misuse risk than the same models in chatbot configurations.

AgentHarm establishes the research baseline for the exact question Potestas AI's flagship study operationalizes: will an agent actually execute the action, not just discuss it? Comparing an organization's model against this baseline requires a behavioral evidence record — not a benchmark score — because the risk is in what the agent does, not what it says.

In realistic agentic task environments — email, calendar, finance, travel — prompt injection attacks cause LLM agents to execute attacker-controlled instructions using real tools on behalf of users. The framework demonstrates that utility and security are in direct tension: more capable agents are more easily exploited.

AgentDojo frames the adversarial surface as it actually exists in deployed systems — not hypothetical, but embedded in normal workflows. Auditing an agentic system requires simulating the exact environment it operates in, with adversarial inputs injected through the channels the system trusts. Static red-teaming does not replicate this.

The global open-source security community — over 100 security researchers, industry practitioners, and government representatives — has codified ten critical risk classes for agentic AI: Goal Hijack, Tool Misuse, Identity & Privilege Abuse, Memory Poisoning, and six others. This is the emerging industry standard for agentic AI security, referenced by NIST, NVIDIA, AWS, and Microsoft's AI Red Team.

The OWASP Agentic Top 10 is now the reference taxonomy a security buyer will bring to a procurement conversation. An audit that cannot map its probe categories to this taxonomy cannot answer the buyer's first question. Katana Auditor's 27-category probe structure is aligned with the Agentic Top 10 and the OWASP LLM Top 10 (2025).

Three new standard endorsements, effective January 2026, allow carriers to exclude generative AI claims from CGL policies. CG 40 47 is the broad exclusion — removing bodily injury, property damage, and personal/advertising injury arising from generative AI. CG 40 48 excludes Coverage B only. Multiple carriers have adopted them at renewal. The effect is rolling across the market.

Organizations deploying AI systems that cannot document behavioral testing may find that AI-related claims fall outside their CGL coverage. A forensic audit report — with chain-of-custody evidence of pre-deployment behavioral testing — is precisely the documentation an underwriter will require to negotiate around or alongside these exclusions.

Over half of U.S. states have adopted the NAIC Model Bulletin requiring insurers to maintain documented AI governance programs and third-party vendor oversight. A 12-state pilot of the AI Systems Evaluation Tool is running through September 2026, with full NAIC adoption expected at the 2026 Fall Meeting. A model law on third-party AI vendor licensing is anticipated in 2026, which would directly regulate AI auditors and testing vendors.

Insurers subject to NAIC oversight are already required to document vendor AI diligence with audit rights and contractual controls. When the third-party model law arrives, forensic audit documentation from a credentialed vendor will not be optional — it will be the evidence regulators pull during market conduct examinations. The regulatory window to build that record is now.

The de facto federal standard for AI governance defines four core functions: Govern, Map, Measure, and Manage. The framework explicitly requires that AI systems be evaluated for trustworthiness — validity, reliability, safety, security, privacy, and explainability — with documentation sufficient to support regulatory inquiry. It is the reference framework cited by NAIC, state insurance regulators, and federal procurement.

The AI RMF's MEASURE function calls for systematic evaluation of AI system behavior and risk — not self-reported scores. A Katana forensic evidence pack is structured to serve as the documentation layer for MEASURE and MANAGE functions, giving an organization a ready response when a regulator, auditor, or procuring agency asks how AI system behavior was verified.

The GenAI Profile extends AI RMF 1.0 to generative AI systems, identifying twelve risk areas specific to or amplified by generative AI — including confabulation, information security, value chain integrity, and human-AI configuration failures — and providing over 200 suggested governance actions. This is the operational federal standard for organizations deploying LLMs, released pursuant to Executive Order 14110.

Federal agencies and regulated entities deploying LLMs are expected to align with AI 600-1. An auditor who cannot map their findings to the twelve GenAI risk areas defined in this profile cannot serve a federal buyer. Katana's 27-category probe structure maps to the AI 600-1 risk taxonomy, giving federal clients a report they can hand directly to their compliance function.

NIST released a concept note in April 2026 for a critical infrastructure AI profile, extending the RMF specifically to operators of critical infrastructure deploying AI-enabled capabilities. This signals that the federal AI governance architecture is extending downward into the sectors — finance, defense, healthcare — where Potestas AI's buyer lives.

As the critical infrastructure profile matures, behavioral testing and documented evidence of AI system behavior will become part of the compliance baseline for operators in these sectors. Organizations that build their behavioral evidence record now — before the profile is final — will have documented compliance history. Those that wait will scramble to retrofit it.