Artificial Analysis has released its Cyber Index alongside an industry consortium to test how defensive AI security models discover and resolve software vulnerabilities.
Organised under the Cyber Index Alliance with launch partners IBM, NVIDIA, Collinear AI, and Vercel, the initiative measures defensive performance across production software repositories. The testing environment tracks whether agents can audit raw source trees, isolate confirmed weaknesses, and write working patches without breaking surrounding logic.
The evaluation purposely avoids offensive exploit construction. “Exploit realisation, turning a found vulnerability into a working exploit, is out of scope for a defense-focused index,” explained Artificial Analysis.
All automated evaluation runs operate through Stirrup, an open-source agent harness designed by Artificial Analysis to standardise testing across realistic developer environments.
Code auditing reveals persistent patch regressions
Composite index results place Grok 4.7 (xhigh) and MiMo-V2.6-Pro at the top of the leaderboard with matching scores of 56 percent. GPT-6 Luna followed at 53 percent, with GLM-5.3-Flash reaching 50 percent. However, results varied across the individual evaluation suites.
On CWE-Bench-AA – contributed by Collinear AI and comprising 120 private tasks across the OWASP Top 10 categories – both Grok 4.7 and DeepSeek V4.1 Flash led the field with a 68 percent success rate. The task suite spans C, C++, Go, Java, JavaScript, TypeScript, Python, and Rust, presenting models with known CVE scenarios where only the broad area of concern is disclosed.
Security engineers monitoring the agent traces identified specific trade-offs during audit routines. Models devoted an average of 38 percent of their interaction steps to finding bugs before making their first source edit. They spent the remaining 62 percent writing patches and running tests.
Partial remediations caused 55 percent of all non-refusal failures. In those instances, models addressed the primary vulnerability but left secondary attack paths open.
Capable models also caused unintended breakages. The four highest-scoring systems broke existing functionality in roughly 40 percent of their failed attempts, compared to 15 percent among the lowest-performing cohort.
Safety refusals suppress frontier AI model security tests
Discovery accuracy showed distinct boundaries on Vercel’s DeepsecBench-AA, which measures flagged file reviews against expert-verified flaws using an F2 scoring metric. Even the highest-scoring model found only 41 percent of verified defects.
Straightforward flaws connecting untrusted inputs directly to execution sinks were routinely identified. Complex logic flaws that unfold across execution sequences proved far harder to uncover. GPT-6 Sol and GPT-6 Astra detected sequential flaws in approximately 30 percent of runs, yet achieved 95 percent accuracy when they did report them.
Automated safety guardrails restricted evaluations on Berkeley RDI’s CyberGym-E2E-AA framework, which evaluates end-to-end memory-safety remediation across 131 C and C++ projects. Frontier models – including GPT-6 Astra, GPT-6 Sol, Claude Fable 5.1, and Claude Opus 5.5 – refused at least 98 percent of assigned tasks on safety grounds.
Among systems that executed the suite, MiMo-V2.6-Pro achieved a 79 percent pass rate, followed by GPT-6 Luna at 78 percent and Grok 4.7 at 74 percent. Timeouts limited broader success, with 42 percent of runs hitting the 90-minute limit without producing a reproducible crash.
Models resolved 50 percent of out-of-bounds memory bugs, but their success dropped to 33 percent on use-after-free conditions and 20 percent on arithmetic flaws. Off-target fixes also diluted results, as 31 percent of passing solutions patched unrelated issues, such as shallow null-pointer crashes, rather than the intended vulnerability.
The Cyber Index Alliance says it intends to extend future evaluations to compiled software, live network targets, and incident response workflows.
See also: DeviQA survey links AI code generation to testing queues
Want to learn more about cybersecurity from industry leaders? Check out Cyber Security & Cloud Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the AI & Big Data Expo. Click here for more information.
Developer is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
