1, Nico Schiller2, Hongwei Li3, Srijiith Sesha Narayana2, Milad Nasr5, Nicholas Carlini5, Xiangyu Qi6, Eric Wallace6, Elie Bursztein7, Luca Invernizzi7, Kurt Thomas7, Yan Shoshitaishvili4, Wenbo Guo3, Jingxuan He1, Thorsten Holz2, Dawn Song1 1UC Berkeley · 2Max Planck Institute for Security and Privacy · 3UC Santa Barbara · 4Arizona State University · 5Anthropic · 6OpenAI · 7Google A benchmark of 869 real-world vulnerabilities spanning userspace programs, Chrome's V8 JavaScript engine, and the Linux kernel. Given a vulnerability and an input that triggers it, AI agents are tasked with crafting a full exploit that achieves unauthorized code execution.
Success is the number of instances exploited via the intended vulnerability, split by domain: userspace, browser V8, and Linux kernel. Evaluated under trusted access programs designed for security research; evaluation notes are shown with each submission. Cost, output tokens, time, and LLM calls (under Avg / task) are per-task averages over the full benchmark; hover for the averages over successful exploits.
Frontier agents can take a bug report and a crashing input, reason about memory layouts, chain multiple attack primitives, and produce fully working exploits — work that traditionally required deep human expertise and significant time.
With ASLR, stack canaries, and the V8 heap sandbox enabled, successes dropped substantially but didn't hit zero. Agents found bypasses such as partial-pointer overwrites, known sandbox escapes, and kernel tricks like overwriting modprobe_path.
Automated exploit generation can accelerate severity triage and validate mitigations, but the same capability lowers the barrier for offensive misuse. We believe the responsible path is to measure these capabilities rigorously and openly.
This article was aggregated automatically by CyberWire Daily's newsfeed engine. Original reporting: www.cybergym.io.
