AI coding agents write secure, working code for only a quarter of tasks
Sorted AI15 OctA team of 15 researchers, with Tianyi Wu as first author, built the Last Secure Code Benchmark to test whether code from AI coding agents is safe as well as working. It has 450 tasks drawn from 150 real vulnerabilities across seven languages (Java, C, C++, JavaScript, Python, PHP and Go), and the task instructions never mention security; hidden tests then check both function and safety. Six frontier agents got 73.6% to 91.3% of tasks working, but the best, Claude Fable 5, wrote code that was both working and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%.
This is a preprint posted to arXiv on 10 Oct 2026 and not yet peer reviewed, but the setup matches how most people vibe code: ask for features, never for security.
Why it matters: If your team ships code written by AI agents, assume working code is not secure code. Ask for security explicitly in your prompts, and run a security scanner or reviewer on agent pull requests before they merge.
arXiv ↗AI-assisted, reviewed by Sorted AI
▲ 0
0 comments