TLDR Tech / InfoSec
Benchmarking 13 AI models on rediscovering known CVEs
Aikido ran 13 models from OpenAI, Anthropic, xAI, Google, and open-weight vendors against 26 known GitHub-advisory CVEs inside its production AI Code Analysis harness (pass@3, model pointed at the vulnerable snippet), and found that GPT-5.6 topped recall at 23/26 (88.5%) with Kimi K3 tying it, grok-4.5 at 20, and the Claude Opus models at 15 to 18. The operationally useful finding is variance: single runs miss bugs other runs catch, so pooling three passes of a cheap model beats one pass of a flagship, with three gpt-5.4-nano runs reaching 18/26 for roughly $170 against a flagship single pass, and claude-haiku-4-5 swinging from 7 to 13 on the same task. The capability ceiling is reasoning, not sinks: every model found both critical bugs and obvious injection/deserialization flaws, but only GPT-5.5 and the strongest GPT-5.6 variants traced an indirect SQL injection through an unescaped ORM column alias, so defenders should pool multiple cheaper runs and treat the harness, not raw model tier, as the variable that aims reasoning at the right code.
Read the full story