CWE-bench v2 Raises the Bar for AI Cybersecurity Evaluation

Collinear AI has published CWE-bench v2, an expanded public benchmark that stresses frontier models against real-world software vulnerabilities. The suite comprises 100 self-contained tasks drawn from actual open-source repositories, spanning 91 distinct CWE categories. Each task presents an agent with a repository checkout and a concise instruction to locate and remediate a security flaw without regressing existing functionality. Scoring is fully programmatic and deterministic, enabling reproducible, zero-cost evaluation for any lab.

Results and cost efficiency

Eleven models have been evaluated to date. Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 (67.3% / 78.0%). Ninety-five tasks are solved by at least one model, confirming solvability, yet the best system still fails 19% of attempts on every run — evidence that the long tail of vulnerability classes remains hard. On the cost frontier, DeepSeek V4.1 Flash delivers 57.0% pass@1 at $0.04 per rollout; Opus adds ten points at $1.83; Gemini edges ahead at $3.92.

The public task set, grader, and execution harness are available on the Hugging Face Hub. Teams can run evaluations locally and submit results via pull request to the community leaderboard, while a separate Space tracks API-only models such as Gemini, Claude, and GPT variants.