EVMBENCH LEADERBOARD · UPDATED
EVMBench Leaderboard: Savant Chat is #1 with 95.7% recall
On EVMBench — the smart contract security benchmark from OpenAI and Paradigm — Savant Chat detected 112 of 117 high-severity vulnerabilities from 40 real Code4rena audits. That is the highest published score of any AI auditor or frontier model, and leaves only five bugs on the table: the benchmark is effectively solved.
- Detection recall
- 95.7%
- Rank
- #1
- High-severity bugs found
- 112/117
- vs. best frontier model
- 2.1×
DETECT MODE · RECALL
EVMBench Detect leaderboard
Every published EVMBench Detect result for AI audit agents and frontier models, ranked by recall — the share of ground-truth high-severity vulnerabilities each system detected. Each row links to the source that published it.
| # | Agent / model | Type | Recall | Found | Source |
|---|---|---|---|---|---|
| 1 | Savant ChatSavant ChatAll 117 reports, critic verdicts and reasoning published. | Agent | 95.7% | 112/117 | Full reports ↗ |
| 2 | CecuroCecuro | Agent | 91.45% | — | cecuro.ai ↗ |
| 3 | AzimuthTestMachine | Agent | 86.3% | 101/117 | testmachine.ai ↗ |
| 4 | AuditAgentNethermind Security | Agent | 67.0% | 80/120 | nethermind.io ↗ |
| 5 | KaiDria | Agent | 64.2% | 77/120 | dria.co ↗ |
| 6 | GPT-5.6 (Codex)OpenAI · run by TestMachine | Base model | 61.5% | 72/117 | testmachine.ai ↗ |
| 7 | GuardixGuardix | Agent | 59.8% | 70/117 | guardix.dev ↗ |
| 8 | Claude Opus 4.6Anthropic · Claude Code | Base model | 45.6% | — | EVMBench paper ↗ |
| 9 | GPT-5.3-Codex (xhigh)OpenAI · Codex CLI | Base model | 39.2% | — | EVMBench paper ↗ |
| 10 | GPT-5.2 (xhigh)OpenAI · Codex CLI | Base model | 39.2% | — | EVMBench paper ↗ |
| 11 | GLM 5.2Zhipu AI · run by TestMachine | Base model | 38.3% | 46/120 | testmachine.ai ↗ |
| 12 | Claude Opus 4.5Anthropic · Claude Code | Base model | 36.1% | — | EVMBench paper ↗ |
| 13 | GPT-5.2 (OpenCode)OpenAI · OpenCode | Base model | 30.0% | — | EVMBench paper ↗ |
| 14 | GPT-5OpenAI | Base model | 23.3% | — | EVMBench paper ↗ |
| 15 | Gemini 3 ProGoogle | Base model | 20.8% | — | EVMBench paper ↗ |
| 16 | OpenAI o3OpenAI | Base model | 10.6% | — | EVMBench paper ↗ |
Agent — a multi-step system with tools and pipeline logic. Base model — a frontier LLM in a standard coding-agent scaffold, as evaluated in the EVMBench paper. Vendor results are self-reported; base-model results come from the EVMBench paper (Table 9) unless noted. Scores out of 120 predate OpenAI's removal of three findings; percentages are comparable.
THE BENCHMARK
What is EVMBench?
EVMBench is the smart contract security benchmark released by OpenAI together with Paradigm in February 2026. It is built from 40 historical Code4rena audit contests — real protocols, real money, and 117 high-severity, loss-of-funds vulnerabilities confirmed by human auditors and judges. Unlike synthetic CTF challenges, every bug was found in production-grade code.
The benchmark has three modes — Detect, Patch and Exploit. Detect is the one that matters for auditing: the agent reviews the repository and writes an audit report, and an LLM judge decides, vulnerability by vulnerability, whether the report found the real bug. In the paper, the best frontier model (Claude Opus 4.6 in Claude Code) detected 45.6%.
- 40Code4rena audit repositories
- 117ground-truth high-severity vulnerabilities
- $218Kin historical audit awards behind those bugs
- 45.6%best frontier-model recall in the paper
METHODOLOGY
How Savant Chat scored 95.7% — and how to check it
Scored exactly the way EVMBench scores: one audit report per issue, judged against that issue's ground truth by EVMBench's own grader.
- One report per issue. Savant Chat produced an audit report for each of the 117 issues in the canonical high-severity set — no cherry-picked subset.
- EVMBench's own judge. Each report was graded by EVMBench's unmodified
DetectGraderwith its GPT-5 critic at high reasoning effort and structured verdicts. Same mechanism, same code path, same fix — "same area, similar impact" does not count. - One verdict, no retries. One verdict per report — no voting, no re-rolls, no partial credit. Result: 112 hits, 5 misses.
- Everything is public. All 117 reports, every critic verdict with its full reasoning, and the ground truth are published so anyone can re-run the grader without Savant Chat.
FAQ
EVMBench questions, answered
What is EVMBench?
EVMBench is a benchmark created by OpenAI and Paradigm that measures how well AI agents detect, patch and exploit high-severity smart contract vulnerabilities. Its Detect mode scores recall against 117 ground-truth high-severity, loss-of-funds bugs from 40 real Code4rena audit contests.
Which AI auditor ranks #1 on EVMBench?
Savant Chat ranks #1 on EVMBench Detect with 95.7% recall — it detected 112 of the 117 high-severity vulnerabilities, 4.25 points ahead of the next-best published result and more than twice the best frontier model in the EVMBench paper (Claude Opus 4.6, 45.6%).
How is EVMBench Detect scored?
Each audit report is compared with a ground-truth finding by an LLM judge using the official EVMBench grading prompt. A vulnerability counts as detected only when the report describes the same mechanism, the same code path and the same fix. The headline metric is recall: the share of ground-truth vulnerabilities detected.
Why do some EVMBench scores use 117 and others 120?
EVMBench launched with 120 vulnerabilities; shortly after release OpenAI removed three low-quality findings, leaving 117 as the canonical set. Results published before the patch are scored out of 120, later ones out of 117. Recall percentages remain directly comparable.
Can I verify Savant Chat's EVMBench result?
Yes. All 117 audit reports, EVMBench's own GPT-5 critic verdict for each, the critic's full reasoning and the ground truth are public in savantchat/savant-chat-evmbench-report. Every verdict is a single call to EVMBench's unmodified DetectGrader, so anyone can re-run it without Savant Chat.
Can I run the EVMBench-leading auditor on my own code?
Yes. The same Savant Chat engine audits Solidity, Vyper and Rust smart contracts — and any other code — from $0.07 per line, with $75 in free credits on signup and no credit card required. A typical codebase is audited in 10–30 minutes.
SOURCES
Where these numbers come from
- Savant Chat EVMBench report — 117 reports, verdicts and reasoning (112/117, 95.7%).
- EVMBench paper (OpenAI, Paradigm) — base-model Detect results, Table 9.
- Introducing EVMBench (OpenAI) and the openai/frontier-evals harness.
- Cecuro — Cecuro — self-reported 91.45%.
- TestMachine — Azimuth — self-reported 86.3% (101/117).
- Nethermind Security — AuditAgent — self-reported 67% (80/120).
- Dria — Kai — self-reported 64.2% (77/120).
- Guardix — Guardix — self-reported 59.8% (70/117).