SEC-bench Pro
The pass@1 for vulnerability discovery (creating a PoC).
- Category
- Other
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 2total
- All model rankings (records) · Ranked results: 3
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 3)
official results (Gemini 4 Carbon (Google DeepMind models page, benchmark table))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon | High | 88.2% | — | Gemini 4 Carbon (Google DeepMind models page, benchmark table)Retrieved 2026-11-12 · Vendor-reported value. Google published table. methodology PDF: Gemini 4 Carbon/Argon used the highest Gemini API thinking setting, pass@1, and a single trial. Row-specific source: self-measured. 183 tasks (2026-05 JS engine split), mean Pass@1 over 3 trials, Antigravity harness (via Fairwind Program)Source excerptSEC-bench Pro: Gemini 4 Carbon 88.2 | |
| 2 | Gemini 4 Argon | High | 83.4% | — | Gemini 4 Carbon (Google DeepMind models page, benchmark table)Retrieved 2026-11-12 · Vendor-reported value. Google published table. methodology PDF: Gemini 4 Carbon/Argon used the highest Gemini API thinking setting, pass@1, and a single trial. Row-specific source: self-measured. 183 tasks (2026-05 JS engine split), mean Pass@1 over 3 trials, Antigravity harness (via Fairwind Program)Source excerptSEC-bench Pro: Gemini 4 Argon 83.4 |
official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-V4.1-Flash | not stated | 62.8% | — | DeepSeek | DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) has no official statement, so effort is null. Evaluated with the Claude Code harness.Source excerptSEC-Bench Pro (Pass@1) | 62.8Location HF model card "Comparison with frontier models (Max reasoning effort)" table, SEC-Bench Pro (Pass@1) row, DS-V4.1-Flash column (rightmost) |