SWE-Bench Pro
Resolution rate on agentic software engineering tasks (public set). Task modifications and harness differ by source (stated in conditions).
- Category
- Agentic coding
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 3total
- All model rankings (records) · Ranked results: 5
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 5)
official results (Gemini 3.5 Flash model card)
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Gemini 3.5 Flash | not stated | 55.1% | — | Gemini 3.5 Flash model cardRetrieved 2026-11-13 · Vendor-reported value. Table in the Google model card. methodology PDF (deepmind.google/models/evals-methodology/gemini-3-5-flash): All of the results are all run with the Gemini API ... with default sampling settings unless indicated otherwise. Gemini computed in-house; average of 5 runs; internal version of the Antigravity harness; single trial. thinking level not statedSource excerptSWE-Bench Pro (Public) Diverse agentic coding tasks | Single attempt | 55.1%Location Gemini 3.5 Flash model card Evaluation table, SWE-Bench Pro (Public) row, Gemini 3.5 Flash column. In-house computation basis: methodology PDF, Coding / SWE-Bench Pro | |
| 2 | Gemini 3.1 Pro Preview | not stated | 54.2% | — | Gemini 3.5 Flash model cardRetrieved 2026-11-13 · Vendor-reported value. Table in the Google model card. methodology PDF (deepmind.google/models/evals-methodology/gemini-3-5-flash): All of the results are all run with the Gemini API ... with default sampling settings unless indicated otherwise. Gemini computed in-house; average of 5 runs; internal version of the Antigravity harness; single trial. Column heading is Gemini 3.1 Pro (gemini-3.1-pro-preview is the only match in the Gemini API model list). thinking level not statedSource excerptSWE-Bench Pro (Public) Diverse agentic coding tasks | Single attempt | 54.2%Location Gemini 3.5 Flash model card Evaluation table, SWE-Bench Pro (Public) row, Gemini 3.1 Pro column. In-house computation basis: methodology PDF, Coding / SWE-Bench Pro | |
| 3 | Gemini 3 Flash Preview | not stated | 49.6% | — | Gemini 3.5 Flash model cardRetrieved 2026-11-13 · Vendor-reported value. Table in the Google model card. methodology PDF (deepmind.google/models/evals-methodology/gemini-3-5-flash): All of the results are all run with the Gemini API ... with default sampling settings unless indicated otherwise. Gemini computed in-house; average of 5 runs; internal version of the Antigravity harness; single trial. Column heading is Gemini 3 Flash (gemini-3-flash-preview is the only match in the Gemini API model list). thinking level not statedSource excerptSWE-Bench Pro (Public) Diverse agentic coding tasks | Single attempt | 49.6%Location Gemini 3.5 Flash model card Evaluation table, SWE-Bench Pro (Public) row, Gemini 3 Flash column. In-house computation basis: methodology PDF, Coding / SWE-Bench Pro |
official results (Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-Max | not stated | 67.7% | — | Alibaba (Qwen) | Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Table column is Qwen3.8-Max (card: Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B). Effort not stated (the default for reasoning_effort is xhigh). Claude Code harness; 256K. All baselines evaluated on a version with problematic tasks fixed.Source excerptSWE-bench Pro | 67.7Location HF model card "Benchmark Results" table, SWE-bench Pro row, Qwen3.8-Max column (rightmost). Basis: note below the table 3 |
official results (Qwen3.8-27B (Hugging Face model card, Qwen))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-27B | not stated | 61.7% | — | Alibaba (Qwen) | Qwen3.8-27B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Effort not stated (the default for reasoning_effort is xhigh). Claude Code harness; 256K. Version with problematic tasks fixed.Source excerptAgentic codingSWE-bench Pro | 61.7Location HF model card benchmark table, SWE-bench Pro row, Qwen3.8-27B column (leftmost). Basis: note below the table SWE-bench Pro |