modelbench日本語

DeepSWE v1.1

Success rate on long-term software engineering tasks (v1.1).

Category
Agentic coding
Unit
Percent (%)
Higher is better
Comparison groups
13total
All model rankings (records) · Ranked results: 14

All model rankings (records)

These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.

Show rankings including legacy and unavailable models (entries: 14)

official results (Gemini 4 Argon model card)

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Gemini 4 Argonnot stated77.9%—GoogleGemini 4 Argon model cardRetrieved 2026-11-12 · Vendor-reported value. Self-reported by Google (model card, as of 2026-10). thinking setting not stated
Source excerptDeepSWE v1.1 | 77.9%

official results (Gemini 3.8 Flash model card)

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Gemini 3.8 Flashnot stated73.7%—GoogleGemini 3.8 Flash model cardRetrieved 2026-11-12 · Vendor-reported value. Self-reported by Google (model card, as of 2026-09). thinking setting not stated
Source excerptDeepSWE v1.1 | 73.7%

official results (Grok 4.7 (xAI news))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Grok 4.7High71.0%—xAIGrok 4.7 (xAI news)Retrieved 2026-11-12 · Vendor-reported value. Self-reported by xAI. Table heading is Grok 4.7 (xHigh). A footnote states that DeepSWE is a high effort value
Source excerptDeepSWE v1.1 | Grok 4.7 (xHigh) 71.0(footnote: high effort)
2Grok 4.6High65.2%—xAIGrok 4.7 (xAI news)Retrieved 2026-11-12 · Vendor-reported value. Self-reported by xAI (comparison column on the Grok 4.7 announcement page)
Source excerptDeepSWE v1.1 | Grok 4.6 (High) 65.2

official results (GLM-5.3 (Z.ai docs))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1GLM-5.3not stated66.9%—Z.aiGLM-5.3 (Z.ai docs)Retrieved 2026-11-12 · Vendor-reported value. Self-reported by Z.ai. effort not stated (default is max)
Source excerptDeepSWE v1.1 66.9

official results (Introducing Mistral Large 4)

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Mistral Large 4not stated61.7%—Mistral AIIntroducing Mistral Large 4Retrieved 2026-11-12 · Vendor-reported value. Self-reported by Mistral (as of the 2026-10-06 preview version; a separate article says the later open weights version improved with additional RL)
Source excerptDeepSWE v1.1: 61.7%

official results (Kimi K3.1 Pro (Hugging Face model card, moonshotai))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Kimi K3.1 ProMax75.5%—Moonshot AIKimi K3.1 Pro (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table column heading Kimi K3.1 Pro (max), table note: all Kimi K3.1 Pro results use reasoning effort max, temperature 1.0, top-p 0.95. Evaluated with Claude Code.
Source excerptDeepSWE v1.1 | 75.5Location HF model card "3. Evaluation Results" table, DeepSWE v1.1 row, Kimi K3.1 Pro (max) column (leftmost numeric column). Supporting footnote: Coding 5

official results (Kimi K3 (Hugging Face model card, moonshotai))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Kimi K3Max67.5%—Moonshot AIKimi K3 (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table note: All Kimi K3 results are obtained with reasoning effort set to 'max' and temperature = 1.0. Kimi Code harness; DeepSWE v1.1 tasks (67.3 is separately stated for mini-SWE-agent).
Source excerptDeepSWE | 67.5Location HF model card "3. Evaluation Results" table, DeepSWE row, Kimi K3 (max) column (leftmost numeric column). Supporting footnote: Coding / DeepSWE

official results (GLM-5.3-Flash (Z.ai docs))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1GLM-5.3-Flashnot stated63.4%—Z.aiGLM-5.3-Flash (Z.ai docs)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Z.ai). Value in the main text. Evaluation conditions are from the Footnotes of the HF card (zai-org/GLM-5.3-Flash, MIT) for the same announcement. Effort not stated (recommended setting: reasoning_effort: max). mini-swe-agent harness; 6h timeout.
Source excerptGLM-5.3-Flash … 63.4 … on DeepSWE v1.1Location Text in the "Competitive Performance at Flash Cost" section of the docs.z.ai GLM-5.3-Flash page. Conditions: HF card Footnotes, DeepSWE (We run DeepSWE using the mini-swe-agent harness)

official results (Gemini 3.7 Flash model card)

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Gemini 3.7 FlashHigh65.3%—GoogleGemini 3.7 Flash model cardRetrieved 2026-11-13 · Vendor-reported value. Table in the Google model card. methodology PDF (deepmind.com/models/evals-methodology/gemini-3-7-flash): All of the results are run with the Gemini API ... with default sampling settings unless indicated otherwise. Results for Gemini 3.7 Flash are self computed, and use a mini swe agent harness with LiteLLM version 1.96 and high thinking (3.6 Flash is not used because its value is from the Datacurve leaderboard)
Source excerptDeepSWE v1.1 Long-horizon software engineering | 65.3%Location Gemini 3.7 Flash model card Evaluation table, DeepSWE v1.1 row, Gemini 3.7 Flash column. In-house computation basis: methodology PDF, Coding / DeepSWE v1.1

official results (Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Qwen3.8-Maxnot stated56.6%—Alibaba (Qwen)Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Table column is Qwen3.8-Max (card: Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B). Effort not stated (the default for reasoning_effort is xhigh). Higher of Claude Code and mini-SWE-agent (Claude Code for Qwen3.8-Max).
Source excerptDeepSWE 1.1 | 56.6Location HF model card "Benchmark Results" table, DeepSWE 1.1 row, Qwen3.8-Max column (rightmost). Basis: note below the table 4

official results (Qwen3.8-27B (Hugging Face model card, Qwen))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1Qwen3.8-27Bnot stated42.2%—Alibaba (Qwen)Qwen3.8-27B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Effort not stated (the default for reasoning_effort is xhigh). Claude Code harness; 256K.
Source excerptAgentic codingDeepSWE 1.1 | 42.2Location HF model card benchmark table, DeepSWE 1.1 row, Qwen3.8-27B column (leftmost). Basis: note below the table DeepSWE 1.1

official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1DeepSeek-V4.1-Flashnot stated74.2%—DeepSeekDeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) has no official statement, so effort is null. Evaluated with the mini-SWE harness.
Source excerptDeepSWE v1.1 (Resolved) | 74.2Location HF model card "Comparison with frontier models (Max reasoning effort)" table, DeepSWE v1.1 (Resolved) row, DS-V4.1-Flash column (rightmost)

official results (DeepSeek-V4.1-Flash-1017 (Hugging Face model card, deepseek-ai))

RankModeleffortScoreSource cost (metric)measured/reported bysource
1DeepSeek-V4.1-Flash-1017not stated77.0%—DeepSeekDeepSeek-V4.1-Flash-1017 (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card note: We use the Claude Code harness for Terminal-Bench 4.0 and the Mini-SWE-V2 harness for DeepSWE v1.1. Effort not stated. Mini-SWE-V2 harness.
Source excerptDeepSWE v1.1 | 77.0Location Comparison table at the start of the HF model card, DeepSWE v1.1 row, DS-V4.1-Flash-1017 column (leftmost numeric column)