ProgramBench(Almost Solved)
Proportion of tasks almost solved (score of at least 95%) in challenges to build programs from scratch.
- Category
- Agentic coding
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 2total
- All model rankings (records) · Ranked results: 2
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 2)
official results (Kimi K3.1 Pro (Hugging Face model card, moonshotai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Kimi K3.1 Pro | Max | 31.5% | — | Moonshot AI | Kimi K3.1 Pro (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table column heading Kimi K3.1 Pro (max), table note: all Kimi K3.1 Pro results use reasoning effort max, temperature 1.0, top-p 0.95. Binary aggregation assigning 1 when a task's ProgramBench score is at least 95%; evaluated with Claude Code.Source excerptProgram Bench | 31.5Location HF model card "3. Evaluation Results" table, Program Bench row, Kimi K3.1 Pro (max) column (leftmost numeric column). Supporting footnote: Coding 4 |
official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-V4.1-Flash | not stated | 20.3% | — | DeepSeek | DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) has no official statement, so effort is null. Evaluated in Minimal mode of DeepSeek Harness.Source excerptProgramBench (Almost@1) | 20.3Location HF model card "Comparison with frontier models (Max reasoning effort)" table, ProgramBench (Almost@1) row, DS-V4.1-Flash column (rightmost) |