NL2Repo-Bench
Score on tasks to build an entire repository from natural-language specifications.
- Category
- Agentic coding
- Unit
- Score
- Higher is better
- Comparison groups
- 4total
- All model rankings (records) · Ranked results: 4
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 4)
official results (GLM-5.3 (Hugging Face model card, zai-org))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | GLM-5.3 | not stated | 58 | — | Z.ai | GLM-5.3 (Hugging Face model card, zai-org)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Z.ai). Card Note: the default for reasoning_effort is max; use the default max to reproduce the benchmarks. 1M context. Effort not stated in this footnote.Source excerptNL2Repo | 58.0Location HF model card "Benchmark" table, NL2Repo row, GLM-5.3 column (leftmost numeric column). Basis: Footnotes, NL2Repo |
official results (Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-Max | not stated | 55.9 | — | Alibaba (Qwen) | Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Table column is Qwen3.8-Max (card: Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B). Effort not stated (the default for reasoning_effort is xhigh). Claude Code harness.Source excerptNL2Repo-Bench | 55.9Location HF model card "Benchmark Results" table, NL2Repo-Bench row, Qwen3.8-Max column (rightmost). Basis: note below the table 5 |
official results (Qwen3.8-27B (Hugging Face model card, Qwen))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-27B | not stated | 42.3 | — | Alibaba (Qwen) | Qwen3.8-27B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Effort not stated (the default for reasoning_effort is xhigh). Claude Code harness.Source excerptRepo-level code generationNL2Repo-Bench | 42.3Location HF model card benchmark table, NL2Repo-Bench row, Qwen3.8-27B column (leftmost). Basis: note below the table NL2Repo-Bench |
official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-V4.1-Flash | not stated | 64 | — | DeepSeek | DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) has no official statement, so effort is null. Evaluated in Minimal mode of DeepSeek Harness. The comparison table in the HF model card for DeepSeek-V4.1-Flash-1017 gives 65.4 for NL2Repo-Bench on the same DS-V4.1-Flash, differing from 64.0 in this card (the value follows this card).Source excerptNL2Repo-Bench (Score) | 64.0Location HF model card "Comparison with frontier models (Max reasoning effort)" table, NL2Repo-Bench (Score) row, DS-V4.1-Flash column (rightmost) |