AutomationBench (600 public tasks)
Subset of AutomationBench's 600 public tasks.
- Category
- Tool use
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 2total
- All model rankings (records) · Ranked results: 2
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 2)
official results (Kimi K3 (Hugging Face model card, moonshotai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Kimi K3 | Max | 30.8% | — | Moonshot AI | Kimi K3 (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table note: All Kimi K3 results are obtained with reasoning effort set to 'max' and temperature = 1.0. Subset of 600 public tasks; official GitHub configuration.Source excerptAutomationBench | 30.8Location HF model card "3. Evaluation Results" table, AutomationBench row, Kimi K3 (max) column (leftmost numeric column). Supporting footnote: Agentic / AutomationBench |
official results (Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-Max | not stated | 27.3% | — | Alibaba (Qwen) | Qwen3.8-2.4T-A95B (Hugging Face model card, Qwen)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Alibaba's Qwen team). Table column is Qwen3.8-Max (card: Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B). Effort not stated (the default for reasoning_effort is xhigh). Subset of 600 public tasks.Source excerptAutomation-Bench (Pass@1) | 27.3Location HF model card "Benchmark Results" table, Automation-Bench (Pass@1) row, Qwen3.8-Max column (rightmost). Basis: note below the table 16 |