AutomationBench v1.0.6
Business workflow automation created by Zapier (v1.0.6). Kept separate from automationbench of a different version (Anthropic figure) and the public 600-task version.
- Category
- Tool use
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 3total
- All model rankings (records) · Ranked results: 3
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
Show rankings including legacy and unavailable models (entries: 3)
official results (GLM-5.3 (Hugging Face model card, zai-org))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | GLM-5.3 | not stated | 48.2% | — | Z.ai | GLM-5.3 (Hugging Face model card, zai-org)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Z.ai). Card Note: the default for reasoning_effort is max; use the default max to reproduce the benchmarks. v1.0.6 (including the fix in PR #13). Effort not stated in this footnote.Source excerptAutomationBench (v1.0.6) | 48.2Location HF model card "Benchmark" table, AutomationBench (v1.0.6) row, GLM-5.3 column (leftmost numeric column). Basis: Footnotes, AutomationBench |
official results (GLM-5.3-Flash (Z.ai docs))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | GLM-5.3-Flash | not stated | 48.8% | — | Z.ai | GLM-5.3-Flash (Z.ai docs)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Z.ai). Value in the main text. Evaluation conditions are from the Footnotes of the HF card (zai-org/GLM-5.3-Flash, MIT) for the same announcement. Effort not stated (recommended setting: reasoning_effort: max). AutomationBench v1.0.6.Source excerptGLM-5.3-Flash … 48.8 … on AutomationBenchLocation Text in the "Competitive Performance at Flash Cost" section of the docs.z.ai GLM-5.3-Flash page. Version: HF card Footnotes, AutomationBench (We evaluate on AutomationBench v1.0.6) |
official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-V4.1-Flash | not stated | 54.8% | — | DeepSeek | DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) is not stated officially, so effort is null. Evaluated using the official scaffold. The version is absent from this card, but the table in the DeepSeek-V4.1-Flash-1017 card lists the same 54.8 in the "Automation-Bench v1.0.6" row.Source excerptAutomationBench (Pass@1) | 54.8Location HF model card "Comparison with frontier models (Max reasoning effort)" table, AutomationBench (Pass@1) row, DS-V4.1-Flash column (rightmost) |