ZeroBench main (with tools, pass@5)

The pass@5 on difficult image reasoning problems (main set), with tools over 5 trial runs.

Category: Multimodal
Unit: Percent (%); Higher is better
Comparison groups: 3total; All model rankings (records) · Ranked results: 3

All model rankings (records)

These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.

Show rankings including legacy and unavailable models (entries: 3)

official results (Kimi K3.1 Pro (Hugging Face model card, moonshotai))

Rank	Model	effort	Score	Source cost (metric)	measured/reported by	source
1	Kimi K3.1 Pro	Max	63.0%	—	Moonshot AI	Kimi K3.1 Pro (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table column heading Kimi K3.1 Pro (max), table note: all Kimi K3.1 Pro results use reasoning effort max, temperature 1.0, top-p 0.95. With tools (Python); 5 runs. Source excerpt ZeroBench_main w/ python (pass@5) \| 63.0Location HF model card "3. Evaluation Results" table, ZeroBench_main w/ python (pass@5) row, Kimi K3.1 Pro (max) column (leftmost numeric column). Supporting footnote: Agentic 11

official results (Kimi K3 (Hugging Face model card, moonshotai))

Rank	Model	effort	Score	Source cost (metric)	measured/reported by	source
1	Kimi K3	Max	41.0%	—	Moonshot AI	Kimi K3 (Hugging Face model card, moonshotai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by Moonshot AI). Table note: All Kimi K3 results are obtained with reasoning effort set to 'max' and temperature = 1.0. With tools (Python). Source excerpt ZeroBench (pass@5) \| 23.0 / 41.0Location HF model card "3. Evaluation Results" table, ZeroBench (pass@5) (right-hand value in the without tools / with tools pair) row, Kimi K3 (max) column (leftmost numeric column). Supporting footnote: Multimodal (5 runs for ZeroBench under the official settings)

official results (DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai))

Rank	Model	effort	Score	Source cost (metric)	measured/reported by	source
1	DeepSeek-V4.1-Flash	not stated	49.0%	—	DeepSeek	DeepSeek-V4.1-Flash (Hugging Face model card, deepseek-ai)Retrieved 2026-11-13 · Vendor-reported value (self-reported by DeepSeek). Card "Instruct Model": All instruct results below use the maximum effort setting (reasoning_effort=100), temperature=1.0, top_p=0.95. The correspondence of reasoning_effort=100 to the API effort names (low/high/max) has no official statement, so effort is null. Evaluated with the Claude Code harness and 512k context. Source excerpt ZeroBench-main w/ tools (Pass@5) \| 49.0Location HF model card "Comparison with frontier models (Max reasoning effort)" table, ZeroBench-main w/ tools (Pass@5) row, DS-V4.1-Flash column (rightmost)