τ²-bench telecom (v1.0.1)
Measures whether a conversational agent using tools can interact with a simulator playing the user and complete telecom technical support tasks according to policy (dual control, with the user also operating the device) (pass^1). Submissions with tau2_bench_version v1.0.1 (the user simulator is gpt-5.2, the submission's user_simulator). There is no item for telecom task fixes in [1.0.0] of the CHANGELOG. The v1.0.1 scoring changes apply only to banking_knowledge and do not affect this domain ("Other domains are unaffected" in [1.0.1] of the CHANGELOG).
- Category
- Tool use
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 1total
- All model rankings (records) · Ranked results: 8
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
- View all model rankings (by metric)
- Cost balance (all models, records)
- Overall ranking (all models, records)
Show rankings including legacy and unavailable models (entries: 8)
third-party results (τ²-bench v1.0.1 (run by Sierra))
Submissions run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted), with user simulator gpt-5.2. Comparisons are made within each domain.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.5-397B-A17B | not stated | 97.8% | — | Sierra | τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Alibaba Cloud": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "enabled" (thinking enabled). The HF model card only distinguishes whether thinking is enabled or disabled and has no effort levels, so no effort. User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-27. Notes summary: 4 trial runs; seed 300; temperature 1.0; top_p 0.95; self-hosted with vLLM.Source excerpttelecom.pass_1=97.81Location In submissions/qwen3.5-397b-a17b-think_sierra_2026-03-02/submission.json: results.telecom.pass_1 (line 36 of file). model_name "Qwen3.5-397B-A17B". methodology.notes (line 56) |
| 2 | Claude Opus 4.5 | High | 92.3% | $0.72Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2. 4 trial runs (based on "_gpt-5.2_4trials" in the trajectories filename). The notes and evaluation_date (2026-05-05) refer to banking_knowledge (AllTools) ("Banking_knowledge domain only"); the evaluation date, seed, and simulator effort for this domain are not stated.Source excerpttelecom.pass_1=92.32; telecom.cost=0.71992Location In submissions/claude-opus-4-5_sierra_2026-02-26/submission.json: results.telecom.pass_1 (line 36 of file), results.telecom.cost (line 40). model_name "Claude Opus 4.5". trajectory_files.telecom (line 16), methodology.user_simulator (line 55). methodology.notes (line 56) and evaluation_date (line 53) refer to banking_knowledge (AllTools) |
| 3 | Gemini 3 Flash Preview | High | 91.2% | — | Sierra | τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-03-02. Notes summary: 4 trial runs; seed 123; 20 cases terminated due to too_many_errors and scored 0 points.Source excerpttelecom.pass_1=91.23Location In submissions/gemini-3-flash_sierra_2026-03-02/submission.json: results.telecom.pass_1 (line 36 of file). model_name "Gemini 3 Flash". methodology.notes (line 56) |
| 4 | Gemini 3 Pro Preview | High | 91.0% | — | Sierra | τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-03-02. Notes summary: 4 trial runs; seed not stated; infrastructure errors treated as failures (0 points) (2 cases).Source excerpttelecom.pass_1=91.01Location In submissions/gemini-3-pro_sierra_2026-03-02/submission.json: results.telecom.pass_1 (line 36 of file). model_name "Gemini 3 Pro". methodology.notes (line 56) |
| 5 | GPT-5.2 | High | 89.7% | $0.093Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2. 4 trial runs (based on "_gpt-5.2_4trials" in the trajectories filename). The notes and evaluation_date (2026-05-05) refer to banking_knowledge (AllTools) ("Banking_knowledge domain only"); the evaluation date, seed, and simulator effort for this domain are not stated.Source excerpttelecom.pass_1=89.69; telecom.cost=0.09312Location In submissions/gpt-5-2_sierra_2026-02-26/submission.json: results.telecom.pass_1 (line 36 of file), results.telecom.cost (line 40). model_name "GPT-5.2". trajectory_files.telecom (line 16), methodology.user_simulator (line 55). methodology.notes (line 56) and evaluation_date (line 53) refer to banking_knowledge (AllTools) |
| 6 | GLM-5 | not stated | 86.8% | — | Sierra | τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Zhipu AI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The official effort list has not been collected, so no effort for the source effort label "enabled" (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-27. Notes summary: 4 trial runs; seed 300; temperature 1.0; top_p 0.95; infrastructure errors treated as failures (0 points) (number of cases not stated).Source excerpttelecom.pass_1=86.84Location In submissions/glm-5-think_sierra_2026-03-02/submission.json: results.telecom.pass_1 (line 36 of file). model_name "GLM-5". methodology.notes (line 56) |
| 7 | Claude Sonnet 4.5 | not stated | 84.9% | $0.49Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The source label for effort, "enabled", is not in the official effort list, so no effort (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-24. Notes summary: 4 trial runs; seed 300; extended thinking enabled.Source excerpttelecom.pass_1=84.87; telecom.cost=0.48769Location In submissions/claude-sonnet-4-5_sierra_2026-02-26/submission.json: results.telecom.pass_1 (line 36 of file), results.telecom.cost (line 40). model_name "Claude Sonnet 4.5". methodology.notes (line 56) |
| 8 | GPT-5.2 | None | 57.2% | $0.049Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "none". User simulator gpt-5.2 (reasoning_effort low). Submission-level evaluation_date 2026-05-26. Notes summary: 4 trial runs; seed 300.Source excerpttelecom.pass_1=57.23684210526315; telecom.cost=0.04937742280701755Location In submissions/gpt-5-2-none_sierra_2026-02-26/submission.json: results.telecom.pass_1 (line 36 of file), results.telecom.cost (line 40). model_name "GPT-5.2". methodology.notes (line 56) |