τ²-bench airline (v1.0.1)
Measures whether a conversational agent using tools can interact with a simulator playing the user and complete airline customer service tasks according to policy (pass^1). Submissions with tau2_bench_version v1.0.1 (the user simulator is gpt-5.2, the submission's user_simulator). Airline task fixes were made in 1.0.0 ("Airline task fixes (27 tasks)" in [1.0.0] of the CHANGELOG), but the submission_date of the submissions shown here (2026-02-26 to 03-02) precedes the release of 1.0.0 (2026-03-18), and the submissions do not reveal whether they include the fixes. The v1.0.1 scoring changes apply only to banking_knowledge and do not affect this domain ("Other domains are unaffected" in [1.0.1] of the CHANGELOG).
- Category
- Tool use
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 1total
- All model rankings (records) · Ranked results: 8
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
- View all model rankings (by metric)
- Cost balance (all models, records)
- Overall ranking (all models, records)
Show rankings including legacy and unavailable models (entries: 8)
third-party results (τ²-bench v1.0.1 (run by Sierra))
Submissions run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted), with user simulator gpt-5.2. Comparisons are made within each domain.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.5 | High | 84.0% | $0.40Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2. 4 trial runs (based on "_gpt-5.2_4trials" in the trajectories filename). The notes and evaluation_date (2026-05-05) refer to banking_knowledge (AllTools) ("Banking_knowledge domain only"); the evaluation date, seed, and simulator effort for this domain are not stated.Source excerptairline.pass_1=84.0; airline.cost=0.39919Location In submissions/claude-opus-4-5_sierra_2026-02-26/submission.json: results.airline.pass_1 (line 22 of file), results.airline.cost (line 26). model_name "Claude Opus 4.5". trajectory_files.airline (line 14), methodology.user_simulator (line 55). methodology.notes (line 56) and evaluation_date (line 53) refer to banking_knowledge (AllTools) |
| 2 | GPT-5.2 | High | 83.0% | $0.11Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2. 4 trial runs (based on "_gpt-5.2_4trials" in the trajectories filename). The notes and evaluation_date (2026-05-05) refer to banking_knowledge (AllTools) ("Banking_knowledge domain only"); the evaluation date, seed, and simulator effort for this domain are not stated.Source excerptairline.pass_1=83.0; airline.cost=0.11379Location In submissions/gpt-5-2_sierra_2026-02-26/submission.json: results.airline.pass_1 (line 22 of file), results.airline.cost (line 26). model_name "GPT-5.2". trajectory_files.airline (line 14), methodology.user_simulator (line 55). methodology.notes (line 56) and evaluation_date (line 53) refer to banking_knowledge (AllTools) |
| 3 | Gemini 3 Flash Preview | High | 82.5% | — | Sierra | τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-03-02. Notes summary: 4 trial runs; seed 123.Source excerptairline.pass_1=82.5Location In submissions/gemini-3-flash_sierra_2026-03-02/submission.json: results.airline.pass_1 (line 22 of file). model_name "Gemini 3 Flash". methodology.notes (line 56) |
| 3 | GLM-5 | not stated | 82.5% | — | Sierra | τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Zhipu AI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The official effort list has not been collected, so no effort for the source effort label "enabled" (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-27. Notes summary: 4 trial runs; seed 300; temperature 1.0; top_p 0.95; infrastructure errors treated as failures (0 points) (number of cases not stated).Source excerptairline.pass_1=82.5Location In submissions/glm-5-think_sierra_2026-03-02/submission.json: results.airline.pass_1 (line 22 of file). model_name "GLM-5". methodology.notes (line 56) |
| 5 | Qwen3.5-397B-A17B | not stated | 81.5% | — | Sierra | τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Alibaba Cloud": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "enabled" (thinking enabled). The HF model card only distinguishes whether thinking is enabled or disabled and has no effort levels, so no effort. User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-27. Notes summary: 4 trial runs; seed 300; temperature 1.0; top_p 0.95; self-hosted with vLLM.Source excerptairline.pass_1=81.5Location In submissions/qwen3.5-397b-a17b-think_sierra_2026-03-02/submission.json: results.airline.pass_1 (line 22 of file). model_name "Qwen3.5-397B-A17B". methodology.notes (line 56) |
| 6 | Gemini 3 Pro Preview | High | 80.5% | — | Sierra | τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-03-02. Notes summary: 4 trial runs; seed not stated; infrastructure errors treated as failures (0 points) (3 cases).Source excerptairline.pass_1=80.5Location In submissions/gemini-3-pro_sierra_2026-03-02/submission.json: results.airline.pass_1 (line 22 of file). model_name "Gemini 3 Pro". methodology.notes (line 56) |
| 7 | Claude Sonnet 4.5 | not stated | 72.0% | $0.30Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The source label for effort, "enabled", is not in the official effort list, so no effort (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). Evaluation date 2026-02-24. Notes summary: 4 trial runs; seed 300; extended thinking enabled.Source excerptairline.pass_1=72.0; airline.cost=0.29646Location In submissions/claude-sonnet-4-5_sierra_2026-02-26/submission.json: results.airline.pass_1 (line 22 of file), results.airline.cost (line 26). model_name "Claude Sonnet 4.5". methodology.notes (line 56) |
| 8 | GPT-5.2 | None | 52.5% | $0.054Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "none". User simulator gpt-5.2 (reasoning_effort low). Submission-level evaluation_date 2026-05-26. Notes summary: 4 trial runs; seed 300.Source excerptairline.pass_1=52.5; airline.cost=0.053692803500000004Location In submissions/gpt-5-2-none_sierra_2026-02-26/submission.json: results.airline.pass_1 (line 22 of file), results.airline.cost (line 26). model_name "GPT-5.2". methodology.notes (line 56) |