τ²-bench banking_knowledge (v1.0.1)
Measures whether a conversational agent using tools can interact with a simulator playing the user and complete banking customer service tasks that require knowledge base search according to policy (pass^1). Submissions with tau2_bench_version v1.0.1. The banking_knowledge scoring was corrected in v1.0.1, so values cannot be compared with those from versions before 1.0.1 ("banking_knowledge scores are not comparable across this release" in [1.0.1] of the CHANGELOG).
- Category
- Tool use
- Unit
- Percent (%)
- Higher is better
- Comparison groups
- 4total
- All model rankings (records) · Ranked results: 22
All model rankings (records)
These rankings are recalculated by comparison group including legacy models and models unavailable in Claude Code / Codex. They differ from the rankings above because the set of models differs.
- View all model rankings (by metric)
- Cost balance (all models, records)
- Overall ranking (all models, records)
Show rankings including legacy and unavailable models (entries: 22)
third-party results (τ²-bench v1.0.1 banking_knowledge (run by Sierra, search alltools))
Submissions for banking_knowledge run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted) and have the search configuration (retrieval_config) alltools. Includes submissions with trajectories suppressed for a preliminary comparison ("Trajectories suppressed" in verification.details). Results with a different search configuration are not compared.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Qwen3.8-Max | Xhigh | 55.2% | — | Sierra | τ²-bench leaderboard submission qwen3-8-max_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission qwen3-8-max_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Qwen": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "xhigh". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-08-03. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=55.154639175257735Location In submissions/qwen3-8-max_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Qwen 3.8 Max". methodology.notes (line 36) |
| 2 | Claude Opus 5 | Max | 48.7% | — | Sierra | τ²-bench leaderboard submission claude-opus-5_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-5_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-08-03. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=48.71134020618557Location In submissions/claude-opus-5_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Claude Opus 5". methodology.notes (line 36) |
| 3 | Grok 4.5 | High | 47.9% | — | Sierra | τ²-bench leaderboard submission grok-4-5_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission grok-4-5_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "xAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-24. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=47.93814432989691Location In submissions/grok-4-5_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Grok 4.5". methodology.notes (line 36) |
| 4 | GPT-5.6 Sol | Xhigh | 46.9% | — | Sierra | τ²-bench leaderboard submission gpt-5-6-sol_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-6-sol_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "xhigh". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-22. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=46.90721649484536Location In submissions/gpt-5-6-sol_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "GPT-5.6-sol". methodology.notes (line 36) |
| 5 | GPT-5.5 | Xhigh | 44.6% | — | Sierra | τ²-bench leaderboard submission gpt-5-5_sierra_2026-05-05 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-5_sierra_2026-05-05 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "xhigh". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-22. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=44.58762886597938Location In submissions/gpt-5-5_sierra_2026-05-05/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "GPT-5.5". methodology.notes (line 36) |
| 6 | Claude Opus 4.7 | Max | 40.2% | — | Sierra | τ²-bench leaderboard submission claude-opus-4-7_sierra_2026-05-05 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-7_sierra_2026-05-05 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-23. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=40.20618556701031Location In submissions/claude-opus-4-7_sierra_2026-05-05/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Claude Opus 4.7". methodology.notes (line 36) |
| 7 | Claude Fable 5 | Max | 39.7% | — | Sierra | τ²-bench leaderboard submission claude-fable-5_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-fable-5_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-23. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=39.69072164948454Location In submissions/claude-fable-5_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Claude Fable 5". methodology.notes (line 36) |
| 7 | Claude Opus 4.8 | Max | 39.7% | — | Sierra | τ²-bench leaderboard submission claude-opus-4-8_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-8_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-23. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=39.69072164948454Location In submissions/claude-opus-4-8_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Claude Opus 4.8". methodology.notes (line 36) |
| 9 | GPT-5.4 | Xhigh | 39.4% | $1.84Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-4_sierra_2026-03-25 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-4_sierra_2026-03-25 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "xhigh". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-06. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=39.43; banking_knowledge.cost=1.8423Location In submissions/gpt-5-4_sierra_2026-03-25/submission.json: results.banking_knowledge.pass_1 (line 26 of file), results.banking_knowledge.cost (line 30). model_name "GPT-5.4". methodology.notes (line 38) |
| 10 | GLM-5.2 | not stated | 37.1% | — | Sierra | τ²-bench leaderboard submission glm-5-2_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission glm-5-2_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Z.ai": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The official effort list has not been collected, so no effort for the source effort label "xhigh" (reasoning_effort "xhigh"). User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-24. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=37.11340206185567Location In submissions/glm-5-2_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "GLM-5.2". methodology.notes (line 36) |
| 10 | Kimi K3 | Max | 37.1% | — | Sierra | τ²-bench leaderboard submission kimi-k3_sierra_2026-08-04 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission kimi-k3_sierra_2026-08-04 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Moonshot AI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-07-24. Notes summary: 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=37.11340206185567Location In submissions/kimi-k3_sierra_2026-08-04/submission.json: results.banking_knowledge.pass_1 (line 17 of file). model_name "Kimi K3". methodology.notes (line 36) |
| 12 | GPT-5.2 | High | 32.2% | — | Sierra | τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-05. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=32.22Location In submissions/gpt-5-2_sierra_2026-02-26/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "GPT-5.2". methodology.notes (line 56) |
| 13 | Claude Opus 4.6 | Max | 27.3% | — | Sierra | τ²-bench leaderboard submission claude-opus-4-6_sierra_2026-05-05 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-6_sierra_2026-05-05 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified by the leaderboard (prompts not modified, no questions omitted). Trajectories are not public (preliminary comparison). The value is pass^1 (%). reasoning_effort "max". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-06. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=27.32Location In submissions/claude-opus-4-6_sierra_2026-05-05/submission.json: results.banking_knowledge.pass_1 (line 22 of file). model_name "Claude Opus 4.6". verification.details (line 39) is "Trajectories suppressed for this preliminary AllTools comparison". methodology.notes (line 35) |
| 14 | Gemini 3.1 Pro Preview | High | 26.0% | — | Sierra | τ²-bench leaderboard submission gemini-3-1-pro-preview_sierra_2026-05-05 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-1-pro-preview_sierra_2026-05-05 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified by the leaderboard (prompts not modified, no questions omitted). Trajectories are not public (preliminary comparison). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-05. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=26.03Location In submissions/gemini-3-1-pro-preview_sierra_2026-05-05/submission.json: results.banking_knowledge.pass_1 (line 22 of file). model_name "Gemini 3.1 Pro Preview". verification.details (line 39) is "Trajectories suppressed for this preliminary AllTools comparison". methodology.notes (line 35) |
| 15 | Claude Opus 4.5 | High | 24.7% | — | Sierra | τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-opus-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-05. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=24.74Location In submissions/claude-opus-4-5_sierra_2026-02-26/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "Claude Opus 4.5". methodology.notes (line 56) |
| 16 | Gemini 2.5 Pro | High | 13.7% | — | Sierra | τ²-bench leaderboard submission gemini-2-5-pro_sierra_2026-05-05 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-2-5-pro_sierra_2026-05-05 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified by the leaderboard (prompts not modified, no questions omitted). Trajectories are not public (preliminary comparison). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "alltools". Evaluation date 2026-05-05. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=13.66Location In submissions/gemini-2-5-pro_sierra_2026-05-05/submission.json: results.banking_knowledge.pass_1 (line 22 of file). model_name "Gemini 2.5 Pro". verification.details (line 39) is "Trajectories suppressed for this preliminary AllTools comparison". methodology.notes (line 35) |
third-party results (τ²-bench v1.0.1 banking_knowledge (run by Sierra, search terminal))
Submissions for banking_knowledge run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted) and have the search configuration (retrieval_config) terminal. Results with a different search configuration are not compared.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | Gemini 3 Flash Preview | High | 27.3% | — | Sierra | τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-flash_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "terminal". Evaluation date 2026-03-02. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 123; 52 cases terminated due to max_steps and scored 0 points.Source excerptbanking_knowledge.pass_1=27.32Location In submissions/gemini-3-flash_sierra_2026-03-02/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "Gemini 3 Flash". methodology.notes (line 56) |
| 2 | Claude Sonnet 4.5 | not stated | 25.3% | $4.05Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission claude-sonnet-4-5_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Anthropic": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The source label for effort, "enabled", is not in the official effort list, so no effort (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "terminal". Evaluation date 2026-02-24. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300; extended thinking enabled.Source excerptbanking_knowledge.pass_1=25.26; banking_knowledge.cost=4.04814Location In submissions/claude-sonnet-4-5_sierra_2026-02-26/submission.json: results.banking_knowledge.pass_1 (line 43 of file), results.banking_knowledge.cost (line 47). model_name "Claude Sonnet 4.5". methodology.notes (line 56) |
| 3 | Gemini 3 Pro Preview | High | 18.0% | — | Sierra | τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gemini-3-pro_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Google": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "high". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "terminal". Evaluation date 2026-03-02. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed not stated; 42 cases terminated due to max_steps and scored 0 points.Source excerptbanking_knowledge.pass_1=18.04Location In submissions/gemini-3-pro_sierra_2026-03-02/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "Gemini 3 Pro". methodology.notes (line 56) |
third-party results (τ²-bench v1.0.1 banking_knowledge (run by Sierra, search text-emb-3-large))
Submissions for banking_knowledge run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted) and have the search configuration (retrieval_config) text-emb-3-large. Results with a different search configuration are not compared.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | GLM-5 | not stated | 9.8% | — | Sierra | τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission glm-5-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Zhipu AI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). The official effort list has not been collected, so no effort for the source effort label "enabled" (reasoning_effort "enabled"). User simulator gpt-5.2 (reasoning_effort low). retrieval_config "text-emb-3-large". Evaluation date 2026-02-27. Note: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300; temperature 1.0; top_p 0.95; infrastructure errors treated as failures (number of cases unknown).Source excerptbanking_knowledge.pass_1=9.79Location In submissions/glm-5-think_sierra_2026-03-02/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "GLM-5". methodology.notes (line 56) |
| 1 | Qwen3.5-397B-A17B | not stated | 9.8% | — | Sierra | τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission qwen3.5-397b-a17b-think_sierra_2026-03-02 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "Alibaba Cloud": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "enabled" (thinking enabled; the HF model card has no levels, so no effort). User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "text-emb-3-large". Evaluation date 2026-02-27. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300; temperature 1.0; top_p 0.95; self-hosted with vLLM.Source excerptbanking_knowledge.pass_1=9.79Location In submissions/qwen3.5-397b-a17b-think_sierra_2026-03-02/submission.json: results.banking_knowledge.pass_1 (line 43 of file). model_name "Qwen3.5-397B-A17B". methodology.notes (line 56) |
third-party results (τ²-bench v1.0.1 banking_knowledge (run by Sierra, search qwen_embeddings))
Submissions for banking_knowledge run by Sierra with tau2_bench_version 1.0.1 that are verified by the leaderboard criteria (trajectories_available is true, prompts not modified, and no questions omitted) and have the search configuration (retrieval_config) qwen_embeddings. Results with a different search configuration are not compared.
| Rank | Model | effort | Score | Source cost (metric) | measured/reported by | source |
|---|---|---|---|---|---|---|
| 1 | GPT-5.2 | None | 12.6% | $0.24Average cost in USD to run one trajectory in this domain | Sierra | τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (sierra-research/tau2-bench, MIT)Retrieved 2026-11-13 · τ²-bench leaderboard submission gpt-5-2-none_sierra_2026-02-26 (submission_type standard, tau2_bench_version 1.0.1). Submitter "Sierra", model_organization "OpenAI": Run by Sierra, which operates τ²-bench (third-party measurement). Verified on the leaderboard (trajectories available, prompts not modified, no questions omitted). The value is pass^1 (%). reasoning_effort "none". User simulator gpt-5.2 (reasoning_effort low). Search configuration retrieval_config "qwen_embeddings". Evaluation date 2026-05-26. Notes summary: rescored on 2026-07-15 following a scoring correction in v1.0.1; 4 trial runs; seed 300.Source excerptbanking_knowledge.pass_1=12.63; banking_knowledge.cost=0.2433424974226804Location In submissions/gpt-5-2-none_sierra_2026-02-26/submission.json: results.banking_knowledge.pass_1 (line 43 of file), results.banking_knowledge.cost (line 47). model_name "GPT-5.2". methodology.notes (line 56) |