About

Data collection

• Model names, effort options, prices, and scores are transcribed from source pages with retrieval dates. We do not fill gaps from memory or conjecture.
• For each value, we record the source URL, source type (official, third-party, or paper), retrieval date, source excerpt, and measurement conditions.
• “Source excerpt” in the table presents the relevant source material in its original language without translation. It includes not only excerpts but also formatted table, CSV, and chart labels, so it is not necessarily a verbatim quotation and is not enclosed in quotation marks. “Location” is our note identifying where the value appears in the source.
• Values absent from sources are left blank; we do not interpolate.
• Values read from source charts are identified as approximate (hollow points on charts and ≈ in tables).
• Before citing a site, we check its publication license and terms and record them under Sources.

Cost axis

• API prices (input, output, and cache) are set per model and do not vary by effort. What varies with effort is token consumption.
• The chart's x-axis therefore uses costs reported by sources (per task, per trial, estimates, and so on; the metric appears in the chart and table). We do not mix in values calculated from pricing tables.
• Points without source costs cannot be plotted on a logarithmic axis, so only their count is shown. All points can be displayed with the “effort order” x-axis.

Comparison caveats

• Official announcements, third-party measurements, and papers use different measurement conditions, such as harnesses and settings. Charts, rankings, and table sorting compare only values with the same source type and measurement conditions (from the same source or sources verified to use the same conditions).
• Scores can change with versions or settings even for the same benchmark. Measurement conditions for each value are available on model and benchmark pages.
• When multiple source pages show the same chart and their conditions and values are verified to match, we combine them into one comparison group. If another page also reports the same value as a point in that group, we exclude that row from comparisons (chart comparison groups, rankings, overall aggregations, and role recommendations) to avoid double-counting. It remains on model and benchmark pages and in the chart's data points table as “Evidence (same value as comparison group …)”.
• Benchmark rankings do not guarantee usability in actual work.
• “Only models available in Claude Code / Codex” displays only models confirmed available in each tool's official documentation. Unconfirmed models are not shown.

Rankings and recommendations by role

• Models ranked: Rankings, the home page's overall ranking, and rankings on metric pages include only models currently available in Claude Code / Codex (confirmed in official documentation and not legacy models). Rankings including legacy models and models unavailable in Claude Code / Codex are retained as records in All model rankings and under “All model rankings (records)” on metric pages. Ranking all models and then hiding some would let hidden models affect percentiles and frontiers, so ranks, percentiles, and frontiers are recalculated using only displayed models. The two rankings may therefore differ.
• By metric: Rankings are made within each comparison group (the same source, or sources verified to use the same measurement conditions).
• Overall (performance first): For each source type, only model × effort entries with values for every group in the same set of comparison groups are aggregated. Within each group, included entries are ranked and converted to percentiles (first = 100, last = 0). These are averaged across groups of the same benchmark to obtain a benchmark value, then averaged across benchmarks for ranking. This prevents a benchmark with two comparison groups (for example standard and best-of-16) from counting twice. An aggregation requires at least 3 models and 3 benchmarks (multiple groups for one benchmark count as one). Coverage is always shown: the number of benchmarks used in the aggregation and, for each row, the number with values / the number with values of the same source type. The model and benchmark thresholds are counted among the models being ranked. Aggregations are ordered by descending model count, benchmark count, then entry count. The aggregation containing the most models currently available in Claude Code / Codex opens first (ties follow that order). In the rankings of currently available models, this is the aggregation with the most models. The home page shows through rank 8 in the first aggregation, including all ties at rank 8.
• We use rank percentiles rather than the values themselves (min-max normalization) to avoid dependence on units such as % and Elo, outliers, and models outside the aggregation (the site's model selection). This discards the size of differences: a small gap and a large gap both count as one rank difference. Scores change with the models included and are coarser in smaller aggregations. Do not compare scores across aggregations, as their groups differ.
• Cost balance: Source costs have different meanings across benchmarks, so we neither combine them across benchmarks nor create our own coefficients. Within each comparison group, we show points for which no better score exists at the same or lower cost (the Pareto frontier), as well as score differences and cost ratios between adjacent efforts with values for the same model.
• Recommendations by role: Roles and the benchmark categories emphasized for each role are editorial choices made by this site. For models confirmed available in each tool (Claude Code / Codex), we show the top result and frontier candidates in each comparison group belonging to those categories. A group with only one value has no comparison, so its value is shown as “No comparison” rather than first place. A group with multiple values from only different efforts of the same model is not a comparison between models, so no first place or frontier is shown; values by effort and changes with increasing effort are shown for reference as “Only effort comparisons for the same model”. Groups with more distinct models are listed first. Roles without matching results show “No data”; we do not fill gaps by conjecture.
• Summary by role: For each role × tool, we count the following for current models available in that tool. We do not create coefficients or a single “recommended” model.
  • Denominator N: the number of comparison groups in the role's categories that have at least 2 distinct models and a value for that model. It is always shown. Legacy models are not listed as rows but are counted as comparisons within groups, as in the group details below.
  • Top score a: the number of those N groups in which any effort of the model had the top score, including ties.
  • Plateauing effort k: the number of those N groups with an effort that cost more than the next lower effort with a value for the same model but did not improve the score (did not decrease it for lower-is-better metrics), along with the efforts involved. Groups without an effort pair with source costs at both ends cannot be assessed, so the count assessed is also shown.
  • We do not count appearances on the Pareto frontier because most points lie on the frontier and it does not distinguish models.
  • When few comparison groups can be compared (fewer than 3 at most per model), we note the small count.
• measured/reported by: The party that measured or reported the value. For vendors announcing their own model results, we list the publisher, which does not necessarily mean the publisher measured the value itself. Even official announcements may include values measured by the publisher for other companies' models or values reported by other companies. We show the party if stated in the source and otherwise show “not stated”. We do not infer it from source type or provider.

Company and product names

Company and product names are trademarks or registered trademarks of their respective owners. This site is not affiliated with or endorsed by those companies. Names are shown as text only; we do not use company logos or symbols resembling them. Chart colors do not match company brand colors.

Operator

Operated by MarginalGains, Inc.. This site does not load external scripts, fonts, or trackers.