akiiwa-games/AgentCompass / skills/agent-compass
agent-compass
FlaggedMulti-agent task router and subscription quota scheduler. Choose owner, model and effort, outbound authorization, advisor use, and continuation for delegated work. Reads local usage and effective account policy, profiles the task with the configured System-One scorer, then allocates a compatible worker. Use for independent worker or advisor work; ordinary local execution needs no routing.
calibration/measurements.py
View on GitHub"""Public measurements per model, field and source: `references/public-measurements.json`.
`python -X utf8 measurements.py [--data DIR] [--out PATH]`
Every value carries its provenance (source, as_of, fetched_at, url, method). A source that was not fetched, or that
does not list a model, adds nothing: there are no defaults here. The file is the input of `five_axis.py`; the router
does not read it.
Fields
- `tier`: the model's overall level at High relative to the anchor (`tiers.py`), never a score. One entry per source:
arena.ai Overall at High (`tier.arena_overall`); and, only for a model arena Overall does not list at High, its High
reading on `aa.index` aligned onto that continuous `arena_overall` scale (`tier.aa_index`), with the two neighbours
and the overlap n. `five_axis.py` uses the tier for one thing: estimating a capability facet the model lacks through a
validated `facet~tier` fit.
- `usd_per_hour`, one entry per (source, effort), every entry with its `kind` (`community` / `derived`):
- `arena.agent_sessions` (community): arena.ai Agent / Overall real sessions, Cost/Task P50 / Time/Task P50 x 60.
- `derived.community_mix` (derived, round 4 option B): AA's measured output speed per hour priced at the community
real-traffic token mix (wokey's fixed split, about 92% cache reads; community.py) and AA's list prices, so usd per
worker-hour is on the scale the community capacities are valued on. Cites AA and the wokey mix.
- `price`: input / output / cache-read USD per million tokens (OpenRouter, AA), for cross-checks only.
- The four categories (user, 2026-11-11, Revision 2 of the taxonomy report), one entry per (facet, source); a facet is one
ability, its boards are parallel measurements of it (CAPABILITY_SOURCES):
- `creativity` creative_writing: LMArena Text, Creative Writing (style control).
- `coding` execution: arena.ai Agent / Code (Net Improvement), AA Terminal-Bench 4.0 and AA FrontierCode 2.0;
faithfulness: the arena.ai Agent Alignment Index (acting only as asked, reporting faithfully).
- `aesthetics` design: LMArena WebDev, and Design Arena Website, 3D Design and SVG (blind human votes; each correlates
0.81-0.89 with WebDev over the fleet models both list, as parallel as the execution boards are with each other).
- `tool_operation` responsiveness: AA time to first answer token per effort row, read as -ln(seconds) (a ratio scale:
halving the time is the same step anywhere); image_recognition: AA MMMU-Pro and LMArena Vision.
Image input: a model a vision board lists takes image input (`image_input`); a cited vendor statement that it takes
none (`vendor.text_only`), or OpenRouter's input modalities, otherwise. A public board that lists the model wins.
One conversion for every board, a panel standard score relative to the anchor model (ANCHOR, accounts.json
`calibration_panel.anchor`):
z = (x - x_anchor) / sd_panel(x), x the board's raw value (Elo, rating, success rate, index; -ln seconds for the time to
first answer token). The panel is declared (owner, 2026-11-18): accounts.json `calibration_panel.members`, a dated
maintainer list only the owner changes; a new row is scored against it and never joins it by being listed. The core boards
are the scoring boards that list every sentinel row (`calibration_panel.models`), and every declared member must be
listed on every core board: a member one does not list is an error naming both, never a smaller panel. sd_panel is the
sample standard deviation of the board over the panel. A board that lists the
anchor and at least MIN_PANEL panel members is a source; any other is recorded as missing with the reason. Nothing is
capped or bridged: a new model at the top of a board moves no other model, and the anchor is 0 on every board.
Every board's listings are read at every effort of the one order (user_scores.EFFORTS, `none` and `minimal` included;
round 6 item 5): each entry carries the score at High (else the effort-less listing, else the effort nearest High) and,
when the board lists the model at two efforts or more, `z_by_effort`, every listed effort on the same scale. A board's
`edge_pairs` (Builder.edge_pairs) hold, for every listing with both efforts of an edge's pair, catalog row or not, the
difference in panel z: the sample of the fleet's edge steps (five_axis.edge_step_table). An entry without
`z_by_effort` (`listed_as`, its effort in `listed_effort` when the listing names one) is moved by five_axis.py to other
efforts on the same facet's AA board, which also links parallel boards.
Checks (`checks`, round 059 section 5): structural failures leave the (model, source) out, recorded as missing; a
statistical failure is a yellow note and the value is used.
- structural: a listing whose variant tag names something other than an effort (another harness); one board listing
matched by two fleet rows that are neither the same model (`same_model`) nor a series; a raw value outside the
board's range (a fraction outside 0..1, a non-positive time); two boards with identical raw vectors (the second is
a duplicate and not used).
- statistical: on an Artificial Analysis board, High outside the range of the model's adjacent listed efforts.
`capacity`: community real-usage measurements of subscription window capacity (`community.py`, round 4): every
observation with its source, url, data date and fetch time; stated ratios between windows (`rules`); `errors` (a page
not fetched or no longer carrying its sentence gives no value); `not_found`.
usd_per_hour comes only from public sources (review F2, 2026-11-09): the arena.ai Agent sessions, and AA's measured output
speed at the community token mix. The shipped accounts values (a local 14-day calibration sample) and the community burn
figures (measured locally, no external citation) are not sources.
"""
import argparse
import json
import math
import re
import statistics
import sys
from datetime import datetime, timezone
from pathlib import Path
import arena_capture
import community
import series
import sources
import tiers
import user_scores
# Every capability score is relative to this model (0 on every board): accounts.json `calibration_panel.anchor`.
ANCHOR = tiers.panel(sources.accounts())[0]
MIN_PANEL = 8 # panel members a board must list to be a source (as five_axis.MIN_FIT_POINTS)
OUT = sources.REPO / 'references' / 'public-measurements.json'
SCHEMA = 'agent-compass.public-measurements/1'
# Every scoring board: source -> (category, facet). Parallel boards of one facet are linked and averaged (five_axis.py).
CAPABILITY_SOURCES = {
'lmarena.creative_writing': ('creativity', 'creative_writing'),
'arena.agent_code': ('coding', 'execution'),
'aa.terminalbench_4': ('coding', 'execution'),
'aa.frontiercode_2': ('coding', 'execution'),
'arena.agent_alignment': ('coding', 'faithfulness'),
'lmarena.webdev': ('aesthetics', 'design'),
'designarena.website': ('aesthetics', 'design'),
'designarena.3d': ('aesthetics', 'design'),
'designarena.svg': ('aesthetics', 'design'),
'aa.ttfa': ('tool_operation', 'responsiveness'),
'aa.mmmu_pro': ('tool_operation', 'image_recognition'),
'lmarena.vision': ('tool_operation', 'image_recognition'),
}
AXES = {'creativity': ('creative_writing',), 'coding': ('execution', 'faithfulness'), 'aesthetics': ('design',),
'tool_operation': ('responsiveness', 'image_recognition')}
# arena.ai boards transcribed from the user's browser (arena_capture.py): source -> (capture, category, value, method).
ARENA_CAPTURES = {
'arena.agent_code': ('agent', 'Code', lambda r: (r.get('net_improvement') or [None])[0],
'arena.ai Agent / Code (Best Coding Agents), Net Improvement %, High row (else nearest effort)'),
'arena.agent_alignment': ('alignment', None, lambda r: r['alignment_index'][0],
'arena.ai Agent Alignment Index (Best Aligned Agents), 0-100'),
}
# LMArena boards: source -> (dataset board, category).
LMARENA = {'lmarena.creative_writing': ('text_style_control', 'creative_writing'), 'lmarena.webdev': ('webdev', 'overall'),
'lmarena.vision': ('vision', 'overall')}
DESIGN_ARENA = {'designarena.website': 'designarena-website.html', 'designarena.3d': 'designarena-3d.html',
'designarena.svg': 'designarena-svg.html'}
# Artificial Analysis boards read per effort: source -> sub-benchmark id of the record's `scores` (fractions).
AA_SCORE = {'aa.terminalbench_4': 'terminalbench-4-0', 'aa.frontiercode_2': 'frontiercode-2-0', 'aa.mmmu_pro': 'mmmu-pro'}
AA_TTFA = 'aa.ttfa' # the record's `timeToFirstAnswerToken` (seconds), read as -ln(seconds)
# Value ranges of the raw readings (structural check): fractions 0..1, the Net Improvement -100..100 %, the index 0..100.
RAW_RANGE = {**dict.fromkeys(AA_SCORE, (0.0, 1.0)), 'arena.agent_code': (-100.0, 100.0),
'arena.agent_alignment': (0.0, 100.0)}
# The kind of each source (decision 2, 2026-11-09): vendor = the vendor's own statement or price list, benchmark = a
# third-party measurement, community = a community or packaged estimate with a stated basis.
SOURCE_KIND = {**dict.fromkeys(CAPABILITY_SOURCES, 'benchmark'), 'derived.community_mix': 'derived',
'tier.arena_overall': 'benchmark', 'tier.aa_index': 'benchmark', 'vendor.text_only': 'vendor',
'vendor.image_input': 'vendor', 'maintainer.image_input': 'maintainer', 'maintainer.text_only': 'maintainer',
'aa.cost_time': 'benchmark', 'arena.agent_sessions': 'community',
'aa.price': 'vendor', 'openrouter.price': 'vendor', 'vendor.price': 'vendor', 'openrouter.modality': 'vendor',
'vendor.fast_mode': 'vendor', 'aa.index': 'benchmark'}
IMAGE_BENCHMARKS = ('lmarena.vision', 'aa.mmmu_pro')
ARENA_FILE = {**{src: f'{board}.parquet' for src, (board, _) in LMARENA.items()},
**{src: arena_capture.FILES[spec[0]] for src, spec in ARENA_CAPTURES.items()}, **DESIGN_ARENA,
**dict.fromkeys([*AA_SCORE, AA_TTFA], 'aa-models.html')}
EFFORT_RANK = {e: i for i, e in enumerate(('low', 'medium', 'high', 'xhigh', 'max'))}
EFFORT_TAG = re.compile(
r'^(max|xhigh|high|medium|low|minimal|non-reasoning|none|default|[a-z0-9. ]+ fallback|'
r'thinking|non-thinking|reasoning|instant|thinking\s+\d+k|\d+k|\d{2,4}|\d{6}|\d{8}|'
r'\d{2,4}-\d{2}(?:-\d{2})?|(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\s+\d{4})$'
) # an effort, thinking mode, or release date
def key(name):
return ''.join(ch for ch in str(name).lower() if ch.isalnum())
def candidates(model):
"""Normalised spellings a source may list `model` under: the id, without an effort suffix, without a free-listing
suffix (the same model served free), without a preview or `-1.0` suffix. Exact matches only, never fuzzy."""
out = [model]
for suffix in ('-high', '-medium', '-low', '-free', '-preview', '-1.0'):
out += [m[: -len(suffix)] for m in list(out) if m.endswith(suffix)]
return [key(m) for m in out]
IDENTITY_KINDS = ('vendor', 'community', 'benchmark')
def listed_as_identity(models):
"""Refuse a `listed_as` name a row is not listed under by its own spelling (`candidates`: its id, without an effort,
free-listing, preview or `-1.0` suffix) unless `listed_as_source[board]` cites the public statement that the listed
name is that model (an identity statement as for an alias: kind vendor, community or benchmark, https url, as_of,
statement). A dated maintainer declaration is no statement of identity (round 5 review A)."""
bad = []
for m, row in sorted(models.items()):
for board, name in sorted((row.get('listed_as') or {}).items()):
if key(sources.base_name(name)) in candidates(m):
continue
cite = (row.get('listed_as_source') or {}).get(board) or {}
if not (cite.get('kind') in IDENTITY_KINDS and str(cite.get('url') or '').startswith('https://')
and re.match(r'^\d{4}-\d{2}-\d{2}$', str(cite.get('as_of') or ''))
and str(cite.get('statement') or '').strip()):
bad.append(f'{m} ({board}: {name!r})')
extra = sorted(set(row.get('listed_as_source') or {}) - set(row.get('listed_as') or {}))
if extra:
bad.append(f'{m} (listed_as_source for {", ".join(extra)} without a listed_as)')
if bad:
raise ValueError('accounts.json listed_as under another name needs listed_as_source with the public statement '
'that the listed name is that model: ' + '; '.join(bad))
class Matcher:
def __init__(self, names, strip=None):
# A dated listing is found only under its own name with the date (sources.base_name keeps a date tag), never
# under the name without it: a row never reads a snapshot it does not name (round 5 review A).
self.by_key = {}
for n in names:
base = sources.base_name(n)
s = strip(base) if strip else base
for raw in (s, base, strip(n) if strip else n, n):
k = key(raw)
if k:
self.by_key.setdefault(k, n)
def find(self, model, explicit=None):
if explicit:
return self.by_key.get(key(explicit)) # the board's own listing of that name
return next((self.by_key[k] for k in candidates(model) if k in self.by_key), None)
def aa_variants(records, readings):
"""{base name: {effort: record}} for AA listings whose variant tag is an effort, a reasoning label (read as
`readings`, accounts.json effort_readings, declares: sources.aa_effort) or a date only; of several listings read at one
effort, the one `sources.listing_rank` prefers (the later on a tie)."""
out, rank = {}, {}
for rec in records:
tag = sources.variant(rec['name'])
parts = [p.strip().lower() for p in tag.split(',') if p.strip()]
# A part ending in "fallback" is AA's note of the model its harness falls back to on a refusal (a serving
# configuration of the release it measured), not an effort.
allowed = all(p in sources.EFFORT_WORDS or p in readings['labels'] or p.endswith(' fallback')
or re.fullmatch(r'\d{4}', p) for p in parts)
r = sources.listing_rank(rec['name'], readings) if allowed else None
if r is None:
continue
at = (sources.base_name(rec['name']), sources.aa_effort(rec['name'], readings))
if at in rank and rank[at] < r:
continue
rank[at] = r
out.setdefault(at[0], {})[at[1]] = rec
return out
def aa_release_plan(target, models, readings):
"""({model: (release slug, base name)}, {model: why no release can be chosen}): the Artificial Analysis release each
catalogue model is read from. The leaderboard page embeds records only for releases AA does not mark deprecated, and
names a deprecated release without records. A catalogue model that no leaderboard record matches, but that a
deprecated release is listed under, is read from that release's own page: the same matcher and `listed_as` as every
board, over the base names of the release's listings (`modelNames`, else its name; an effort tag is not part of a
base name, a date tag is: sources.base_name, another snapshot). When several deprecated releases
list a base name, the most recently released one is chosen: the choice depends on the base name alone, so every
model of a base name reads the same release. Releases without a `releaseDate`, or sharing the newest one, leave the
choice ambiguous: the model is returned in the second dict and has no release."""
on_page = Matcher(aa_variants(sources.aa_records(target), readings))
listed = {} # base name -> {slug: releaseDate}
for release in sources.aa_releases(target):
if release.get('deprecated') is True:
for listing in release.get('modelNames') or [release['name']]:
listed.setdefault(sources.base_name(listing), {})[release['slug']] = release.get('releaseDate')
off_page = Matcher(listed)
exact = {key(b) for b in listed}
models = {m: row for m, row in models.items() if isinstance(row, dict)} # accounts.json `models` holds notes too
plan, ambiguous = {}, {}
for model in models:
exp = tiers.listed_as(models, model, 'aa')
found = on_page.find(model, exp) or (exp and on_page.find(model))
# A declared listing a deprecated release names exactly is that release, not a dated listing of another release
# the date fallback of the leaderboard's names reaches (DeepSeek V4.1 Flash beside DeepSeek V4.1 Flash 1017).
if found and not (exp and key(exp) in exact and key(sources.base_name(found)) != key(exp)):
continue
base = off_page.find(model, exp) or (off_page.find(model) if exp else None)
if not base:
continue
slugs = listed[base]
dates = {s: d for s, d in slugs.items() if isinstance(d, str) and d}
newest = [s for s in dates if dates[s] == max(dates.values(), default=None)]
if len(slugs) > 1 and (len(dates) < len(slugs) or len(newest) > 1):
ambiguous[model] = (f"the releases {sorted(slugs)} are all listed under {base!r} and the newest cannot be "
'told from their releaseDate (' + ', '.join(f'{s}: {slugs[s]!r}' for s in sorted(slugs)) + ')')
else:
plan[model] = (next(iter(slugs)) if len(slugs) == 1 else newest[0], base)
return plan, ambiguous
def aa_release_files(plan):
"""{file name: url} of the release pages `aa_release_plan` chose, in slug order (the written files do not depend on
the order a set iterates in)."""
return {sources.AA_RELEASE_FILE.format(slug): sources.AA_RELEASE_URL.format(slug)
for slug in sorted({slug for slug, _ in plan.values()})}
def aa_records_read(target, models, readings):
"""(records, plan, ambiguous, errors) of the Artificial Analysis records the pipeline reads: the leaderboard page's,
then those of the release pages `aa_release_plan` chose and the manifest lists as fetched, in every mode (a page
the plan does not choose is never read, whatever the data directory holds). Of a page only the records of the base
names the plan chose that release for are kept, so a base name has the records of one release. `errors` is
`sources.aa_release_records`'."""
plan, ambiguous = aa_release_plan(target, models, readings)
pages, errors = sources.aa_release_records(target, aa_release_files(plan))
chosen = {}
for slug, base in plan.values():
chosen.setdefault(sources.AA_RELEASE_FILE.format(slug), set()).add(base)
kept = [r for r in pages if sources.base_name(r['name']) in chosen[r['page']]]
return [*sources.aa_records(target), *kept], plan, ambiguous, errors
DATED = re.compile(r'^\d{4}-\d{2}-\d{2}(T\S+)?$')
def inputs_date(doc):
"""The date of a generated document: the latest date of a reading it holds (`fetched_at`, `as_of`; round 7, no clock
in a generated file), so the document is a function of its inputs; a build from inputs with no date stops."""
found = []
def walk(o):
if isinstance(o, dict):
for k, v in o.items():
if k in ('fetched_at', 'as_of') and isinstance(v, str) and DATED.match(v):
when = datetime.fromisoformat(v)
found.append((when if when.tzinfo else when.replace(tzinfo=timezone.utc)).astimezone(timezone.utc))
else:
walk(v)
elif isinstance(o, list):
for v in o:
walk(v)
walk(doc)
if not found:
raise ValueError('the measurements hold no dated reading (fetched_at, as_of): nothing dates the generated file')
return max(found).isoformat(timespec='seconds')
class Builder:
def __init__(self, target, acc):
self.target, self.acc = target, acc
self.models = {k: v for k, v in acc['models'].items() if isinstance(v, dict)}
listed_as_identity(self.models)
anchor, self.panel_ids, self.anchor_name = tiers.panel(acc)
self.members = list(acc['calibration_panel']['members'])
if anchor != ANCHOR:
raise ValueError(f'calibration_panel anchor {anchor!r} differs from the shipped accounts.json anchor {ANCHOR!r} '
'that five_axis.py scales on')
self.manifest = sources.manifest(target)
self.source_rows, self.out, self.names = {}, {m: {} for m in self.models}, {m: {} for m in self.models}
self.missing = {}
self.checks = []
self.readings = sources.effort_readings(acc)
self.aa_records, self.release_plan, self.release_ambiguous, self.release_errors = aa_records_read(
target, self.models, self.readings)
self.release_files = aa_release_files(self.release_plan)
def ok(self, filename):
return (self.manifest.get(filename) or {}).get('status') == 'ok' and (self.target / filename).exists()
def aa_release_rows(self):
"""{file name: manifest row} of the release pages the plan chose that were fetched."""
return {n: self.manifest[n] for n in self.release_files if self.ok(n)}
def register(self, source, filename, as_of, method, count, note=None):
row = self.manifest.get(filename) or {}
self.source_rows[source] = {'file': filename, 'url': row.get('url'), 'fetched_at': row.get('fetched_at'),
'sha256': row.get('sha256'), 'as_of': as_of or (row.get('fetched_at') or '')[:10],
'as_of_kind': 'data date' if as_of else 'fetch date (the source publishes no data date)',
'method': method, 'models_listed': count, 'kind': SOURCE_KIND.get(source, 'benchmark'),
**({'note': note} if note else {})}
pages = self.aa_release_rows() if filename == 'aa-models.html' else {}
if pages: # the release pages are read with the leaderboard: the source is as old as the oldest of its files
self.source_rows[source]['release_pages'] = {n: {k: r.get(k) for k in ('url', 'fetched_at', 'sha256')}
for n, r in pages.items()}
self.source_rows[source]['as_of'] = min(self.source_rows[source]['as_of'],
*(r['fetched_at'][:10] for r in pages.values()))
def cite(self, rec):
"""Where one AA record was read, for the entry made from it: the url and fetch time of its release page. A
leaderboard record carries no `page`; its source row says."""
page = rec.get('page')
if page is None:
return {}
row = self.manifest[page]
return {'url': row['url'], 'fetched_at': row['fetched_at'], 'as_of': row['fetched_at'][:10]}
def add(self, model, field, source, value, method, **extra):
meta = self.source_rows[source]
entry = {'source': source, 'kind': meta.get('kind', SOURCE_KIND.get(source, 'benchmark')), 'value': value,
'as_of': meta['as_of'], 'fetched_at': meta['fetched_at'], 'url': meta['url'], 'method': method, **extra}
self.out[model].setdefault(field, []).append(entry)
def miss(self, source, filename, reason, url=None):
self.missing[source] = {'file': filename, 'reason': reason,
'url': url or (self.manifest.get(filename) or {}).get('url') or sources.PAGES.get(filename)}
def release_gaps(self):
"""Surface each release the leaderboard page and the catalogue ask for that cannot be read: the choice between
releases is ambiguous, the page was never fetched, a record of it has a shape the reader does not know, or it
gives a model no reading of its release (no record the matcher finds, or only records without a figure), or the
matcher resolves the model to another base name than the one the plan chose it for. A page whose fetch failed
is a missing manifest row, surfaced with its error by `build`."""
for model, why in self.release_ambiguous.items():
self.miss(f'aa.release/{model}', 'aa-models.html', why)
variants = aa_variants(self.aa_records, self.readings)
found, empty, bare, wrong = Matcher(variants), {}, {}, {}
for model, (slug, planned) in self.release_plan.items():
name, url = sources.AA_RELEASE_FILE.format(slug), sources.AA_RELEASE_URL.format(slug)
row = self.manifest.get(name)
if row is None or (row.get('status') == 'ok' and not (self.target / name).exists()):
self.miss(name, name, 'not fetched; run `agentcompass data refresh`', url)
elif name in self.release_errors:
self.miss(name, name, f'unreadable: {self.release_errors[name]}', url)
elif self.ok(name):
exp = tiers.listed_as(self.models, model, 'aa')
base = found.find(model, exp) or (found.find(model) if exp else None)
if base and base != planned:
wrong.setdefault((name, url), []).append(f'{model} resolves to {base!r}, not {planned!r}, the base name '
'the plan chose it for on this page')
elif not base:
empty.setdefault((name, url), []).append(model)
elif not any(sources.aa_has_figure(r) for r in variants[base].values()):
bare.setdefault((name, url), []).append(model)
for key_, ms in sorted(bare.items()):
if key_ not in empty and key_ not in wrong: # the board lists the release and publishes no figure for it
self.miss(key_[0], key_[0], f'not used: the page lists the release of {", ".join(ms)} without a figure; '
'its scores come from the declared estimation methods', key_[1])
for key_ in sorted({*empty, *wrong}, key=lambda page: page[0]): # a set's order differs between runs
reasons = [*wrong.get(key_, ())]
if key_ in empty:
reasons.append(f'the page carries no record of its release for {", ".join(empty[key_])} (no record of the '
'planned base name that the matcher finds for it)')
self.miss(key_[0], key_[0], '; '.join(reasons), key_[1])
@staticmethod
def select_variants(items, readings):
"""Group `(listed_name, value, extra)` rows by base model name; select the model's own High row (never xHigh/Max
when High is listed; else an effort-less listing; else the effort nearest High) and keep multi-effort maps over
every effort of the one order (none and minimal too)."""
grouped = {}
for listed_name, val, ext in items:
if val is None or not listed_name:
continue
grouped.setdefault(sources.base_name(listed_name), []).append((str(listed_name), float(val), dict(ext or {})))
board, listed, extra, by_effort = {}, {}, {}, {}
for base, rows in grouped.items():
valid, invalid = [], []
for ln, val, ext in rows:
bad = [p for p in (q.strip().lower() for q in sources.variant(ln).split(',')) if p and not EFFORT_TAG.match(p)]
(invalid if bad else valid).append((ln, val, ext))
if not valid:
ln, val, ext = max(invalid, key=lambda r: r[1])
board[base], listed[base], extra[base] = val, ln, ext
continue
vals = {}
for ln, val, ext in valid:
eff = sources.aa_effort(ln, readings) or 'default'
tag = sources.variant(ln)
date_m = re.search(r'(\d{8}|\d{6}|\d{4}-\d{2}-\d{2}|\d{4})', tag)
date_key = date_m.group(1).replace('-', '') if date_m else ''
thinking = 1 if re.search(r'\b(thinking|reasoning)\b', tag, re.I) and not re.search(r'\bnon-thinking\b', tag, re.I) else 0
rank_key = (thinking, len(date_key), date_key, val)
if eff not in vals or rank_key > vals[eff][3]:
vals[eff] = (val, ln, ext, rank_key)
routable = [e for e in vals if e in EFFORT_RANK]
pick = ('high' if 'high' in vals else
'default' if 'default' in vals else
min(routable, key=lambda e: (abs(EFFORT_RANK[e] - EFFORT_RANK['high']), -EFFORT_RANK[e])) if routable else
next(iter(vals)))
board[base], listed[base], extra[base] = vals[pick][0], vals[pick][1], vals[pick][2]
# every effort of the one effort order (user_scores.EFFORTS: none and minimal too, round 6 item 5)
eff_map = {e: round(vals[e][0], 6) for e in user_scores.EFFORTS if e in vals}
if 'default' in vals and 'high' not in eff_map and eff_map:
eff_map['high'] = round(vals['default'][0], 6)
if len(eff_map) >= 2:
by_effort[base] = eff_map
return board, listed, extra, by_effort
@staticmethod
def edge_pairs(by_effort, anchor_raw, sd, reads):
"""{edge effort: [{'listed', 'efforts', 'difference'}]} of one board (round 6 item 5): for every listing the board
holds at both efforts of an edge's pair (user_scores.EDGES: `minimal` low - medium, `ultra` max - xhigh, `none`
none - the lowest listed reasoning effort), catalog row or not, the difference in the board's panel z. A listing
without an effort label reads at the effort `reads` names (accounts.json effort_readings) unless that is listed.
The raw readings, before any bound: a step is a property of the board's readings."""
out = {}
for name, raw in sorted(by_effort.items()):
row = {e: v for e, v in raw.items() if e in user_scores.EFFORTS}
for label, effort in reads.items():
if label in raw and effort not in row:
row[effort] = raw[label]
for edge, (_, (a, b)) in user_scores.EDGES.items():
if b == user_scores.LOWEST_REASONING:
b = next((e for e in user_scores.EFFORTS[1:] if e in row), None)
if a in row and b in row:
out.setdefault(edge, []).append({'listed': name, 'efforts': [a, b],
'difference': round((row[a] - row[b]) / sd, 4)})
return out
# --- capability boards ----------------------------------------------------------------------
def boards(self):
"""{source: {'board': {name: raw}, 'as_of', 'method', 'strip', 'listed': {name: listed variant}, 'extra': {name:
{...}}, and 'by_effort': {name: {effort: raw}}}} of every scoring board fetched."""
t, out = self.target, {}
for src, (board, cat) in LMARENA.items():
if not self.ok(ARENA_FILE[src]):
self.miss(src, ARENA_FILE[src], 'not fetched'); continue
rows = sources.arena_rows(t, board, cat)
brd, lst, ext, by_eff = self.select_variants((
(r[3], r[0], {'ci': [round(r[1], 1), round(r[2], 1)]}) for r in rows), self.readings)
out[src] = {'board': brd, 'as_of': rows[0][4] if rows else None,
'method': f'LMArena {board}/{cat} rating, High row (else effort-less or nearest effort)',
'strip': None, 'listed': lst, 'extra': ext,
**({'by_effort': by_eff} if by_eff else {})}
for src, (kind, category, value, method) in ARENA_CAPTURES.items():
doc = arena_capture.read(t, kind) if self.ok(arena_capture.FILES[kind]) else None
page = ((doc or {}).get('categories') or {}).get(category) if category else doc
if not page:
self.miss(src, arena_capture.FILES[kind], f'not captured: transcribe {arena_capture.URLS[kind]}'
+ (f' ({category})' if category else '') + ' in the browser and run arena_capture.py import')
continue
brd, lst, ext, by_eff = self.select_variants((
(r['name'], value(r), {}) for r in page['rows'] if r.get('name')), self.readings)
out[src] = {'board': brd, 'as_of': page['as_of'],
'method': f'{method}; transcribed {(doc or {}).get("captured_at", "")[:10]} by {(doc or {}).get("transcribed_by")}'
+ (f'; {page["sign_rule"]}' if page.get('sign_rule') else ''),
'strip': None, 'listed': lst, 'extra': ext,
**({'by_effort': by_eff} if by_eff else {})}
for src, filename in DESIGN_ARENA.items():
if not self.ok(filename):
self.miss(src, filename, 'not fetched'); continue
rows = sources.designarena(t, filename)
if not rows:
self.miss(src, filename, 'the page carries no main board rows'); continue
brd, lst, ext, by_eff = self.select_variants((
(r['name'], r['rating'], {'battles': r['battles'], 'designarena_id': mid}) for mid, r in rows.items()), self.readings)
out[src] = {'board': brd, 'as_of': None,
'method': (f'Design Arena {sources.DESIGN_ARENA[filename][1]} board (main public board): Bradley-Terry '
'rating of blind human pairwise votes, High row (else nearest effort); no standard error published'),
'strip': None, 'listed': lst, 'extra': ext,
**({'by_effort': by_eff} if by_eff else {})}
if self.ok('aa-models.html'):
variants = self.aa_variants()
for src in [*AA_SCORE, AA_TTFA]:
board, by_effort, listed, extra = {}, {}, {}, {}
for base, recs in variants.items():
vals = {(eff or 'default'): (v, rec) for eff, rec in recs.items()
if (v := self.aa_value(src, rec)) is not None}
if not vals:
continue
routable = [e for e in vals if e in EFFORT_RANK]
pick = ('high' if 'high' in vals else
'default' if 'default' in vals else
min(routable, key=lambda e: (abs(EFFORT_RANK[e] - EFFORT_RANK['high']), -EFFORT_RANK[e])) if routable else
next(iter(vals)))
board[base], listed[base] = vals[pick][0], vals[pick][1]['name']
extra[base] = self.cite(vals[pick][1])
by_effort[base] = {e: round(v, 6) for e, (v, _) in vals.items()}
what = ('time to first answer token, read as -ln(seconds)' if src == AA_TTFA else f'`{AA_SCORE[src]}`')
out[src] = {'board': board, 'as_of': None, 'strip': None, 'listed': listed, 'extra': extra,
'by_effort': by_effort,
'method': f'Artificial Analysis {what}, every listed effort; the score is the High row (else the only listing)'}
else:
for src in [*AA_SCORE, AA_TTFA]:
self.miss(src, 'aa-models.html', 'not fetched')
return out
def aa_value(self, src, rec):
"""One AA record's raw reading on `src`, or None. A time that is not positive is out of range (a structural check)."""
if src == AA_TTFA:
t = rec.get('timeToFirstAnswerToken')
if not isinstance(t, (int, float)) or isinstance(t, bool):
return None
if t <= 0:
self.checks.append({'source': src, 'listed_as': rec.get('name'), 'check': 'raw_out_of_range',
'level': 'structural', 'evidence': f'time to first answer token {t} s is not positive'})
return None
return -math.log(float(t))
v = (rec.get('scores') or {}).get(AA_SCORE[src])
return float(v) if isinstance(v, (int, float)) and not isinstance(v, bool) else None
def aa_variants(self):
"""{base name: {effort: record}} for AA listings whose variant tag is an effort or a date only."""
return aa_variants(self.aa_records, self.readings)
def capabilities(self):
boards, raw, names = self.boards(), {}, {}
for src, b in boards.items():
self.register(src, ARENA_FILE[src], b['as_of'], b['method'], len(b['board']))
match = Matcher(b['board'], b['strip'])
family = src.split('.')[0]
explicit_key = {'lmarena': 'arena', 'arena': 'arena', 'aa': 'aa'}.get(family)
raw[src], names[src] = {}, {}
for model in self.models:
exp = tiers.listed_as(self.models, model, explicit_key) if explicit_key else None
name = match.find(model, exp) or (match.find(model) if exp else None)
if name is not None:
raw[src][model], names[src][model] = b['board'][name], name
self.structural(boards, raw, names)
panel, core, scale = self.panel(raw)
self.panel_doc = {'models': panel, 'n': len(panel), 'core_boards': core, 'min_panel': MIN_PANEL,
'rule': 'the declared members (accounts.json calibration_panel.members), each listed on every '
f'core board (the scoring boards that list {", ".join(self.panel_ids)}); sd_panel is the '
'sample standard deviation over them'}
for src, sc in scale.items():
axis, facet = CAPABILITY_SOURCES[src]
b = boards[src]
self.source_rows[src]['scale'] = {'anchor': ANCHOR, 'anchor_raw': round(sc['anchor_raw'], 6),
'sd_panel': round(sc['sd'], 6), 'n_panel': sc['n'], 'core': src in core}
if b.get('by_effort'):
pairs = self.edge_pairs(b['by_effort'], sc['anchor_raw'], sc['sd'], self.readings['reads'])
if pairs:
self.source_rows[src]['edge_pairs'] = pairs
how = (f"z = (x - x_anchor) / sd_panel; x_anchor {sc['anchor_raw']:.6g} ({ANCHOR}), sd_panel {sc['sd']:.6g} "
f"over {sc['n']} panel models")
for model, x in raw[src].items():
name = names[src][model]
listed = b['listed'].get(name, name)
extra = dict(b['extra'].get(name) or {})
if 'by_effort' in b and name in b['by_effort']:
extra['z_by_effort'] = {e: round((v - sc['anchor_raw']) / sc['sd'], 4) for e, v in b['by_effort'][name].items()}
extra['raw_by_effort'] = b['by_effort'][name]
if not src.startswith('aa.'):
extra['listed_effort'] = sources.aa_effort(listed, self.readings) or None
self.names[model][src] = name
self.add(model, axis, src, round((x - sc['anchor_raw']) / sc['sd'], 4), f"{b['method']}; {how}",
facet=facet, listed_as=listed, raw=round(x, 6), **extra)
if src in IMAGE_BENCHMARKS:
self.add(model, 'image_input', src, True, f"{b['method']}: measured on image input", listed_as=listed,
**{k: extra[k] for k in ('url', 'fetched_at', 'as_of') if k in extra})
self.conflicts = []
iso_date = re.compile(r'^\d{4}-\d{2}-\d{2}$')
for model, info in self.models.items():
img_decl = info.get('image_input_source')
if isinstance(img_decl, dict) and (
img_decl.get('url') or (img_decl.get('kind') == 'maintainer' and iso_date.match(str(img_decl.get('as_of') or '')))
):
is_maint = img_decl.get('kind') == 'maintainer'
val = bool(img_decl.get('value', True))
bench_img = [e['source'] for e in self.out[model].get('image_input', []) if e.get('kind') == 'benchmark']
if not val and bench_img:
self.conflicts.append({'model': model, 'field': 'image_input', 'reason': 'sources_disagree',
'detail': f'image_input_source says False, but {", ".join(bench_img)} list the model',
'resolution': 'the public leaderboard wins (user, 2026-11-09)'})
else:
self.out[model].setdefault('image_input', []).append(
{'source': 'maintainer.image_input' if is_maint else 'vendor.image_input',
'kind': 'maintainer' if is_maint else 'vendor', 'value': val,
'as_of': img_decl.get('as_of'), 'fetched_at': None, 'url': img_decl.get('url'),
'statement': img_decl.get('statement'),
'method': 'maintainer declaration of image-input modality' if is_maint
else 'vendor statement of image-input modality'})
stated = info.get('text_only_source')
if not (isinstance(stated, dict) and (
stated.get('url') or (stated.get('kind') == 'maintainer' and iso_date.match(str(stated.get('as_of') or '')))
)):
continue # the local visual / text_only flags are routing data, not a source (review F1)
is_maint = stated.get('kind') == 'maintainer'
bench_img = [e['source'] for e in self.out[model].get('image_input', []) if e.get('kind') == 'benchmark']
if bench_img: # a board measured image input on a model a statement calls text-only
self.conflicts.append({'model': model, 'field': 'image_input', 'reason': 'sources_disagree',
'detail': f'text_only_source says text-only, but {", ".join(bench_img)} list the model',
'resolution': 'the public leaderboard wins (user, 2026-11-09)'})
continue
self.out[model].setdefault('image_input', []).append(
{'source': 'maintainer.text_only' if is_maint else 'vendor.text_only',
'kind': 'maintainer' if is_maint else 'vendor', 'value': False,
'as_of': stated.get('as_of'), 'fetched_at': None, 'url': stated.get('url'),
'statement': stated.get('statement'),
'method': 'maintainer declaration that the model takes no image input' if is_maint
else 'vendor statement that the model takes no image input'})
if self.ok('openrouter-models.json'):
mods_table = sources.openrouter_modalities(self.target)
self.register('openrouter.modality', 'openrouter-models.json', None,
'OpenRouter /api/v1/models architecture.input_modalities', len(mods_table))
match = Matcher(mods_table, lambda n: n.split('/')[-1])
for model in self.models:
name = match.find(model)
if not name:
continue
mods = mods_table[name]
has_img = 'image' in mods
bench_img = [e['source'] for e in self.out[model].get('image_input', []) if e.get('kind') == 'benchmark']
if not has_img and bench_img:
self.conflicts.append({'model': model, 'field': 'image_input', 'reason': 'sources_disagree',
'detail': f'OpenRouter lists input_modalities={mods}, but {", ".join(bench_img)} measured image input',
'resolution': 'the public leaderboard wins (user, 2026-11-09)'})
self.add(model, 'image_input', 'openrouter.modality', has_img,
f'OpenRouter architecture.input_modalities ({", ".join(mods)})',
listed_as=name, input_modalities=mods)
def structural(self, boards, raw, names):
"""The structural checks of the module docstring: a failing (model, source) leaves `raw`, recorded in `checks`."""
def drop(model, src, check, evidence):
self.checks.append({'model': model, 'source': src, 'check': check, 'level': 'structural', 'evidence': evidence})
raw[src].pop(model, None)
names[src].pop(model, None)
groups, same = series.groups(self.acc, 'ability'), series.same_model(self.acc)
for src, b in boards.items():
for model, name in list(names[src].items()):
listed = b['listed'].get(name, name)
bad = [p for p in (q.strip().lower() for q in sources.variant(listed).split(',')) if p and not EFFORT_TAG.match(p)]
if bad:
drop(model, src, 'variant_not_effort', f'listed as {listed!r}: the tag names {", ".join(bad)}, not an effort')
continue
lo_hi = RAW_RANGE.get(src)
if lo_hi and not lo_hi[0] <= raw[src][model] <= lo_hi[1]:
drop(model, src, 'raw_out_of_range', f'{raw[src][model]} outside {lo_hi[0]:g}..{lo_hi[1]:g}')
by_name = {}
for model, name in names[src].items():
by_name.setdefault(name, []).append(model)
for name, models in by_name.items():
related = all(m2 in groups.get(m1, ()) or (same.get(m1) or {}).get('model') == m2
or (same.get(m2) or {}).get('model') == m1 for m1 in models for m2 in models if m1 != m2)
if not related:
for model in models:
drop(model, src, 'listing_matched_twice', f'{name!r} matched {", ".join(sorted(models))}, neither the '
'same model nor one series')
if 'by_effort' in b: # statistical: High between the adjacent listed efforts
for model, name in names[src].items():
row = b['by_effort'].get(name) or {}
if 'high' in row and 'medium' in row and 'xhigh' in row and \
not min(row['medium'], row['xhigh']) - 1e-9 <= row['high'] <= max(row['medium'], row['xhigh']) + 1e-9:
self.checks.append({'model': model, 'source': src, 'check': 'high_outside_adjacent_efforts',
'level': 'statistical', 'evidence': f"High {row['high']:.4g}, Medium {row['medium']:.4g}, "
f"xHigh {row['xhigh']:.4g}"})
seen = {}
for src in sorted(raw):
vector = tuple(sorted((m, round(v, 9)) for m, v in raw[src].items()))
if len(vector) >= 3 and vector in seen:
self.checks.append({'source': src, 'check': 'duplicate_board', 'level': 'structural',
'evidence': f'raw values identical to {seen[vector]} on {len(vector)} models'})
self.miss(src, ARENA_FILE[src], f'not used: identical to {seen[vector]}')
raw[src], names[src] = {}, {}
else:
seen.setdefault(vector, src)
def panel(self, raw):
"""(panel models, core boards, {source: {'anchor_raw', 'sd', 'n'}}) of the module docstring."""
core = [src for src, board in raw.items() if board and all(a in board for a in self.panel_ids)]
for src in core:
lost = [m for m in self.members if m not in raw[src]]
if lost:
raise ValueError(f'calibration_panel member(s) {", ".join(lost)} not listed on the core board {src}: the '
'declared panel is never shrunk silently; the owner changes calibration_panel.members')
panel = sorted(self.members)
scale = {}
for src, board in raw.items():
members = [m for m in panel if m in board]
if ANCHOR not in board:
why = f'{ANCHOR} is not listed: the board has no point the scale stands on'
elif len(members) < MIN_PANEL:
why = f'{len(members)} panel models listed (a source needs {MIN_PANEL})'
elif statistics.stdev(board[m] for m in members) == 0:
why = 'every panel model has the same value'
else:
scale[src] = {'anchor_raw': board[ANCHOR], 'sd': statistics.stdev(board[m] for m in members), 'n': len(members)}
continue
self.source_rows[src]['not_scaled'] = why
self.miss(src, self.source_rows[src]['file'], f'not used: {why}')
return panel, core, scale
# --- tier -------------------------------------------------------------------------------------
def tier(self):
"""The tier entries (module docstring): arena.ai Overall at High, else AA Intelligence Index at High."""
fitter = tiers.Fitter(self.target, self.anchor_name, self.aa_records, self.readings)
overall = fitter.overall
as_of = max(r[4] for r in overall.values())
self.register('tier.arena_overall', 'text_style_control.parquet', as_of,
'tiers.py: arena.ai Overall (Text, style control) at High standardized relative to '
f'{self.anchor_name} (anchor = 0, continuous z-score)', len(overall))
self.register('tier.aa_index', 'aa-models.html', None,
'tiers.py: Artificial Analysis Intelligence Index at High interpolated onto the continuous '
'arena.ai Overall High scale through the models listed at High on both boards',
len(fitter.aa))
# the release's tier scale and map, so a model entry's tier is put on them (refresh.merge_model_rows, item 8)
self.source_rows['tier.arena_overall']['tier_scale'] = {'anchor': self.anchor_name,
'anchor_raw': round(fitter.anchor_raw, 6),
'sd': round(fitter.sd, 6)}
aa_map = fitter.maps.get('aa.index')
if aa_map is not None:
self.source_rows['tier.aa_index']['tier_map'] = {
'points': [[round(x, 6), round(y, 6), name] for (x, y), name in zip(aa_map.raw, aa_map.names)]}
arena, aa = Matcher(overall), Matcher(fitter.aa)
release_rec = {sources.base_name(r['name']): r for r in self.aa_records if 'page' in r}
for model in self.models:
explicit = tiers.listed_as(self.models, model, 'arena')
found = arena.find(model, explicit) or (arena.find(model) if explicit else None)
if found:
row = overall[found]
t = fitter.arena_overall_tier(found)
self.add(model, 'tier', 'tier.arena_overall', round(t, 4),
tiers.basis_arena(row, t, fitter.anchor_raw, fitter.sd, fitter.anchor_name),
listed_as=row[3], arena_value=round(row[0], 1), raw=round(float(row[0]), 6))
else:
explicit = tiers.listed_as(self.models, model, 'aa')
aa_found = aa.find(model, explicit) or (aa.find(model) if explicit else None)
if aa_found:
for board, r in fitter.other_overall({'aa.index': aa_found}).items():
self.add(model, 'tier', 'tier.aa_index', r['tier'],
f"{r['board']} {r['value']:g} at High interpolated onto the arena.ai Overall High scale "
f"between {r['neighbours']['below']['models']} and {r['neighbours']['above']['models']} "
f"of the {r['n']} models listed at High on both boards",
listed_as=r['listed_as'], board_value=r['value'], neighbours=r['neighbours'], n=r['n'],
fit=r['fit'], residual_sd=r['residual_sd'], loo_rmse=r['loo_rmse'], status=r['status'],
**self.cite(release_rec.get(aa_found, {})))
# --- price and usd per hour ------------------------------------------------------------------
def usd(self):
if not self.ok('aa-models.html'):
self.miss('aa.cost_time', 'aa-models.html', 'not fetched')
return
variants = self.aa_variants()
self.register('aa.cost_time', 'aa-models.html', None,
'AA Intelligence Index Time/Task (s) and outputSpeed (tok/s) per effort variant',
len(variants))
self.register('aa.price', 'aa-models.html', None, 'AA listed price1mInputTokens / price1mOutputTokens / cacheHitPrice', len(variants))
self.register('aa.index', 'aa-models.html', None, 'AA Intelligence Index at the High variant (else the only listing)', len(variants))
match = Matcher(variants)
idx_high, idx_efforts, idx_recs = {}, {}, {}
for model in self.models:
exp = tiers.listed_as(self.models, model, 'aa')
base = match.find(model, exp) or (match.find(model) if exp else None)
if not base:
continue
self.names[model]['aa.cost_time'] = base
for eff, rec in sorted(variants[base].items()):
t = rec.get('intelligenceIndexTimePerTask')
speed = rec.get('outputSpeed')
if t or (isinstance(speed, (int, float)) and speed > 0):
self.add(model, 'throughput', 'aa.cost_time', round(t / 60.0, 3) if t else None,
'AA Intelligence Index Time/Task (s) and outputSpeed (tok/s)',
effort=eff or 'default', listed_as=rec['name'], **self.cite(rec),
time_per_task_s=round(t, 2) if t else None,
time_per_task_min=round(t / 60.0, 3) if t else None,
output_tps=round(speed, 2) if isinstance(speed, (int, float)) and speed > 0 else None)
vals = {(eff or 'default'): float(rec['intelligenceIndex']) for eff, rec in variants[base].items()
if isinstance(rec.get('intelligenceIndex'), (int, float)) and not isinstance(rec.get('intelligenceIndex'), bool)
and rec['intelligenceIndex'] > 0}
rec = variants[base].get('high') or variants[base].get('') or next(iter(variants[base].values()))
if vals:
routable = [e for e in vals if e in EFFORT_RANK]
pick = ('high' if 'high' in vals else
'default' if 'default' in vals else
min(routable, key=lambda e: (abs(EFFORT_RANK[e] - EFFORT_RANK['high']), -EFFORT_RANK[e])) if routable else
next(iter(vals)))
idx_high[model], idx_efforts[model], idx_recs[model] = vals[pick], vals, rec
price = {k: rec.get(src) for k, src in (('input', 'price1mInputTokens'), ('output', 'price1mOutputTokens'),
('cache_read', 'cacheHitPrice')) if isinstance(rec.get(src), (int, float))}
if price:
self.add(model, 'price', 'aa.price', price, 'USD per million tokens', listed_as=rec['name'], **self.cite(rec))
panel_models = (getattr(self, 'panel_doc', None) or {}).get('models') or []
idx_panel = [m for m in panel_models if m in idx_high]
idx_scale = None
if ANCHOR in idx_high and len(idx_panel) >= MIN_PANEL and statistics.stdev(idx_high[m] for m in idx_panel) > 0:
idx_scale = {'anchor_raw': idx_high[ANCHOR], 'sd': statistics.stdev(idx_high[m] for m in idx_panel), 'n': len(idx_panel)}
self.source_rows['aa.index']['scale'] = {'anchor': ANCHOR, 'anchor_raw': round(idx_scale['anchor_raw'], 6),
'sd_panel': round(idx_scale['sd'], 6), 'n_panel': idx_scale['n']}
for model, val in idx_high.items():
rec, vals = idx_recs[model], idx_efforts[model]
extra_idx = {'raw_by_effort': {e: round(v, 4) for e, v in vals.items()}}
if idx_scale:
extra_idx['z_by_effort'] = {e: round((v - idx_scale['anchor_raw']) / idx_scale['sd'], 4) for e, v in vals.items()}
extra_idx['z'] = round((val - idx_scale['anchor_raw']) / idx_scale['sd'], 4)
self.add(model, 'intelligence_index', 'aa.index', round(val, 2), 'AA Intelligence Index, every listed effort',
listed_as=rec['name'], **self.cite(rec), **extra_idx)
if self.ok('openrouter-models.json'):
table = sources.openrouter(self.target)
self.register('openrouter.price', 'openrouter-models.json', None, 'OpenRouter /api/v1/models pricing (USD per million tokens)', len(table))
match = Matcher(table, lambda n: n.split('/')[-1])
for model in self.models:
name = match.find(model)
if name:
self.add(model, 'price', 'openrouter.price', {k: round(v, 4) for k, v in table[name].items()},
'USD per million tokens', listed_as=name)
else:
self.miss('openrouter.price', 'openrouter-models.json', 'not fetched')
self.arena_usd()
def arena_usd(self):
"""`arena.agent_sessions` usd per worker-hour (round 8, user 2026-11-09): arena.ai Agent / Overall real sessions,
Cost/Task (P50) / Time/Task (P50) x 60 per listed model/effort variant."""
fname = arena_capture.FILES['agent']
if not self.ok(fname):
self.miss('arena.agent_sessions', fname, 'not captured: arena_capture.py import ...')
return
doc = arena_capture.read(self.target, 'agent')
page = ((doc or {}).get('categories') or {}).get('Overall')
if not page or not page.get('rows'):
self.miss('arena.agent_sessions', fname, 'Agent / Overall not in capture')
return
by_base = {}
for r in page['rows']:
if r.get('name'):
by_base.setdefault(sources.base_name(r['name']), []).append(r)
priced = {b: [r for r in rows if isinstance(r.get('cost_per_task_usd'), (int, float))
and isinstance(r.get('time_per_task_min'), (int, float)) and r['time_per_task_min'] > 0]
for b, rows in by_base.items()}
self.register('arena.agent_sessions', fname, page.get('as_of'),
'arena.ai Agent / Overall real sessions: Cost/Task (P50) / Time/Task (P50) x 60',
sum(1 for rows in priced.values() if rows))
self.source_rows['arena.agent_sessions']['sessions'] = page.get('sessions')
self.source_rows['arena.agent_sessions']['url'] = (doc or {}).get('url') or arena_capture.URLS['agent']
unpriced = {}
match = Matcher(by_base)
for model in self.models:
exp = tiers.listed_as(self.models, model, 'arena')
base = match.find(model, exp) or (match.find(model) if exp else None)
if not base:
continue
self.names[model]['arena.agent_sessions'] = base
for r in by_base[base]:
t = r.get('time_per_task_min')
tok = r.get('output_tokens_per_task')
if isinstance(t, (int, float)) and t > 0:
eff = sources.aa_effort(r['name'], self.readings) or 'default'
self.add(model, 'throughput', 'arena.agent_sessions', round(t, 2),
'arena.ai Agent / Overall: Time/Task (P50) min and Output Tokens/Task (P50)',
effort=eff, listed_as=r['name'], time_per_task_min=round(t, 2),
output_tokens_per_task=round(tok) if isinstance(tok, (int, float)) else None,
output_tokens_per_min=round(tok / t, 1) if isinstance(tok, (int, float)) and t > 0 else None,
sessions=r.get('sessions'))
if not priced[base]:
r0 = by_base[base][0]
missing_cols = [col for col, key_ in (('Cost/Task', 'cost_per_task_usd'), ('Time/Task', 'time_per_task_min'))
if not (isinstance(r0.get(key_), (int, float)) and r0[key_] > 0)]
unpriced[model] = {'listed_as': r0['name'], 'missing': missing_cols,
'reason': f"listed on arena.ai Agent / Overall as {r0['name']} without {' or '.join(missing_cols)}"}
continue
for r in priced[base]:
c, t = r['cost_per_task_usd'], r['time_per_task_min']
tok = r.get('output_tokens_per_task')
eff = sources.aa_effort(r['name'], self.readings) or 'default'
self.add(model, 'usd_per_hour', 'arena.agent_sessions', round(c * 60.0 / t, 3),
'arena.ai Agent / Overall: Cost/Task (P50) / Time/Task (P50) x 60',
effort=eff, listed_as=r['name'], cost_per_task_usd=round(c, 4),
time_per_task_min=round(t, 2),
output_tokens_per_task=round(tok) if isinstance(tok, (int, float)) else None,
sessions=r.get('sessions'))
if unpriced:
self.source_rows['arena.agent_sessions']['unpriced'] = unpriced
# --- window capacity (community real usage) ---------------------------------------------------
def capacity(self):
rows, obs, rules, errors, mix = community.extract(self.target, self.manifest)
self.source_rows.update(rows)
self.mix = mix
return {'unit': 'usd of API-priced real usage per full window', 'observations': obs, 'rules': rules,
'exhaust': community.EXHAUST, 'exhaust_rejected': community.EXHAUST_REJECTED,
'hourly_checks': getattr(community.extract, 'hourly', []),
'errors': errors, 'not_found': community.NOT_FOUND, 'unmeasured': community.UNMEASURED,
'token_mix': mix}
def community_mix(self):
"""`derived.community_mix` usd per worker-hour (option B, user 2026-11-09): AA's measured output speed per hour
times the vendor list price of one output token's share of the community token mix (output + input + cache
reads, community.token_mix). Needs the mix, the output speed and all three AA prices; else no entry."""
mix = getattr(self, 'mix', None)
if not mix or not self.ok('aa-models.html'):
self.miss('derived.community_mix', 'aa-models.html', 'no community token mix' if not mix else 'not fetched')
return
per_out = {k: mix[k] / mix['output'] for k in ('input', 'cache_read')}
aa = self.source_rows['aa.cost_time']
self.source_rows['derived.community_mix'] = {
'file': aa['file'], 'url': aa['url'], 'fetched_at': aa['fetched_at'], 'sha256': aa['sha256'],
'as_of': min(mix['as_of'], aa['as_of']), 'as_of_kind': 'the older of the mix data date and the AA fetch date',
'method': (f"outputSpeed x 3600 x (output price + {per_out['input']:.2f} x input price + "
f"{per_out['cache_read']:.1f} x cache-read price) / 1e6"),
'models_listed': None, 'kind': 'derived',
'citations': [{'what': 'output speed and list prices', 'source': 'aa.price', 'url': aa['url'], 'as_of': aa['as_of']},
{'what': 'community token mix', 'source': mix['source'], 'url': mix['url'], 'as_of': mix['as_of'],
'quotes': mix['quotes']}]}
variants = self.aa_variants()
pages = {}
for model, base in self.names.items():
base = base.get('aa.cost_time')
if not base:
continue
for eff, rec in sorted(variants[base].items()):
speed = rec.get('outputSpeed')
prices = [rec.get(k) for k in ('price1mOutputTokens', 'price1mInputTokens', 'cacheHitPrice')]
if not isinstance(speed, (int, float)) or speed <= 0 or not all(isinstance(x, (int, float)) for x in prices):
continue
p_out, p_in, p_cr = prices
per_m_output = p_out + per_out['input'] * p_in + per_out['cache_read'] * p_cr
cite = self.cite(rec) # the as_of of this source stays the older of the mix and the AA dates
if cite:
pages[rec['page']] = cite
self.add(model, 'usd_per_hour', 'derived.community_mix', round(speed * 3600 / 1e6 * per_m_output, 3),
'AA output speed x community token mix at AA list prices', effort=eff or 'default',
listed_as=rec['name'], output_tokens_per_hour=round(speed * 3600),
usd_per_million_output_at_mix=round(per_m_output, 3),
prices={'output': p_out, 'input': p_in, 'cache_read': p_cr}, mix_source=mix['source'],
mix_as_of=mix['as_of'], **{k: v for k, v in cite.items() if k != 'as_of'})
self.source_rows['derived.community_mix']['citations'][1:1] = [
{'what': 'output speed and list prices (release page)', 'source': 'aa.price', 'url': c['url'], 'as_of': c['as_of']}
for _, c in sorted(pages.items())]
def fast_mode(self):
"""Cited vendor fast-mode price and speed multipliers and vendor API pricing (round 11, user 2026-11-10): the
accounts.json `models[m].vendor_fast_mode` declarations. Only a model with a cited vendor price and speed multiplier
gets a fast_mode measurement; other fast-capable models have no public multiplier (a surfaced gap)."""
fast = {m: info['vendor_fast_mode'] for m, info in self.models.items() if info.get('vendor_fast_mode')}
self.source_rows['vendor.fast_mode'] = {
'file': None, 'url': 'https://code.claude.com/docs/en/fast-mode', 'fetched_at': None, 'sha256': None,
'as_of': '2026-11-10', 'as_of_kind': 'data date',
'method': 'Anthropic Claude Code fast-mode and API pricing documentation (read 2026-11-10)',
'models_listed': len(fast), 'kind': 'vendor',
'pricing_url': 'https://platform.claude.com/docs/en/about-claude/pricing',
}
self.source_rows['vendor.price'] = {
'file': None, 'url': 'https://platform.claude.com/docs/en/about-claude/pricing', 'fetched_at': None, 'sha256': None,
'as_of': '2026-11-10', 'as_of_kind': 'data date',
'method': 'Anthropic API pricing documentation (read 2026-11-10): USD per million tokens',
'models_listed': len(fast), 'kind': 'vendor',
}
for model, spec in fast.items():
self.add(model, 'fast_mode', 'vendor.fast_mode',
{'price_multiplier': spec['price_multiplier'], 'speed_multiplier': spec['speed_multiplier']},
spec['statement'], fast_price=spec['fast_price'], standard_price=spec['standard_price'],
pricing_url=spec['pricing_url'])
self.add(model, 'price', 'vendor.price', dict(spec['standard_price']),
'USD per million tokens (Anthropic API pricing documentation)', listed_as=model)
def build(self):
self.release_gaps()
self.tier()
self.capabilities()
self.usd()
self.fast_mode()
capacity = self.capacity()
self.community_mix()
missing_sources = {**self.missing}
for name, row in self.manifest.items():
if row.get('status') != 'ok' and (name in self.release_files or not sources.AA_RELEASE_NAME.fullmatch(name)):
missing_sources.setdefault(name, {'file': name, 'url': row.get('url'), 'reason': row.get('error')})
doc = {'schema': SCHEMA, 'generated_at': None,
'conversion': f'z = (x - x_anchor) / sd_panel per board, anchor {ANCHOR}; see calibration/measurements.py',
'panel': self.panel_doc, 'checks': self.checks,
'sources': self.source_rows, 'missing_sources': missing_sources, 'conflicts': self.conflicts,
'axes': {a: list(c) for a, c in AXES.items()}, 'capacity': capacity,
'models': {m: f for m, f in self.out.items()}}
doc['generated_at'] = inputs_date(doc)
return doc
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument('--data')
ap.add_argument('--out', default=str(OUT))
args = ap.parse_args()
doc = Builder(sources.data_dir(args.data), sources.accounts()).build()
Path(args.out).write_text(json.dumps(doc, indent=1, ensure_ascii=False) + '\n', encoding='utf-8', newline='\n')
fields = ('tier', 'usd_per_hour', *AXES)
n = len(doc['models'])
print(f"{n} models -> {args.out}")
for f in fields:
print(f' {f:18s} {sum(1 for v in doc["models"].values() if v.get(f))}/{n} models with at least one source')
cap = doc['capacity']
print(f" capacity {len(cap['observations'])} community observations, {len(cap['rules'])} rules, "
f"{sum(e['level'] == 'red' for e in cap['errors'])} red errors")
for s, row in doc['missing_sources'].items():
print(f' missing source {s}: {row.get("reason")}')
if __name__ == '__main__':
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |