uniffy-io/uniffy / .agents/skills/model-catalog-audit
model-catalog-audit
VerifiedAudit the in-tree model catalog against each provider's live API (Anthropic, OpenAI, Google, OpenRouter, xAI) using the keys in .env. Surfaces new models the catalog is missing, catalog models a provider no longer returns, and price/context drift on the curated OpenRouter subset, so the catalog stays current. Works on both stacks (docker containers or host venv). Run periodically or when a provider ships new models / changes pricing.
Keep src/uniffy/data/models/catalog.json current by diffing
it against what the providers actually serve. The catalog is the source of truth
for the model list, pricing, capabilities, and image-gen support (see
.agents/rules/agents.md), so it drifts as providers ship and retire models.
When to use
- Periodically (e.g. monthly) to catch new models and retirements.
- When someone reports a missing model, or a provider announces a launch.
- After a pricing change announcement (the audit finds the models; you research the prices).
Steps
-
Detect the stack (same protocol as the stack notes in
AGENTS.md) - never assume:docker compose ps -q --status running backend- Output non-empty -> docker stack: the backend container has the venv,
the repo bind-mounted at
/app, and.envinjected viaenv_file. - Empty -> local ONLY if
.venv/bin/pythonexists on the host. A bare.venvdirectory does NOT count (empty or root-owned leftovers happen anduvwill fail trying to populate them). - Neither -> docker: the command below falls back to a one-off
run --rmbackend container (compose builds the dev image on first use). Never fix a broken host.venvby lettinguvsync onto the host.
- Output non-empty -> docker stack: the backend container has the venv,
the repo bind-mounted at
-
Run the audit from the repo root.
deps run -s backendis a rawuvpassthrough that targets whichever stack you detected:# docker stack (default) ./manage.py deps run -s backend run python .agents/skills/model-catalog-audit/scripts/audit_models.py # local stack ./manage.py deps run -s backend --stack local run python .agents/skills/model-catalog-audit/scripts/audit_models.pyIt loads provider keys from
.env(CLAUDE_API_KEY,OPENAI_API_KEY,GOOGLE_GENAI_API_KEY,XAI_API_KEY), lists each provider's live models, and diffs against the catalog. A provider with no key in.envis skipped. OpenRouter uses a public endpoint and needs no key, so it always runs. It is read-only - it never edits the catalog. -
Read the report. Per provider:
- NEW - a live model the catalog does not resolve. These are candidates
to add. Expect noise from OpenAI/Google: embeddings,
tts-*,whisper-*,gpt-realtime-*,*-moderation,aqa,deep-research-*,computer-use-*,sora-*are not chat or image-generation models - skip them unless we add a code path. Focus on chat models and image-gen models. - UNSEEN - a catalog model this key did not return. Often just means the account lacks access or it is a dated snapshot; do not delete blindly.
- openrouter - different semantics. The catalog holds a deliberately
curated subset of 400+ upstream models, so the script never reports NEW
there. Instead it reports MISSING UPSTREAM (a catalog slug or alias
the live list no longer serves - candidate for replacement or
deprecation), PRICE DRIFT (catalog
cost_per_1m_*vs live per-token prices times 1M, beyond a small tolerance), and CONTEXT DRIFT (catalogcontext_windowvs livecontext_length). - xai - standard NEW/UNSEEN via
XAI_API_KEY. When the richer/v1/language-modelsendpoint responds with pricing, the same PRICE DRIFT report runs; otherwise the plain/v1/modelslist is used.
- NEW - a live model the catalog does not resolve. These are candidates
to add. Expect noise from OpenAI/Google: embeddings,
-
Review the findings with the user before adopting anything.
-
Decide what to adopt. For each NEW model worth adding (a real chat or image-gen model), and for any model whose pricing may have changed:
- Research capabilities + pricing on the web - the anthropic/openai/ google APIs do not return pricing. Use the provider's official pricing/docs page (this is the same flow used to add the Gemini "Nano Banana" models). Record the source. For openrouter entries the audit's PRICE DRIFT output already carries the live rates.
- Confirm context window, max output tokens, vision/tools/reasoning support, and reasoning effort levels.
-
Edit
catalog.json(catwalk-shaped - match existing entries):- Chat model:
id,name,cost_per_1m_in/out,cost_per_1m_in_cached(cache write) +cost_per_1m_out_cached(cache read),context_window,default_max_tokens,can_reason(+reasoning_levels/default_reasoning_effortwhen it reasons),supports_attachments. - Image model: set
supports_image_generation: trueand eitherimage_prices(size/quality matrix) orcost_per_image(flat). Leave pricing out if genuinely unknown - the model is still selectable, cost just stays unrecorded. - Use
aliasesfor-preview/-latest/ dated variants that should resolve to one canonical entry. - For confirmed retirements, set
deprecated: trueandsunset_daterather than deleting, so existing references still resolve.
- Chat model:
-
Verify on the same stack you detected in step 1 (add
--stack localfor the host venv). The loader re-reads on file change:./manage.py deps run -s backend run python -c "from uniffy.domains.agents.providers.catalog import loader; loader.load_catalog(); print('catalog valid')" ./manage.py deps run -s backend run pytest src/uniffy/tests/test_pricing.py -qA malformed catalog raises on load (hard fail), so a clean load means the schema is satisfied.
Notes
- Price drift IS auto-detected for openrouter (its API exposes per-token
prices), and for xai when
/v1/language-modelsreports pricing. Anthropic, OpenAI, and Google expose no pricing, so their price changes remain a manual web-research step. - New entries are picked up live (mtime reload); no restart needed to test.
- Do not commit anything from
.envor print key values.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |