uniffy-io/uniffy / .agents/skills/model-catalog-audit
model-catalog-audit
VerifiedAudit the in-tree model catalog against each provider's live API (Anthropic, OpenAI, Google, OpenRouter, xAI) using the keys in .env. Surfaces new models the catalog is missing, catalog models a provider no longer returns, and price/context drift on the curated OpenRouter subset, so the catalog stays current. Works on both stacks (docker containers or host venv). Run periodically or when a provider ships new models / changes pricing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.1 KB |
| scripts/audit_models.py | script | 10.9 KB |
SKILL.md
View on GitHub---
name: model-catalog-audit
description: Audit the in-tree model catalog against each provider's live API (Anthropic, OpenAI, Google, OpenRouter, xAI) using the keys in .env. Surfaces new models the catalog is missing, catalog models a provider no longer returns, and price/context drift on the curated OpenRouter subset, so the catalog stays current. Works on both stacks (docker containers or host venv). Run periodically or when a provider ships new models / changes pricing.
---
Keep `src/uniffy/data/models/catalog.json` current by diffing
it against what the providers actually serve. The catalog is the source of truth
for the model list, pricing, capabilities, and image-gen support (see
`.agents/rules/agents.md`), so it drifts as providers ship and retire models.
## When to use
- Periodically (e.g. monthly) to catch new models and retirements.
- When someone reports a missing model, or a provider announces a launch.
- After a pricing change announcement (the audit finds the models; you research
the prices).
## Steps
1. **Detect the stack** (same protocol as the stack notes in `AGENTS.md`) - never
assume:
```bash
docker compose ps -q --status running backend
```
- Output non-empty -> **docker stack**: the backend container has the venv,
the repo bind-mounted at `/app`, and `.env` injected via `env_file`.
- Empty -> local ONLY if `.venv/bin/python` exists on the host. A bare
`.venv` directory does NOT count (empty or root-owned leftovers happen and
`uv` will fail trying to populate them).
- Neither -> **docker**: the command below falls back to a one-off
`run --rm` backend container (compose builds the dev image on first use).
Never fix a broken host `.venv` by letting `uv` sync onto the host.
2. **Run the audit** from the repo root. `deps run -s backend` is a raw `uv`
passthrough that targets whichever stack you detected:
```bash
# docker stack (default)
./manage.py deps run -s backend run python .agents/skills/model-catalog-audit/scripts/audit_models.py
# local stack
./manage.py deps run -s backend --stack local run python .agents/skills/model-catalog-audit/scripts/audit_models.py
```
It loads provider keys from `.env` (`CLAUDE_API_KEY`, `OPENAI_API_KEY`,
`GOOGLE_GENAI_API_KEY`, `XAI_API_KEY`), lists each provider's live models,
and diffs against the catalog. A provider with no key in `.env` is skipped.
OpenRouter uses a public endpoint and needs no key, so it always runs. It is
read-only - it never edits the catalog.
3. **Read the report.** Per provider:
- **NEW** - a live model the catalog does not resolve. These are candidates
to add. Expect noise from OpenAI/Google: embeddings, `tts-*`, `whisper-*`,
`gpt-realtime-*`, `*-moderation`, `aqa`, `deep-research-*`, `computer-use-*`,
`sora-*` are **not chat or image-generation models** - skip them unless we
add a code path. Focus on chat models and image-gen models.
- **UNSEEN** - a catalog model this key did not return. Often just means the
account lacks access or it is a dated snapshot; do not delete blindly.
- **openrouter** - different semantics. The catalog holds a deliberately
curated subset of 400+ upstream models, so the script never reports NEW
there. Instead it reports **MISSING UPSTREAM** (a catalog slug or alias
the live list no longer serves - candidate for replacement or
deprecation), **PRICE DRIFT** (catalog `cost_per_1m_*` vs live per-token
prices times 1M, beyond a small tolerance), and **CONTEXT DRIFT** (catalog
`context_window` vs live `context_length`).
- **xai** - standard NEW/UNSEEN via `XAI_API_KEY`. When the richer
`/v1/language-models` endpoint responds with pricing, the same PRICE DRIFT
report runs; otherwise the plain `/v1/models` list is used.
4. **Review the findings with the user** before adopting anything.
5. **Decide what to adopt.** For each NEW model worth adding (a real chat or
image-gen model), and for any model whose pricing may have changed:
- **Research capabilities + pricing on the web** - the anthropic/openai/
google APIs do not return pricing. Use the provider's official
pricing/docs page (this is the same flow used to add the Gemini "Nano
Banana" models). Record the source. For openrouter entries the audit's
PRICE DRIFT output already carries the live rates.
- Confirm context window, max output tokens, vision/tools/reasoning support,
and reasoning effort levels.
6. **Edit `catalog.json`** (catwalk-shaped - match existing entries):
- Chat model: `id`, `name`, `cost_per_1m_in/out`, `cost_per_1m_in_cached`
(cache write) + `cost_per_1m_out_cached` (cache read), `context_window`,
`default_max_tokens`, `can_reason` (+ `reasoning_levels` /
`default_reasoning_effort` when it reasons), `supports_attachments`.
- Image model: set `supports_image_generation: true` and either
`image_prices` (size/quality matrix) or `cost_per_image` (flat). Leave
pricing out if genuinely unknown - the model is still selectable, cost just
stays unrecorded.
- Use `aliases` for `-preview` / `-latest` / dated variants that should
resolve to one canonical entry.
- For confirmed retirements, set `deprecated: true` and `sunset_date` rather
than deleting, so existing references still resolve.
7. **Verify** on the same stack you detected in step 1 (add `--stack local` for
the host venv). The loader re-reads on file change:
```bash
./manage.py deps run -s backend run python -c "from uniffy.domains.agents.providers.catalog import loader; loader.load_catalog(); print('catalog valid')"
./manage.py deps run -s backend run pytest src/uniffy/tests/test_pricing.py -q
```
A malformed catalog raises on load (hard fail), so a clean load means the
schema is satisfied.
## Notes
- Price drift IS auto-detected for openrouter (its API exposes per-token
prices), and for xai when `/v1/language-models` reports pricing. Anthropic,
OpenAI, and Google expose no pricing, so their price changes remain a manual
web-research step.
- New entries are picked up live (mtime reload); no restart needed to test.
- Do not commit anything from `.env` or print key values.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |