Skip to content

Kiln-AI/Kiln / .agents/skills/claude-maintain-models

claude-maintain-models

Verified

Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, wants to add a net-new AI provider to Kiln, or wants to write or fix the agent notes and tags of a model, a provider entry or a provider.

Add a New AI Model to Kiln

Branch check first: if the request involves a provider Kiln does not support yet, start at Adding a Net-New Provider. That workflow spans core, server, UI, tests, and tooling, and is gated on a client release — the model-entry steps below are only its final, gated piece.

For a new model on an already-supported provider, integrating it into libs/core/kiln_ai/adapters/ml_model_list.py requires:

  1. ModelName enum – add an enum member
  2. built_in_models list – add a KilnModel(...) entry with providers
  3. ModelFamily enum – only if the vendor is brand-new
  4. Agent notes and tags – on the new KilnModel and its provider entries, when a source supports them (see Agent notes and tags)

After code changes, run paid integration tests, then draft a Discord post.


Global Rules

These apply throughout the entire workflow.

  • Slug verification: NEVER guess or infer model slugs from naming patterns. Every model_id must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.
  • Date awareness: These models are often released very recently. Web search for current info before assuming you know the details.

Phase 1 – Model Discovery (only when asked to find new/missing models)

If the user asks you to find new models, do NOT just web search "new AI models this week" — that only surfaces major releases. Instead, systematically check each family against both the LiteLLM catalog and models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.

  1. Read the ModelFamily and ModelName enums to know what we already have.

  2. Query both catalogs for each family (run in parallel where possible):

    LiteLLM catalog — filters out mirror providers to avoid duplicates:

    curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != "openrouter" and .provider != "bedrock" and .provider != "bedrock_converse" and .provider != "vertex_ai-anthropic_models" and .provider != "azure") | .id] | unique | .[]'
    

    models.dev — search all model IDs across all providers:

    curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"
    

    For details on a specific provider+model: curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

  3. Search terms (one query per term): claude, gpt, o1, o3, o4 (OpenAI reasoning), gemini, llama, deepseek, qwen, qwq, mistral, grok, kimi, glm, minimax, hunyuan, ernie, phi, gemma, seed, step, pangu

  4. Union and cross-reference results from both catalogs against ModelName. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). Skip pure coding models (e.g. codestral, deepseek-coder, qwen-coder).

  5. Run targeted web searches per family to catch very fresh releases not yet in either catalog:

    • "[family] new model [current year]"
    • "[family] release [current month] [current year]"
  6. Present findings as a summary. Let the user decide which to add.


Phase 1B – Lagging-Provider Backfill Check (every run)

Some providers — Fireworks AI, Together AI, SiliconFlow, Cloudflare — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now and miss the window to backfill recently-added models whose provider support has since grown.

Run this check on every invocation of the skill, regardless of whether you're in discovery mode or adding a specific model.

  1. Pull the 10 most recently added models from git history. List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:

    git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20
    
  2. For the model you're adding (if any) AND each of those 10 models, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the Lagging Providers Reference. Do NOT trust models.dev / LiteLLM as the final word for these providers.

  3. If a lagging provider now supports a recently-added model that isn't yet in its KilnModel entry, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.

  4. Run the Cloudflare thinking-level checker (see Cloudflare thinking levels). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.


Phase 2 – Gather Context

  1. Read the predecessor model in ml_model_list.py (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.

  2. Query the LiteLLM catalog for the new model. This is the primary slug source since Kiln uses LiteLLM. See the Slug Lookup Reference for query syntax and all verified sources.

  3. Get the OpenRouter slug via:

    • curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"
    • Fallback: WebSearch for openrouter [model name] model id
  4. Get the direct-provider slug (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the Slug Lookup Reference for provider-specific URLs.

  5. Identify quirks — check the Provider Quirks Reference for the relevant provider, and web search for any new quirks:

    • Structured output mode (JSON schema vs function calling)?
    • Reasoning model (needs reasoning_capable, parsers, OpenRouter options)?
    • Vision/multimodal support? Which MIME types?
    • Provider-specific flags (temp_top_p_exclusive, etc.)?
    • Rate limit concerns (max_parallel_requests)?
  6. Determine thinking levels — does the model support configurable reasoning effort? See Thinking Levels Reference for the full lookup chain. Key quick checks:

    • Check the vendor model page (e.g. OpenAI model pages say "Reasoning.effort supports: X, Y, Z")
    • Check OpenRouter supported_parameters — if reasoning is absent, skip thinking levels
    • R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts

Phase 3 – Code Changes

All changes go in libs/core/kiln_ai/adapters/ml_model_list.py.

3a. ModelName enum

  • snake_case: claude_opus_4_6 = "claude_opus_4_6"
  • Place before predecessor (newer first within group). If the vendor is brand-new there is no predecessor — start a new group at the end of the enum
  • Follow existing grouping (all claude together, all gpt together, etc.)

3b. KilnModel entry in built_in_models

  • Place per the ordering rules in 3c — this placement is user-visible, get it right
  • Copy predecessor's structure and modify: name, friendly_name, model_id per provider, flags
  • friendly_name must follow the existing naming pattern of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use "Claude {version} Sonnet" (e.g. "Claude 4.5 Sonnet"), not "Claude Sonnet {version}". Do NOT use the vendor's marketing name if it differs from Kiln's established convention.

Provider model_id formats:

ProviderFormatNotes
openroutervendor/model-nameAlways verify via API
openaiBare model nameVerify via OpenAI docs
anthropicVariable — older models have date stamps, newer may notAlways verify via Anthropic docs
gemini_apiBare nameVerify via Google AI Studio docs
fireworks_aiaccounts/fireworks/models/...Verify via Fireworks docs
together_aiVendor path formatVerify via Together docs
vertexUsually same as gemini_apiVerify via Vertex docs
siliconflow_cnVendor/model formatVerify via SiliconFlow docs
featherless_aiHuggingFace repo id, case-sensitive (zai-org/GLM-5.2)Verify via their /v1/models — see Featherless
typesafeDotted System One version (jev-1.13.0)Their /v1/models lists aliases only — verify by POST /v1/systemone, see TypeSafe AI
cloudflareCloudflare's own @cf/<author>/<model> id, verbatim (@cf/zai-org/glm-5.3) — no cloudflare/ or openai/ prefixVerify via Cloudflare's model JSON — see Cloudflare

Every single model_id must be verified from an authoritative source. No exceptions.

Setting flags — use catalog data + predecessor as dual signals:

The LiteLLM catalog and models.dev responses include capability flags (supports_vision, supports_function_calling, supports_reasoning, etc.). Use these as the primary signal for what to enable on the new model:

  • If the catalog says supports_vision: true → enable supports_vision, multimodal_capable, and vision MIME types (see 2c)
  • If the catalog says supports_function_calling: true → use StructuredOutputMode.json_schema (or function_calling depending on provider norms — check predecessor)
  • If the catalog says supports_reasoning: true → the model can reason, but do NOT reflexively set reasoning_capable=True — default to reasoning_capable=False (see Reasoning Capable Default). Still add available_thinking_levels if it supports effort levels, and check parser/formatter flags.

Then cross-check against the predecessor. The predecessor tells you how Kiln configures a similar model (which structured_output_mode, which provider-specific flags, etc.). The catalog tells you what the model can do. Use both:

  • Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.
  • Predecessor has temp_top_p_exclusive but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.
  • Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.

Common flags:

  • structured_output_mode – how the model handles JSON output
  • suggested_for_evals / suggested_for_data_gen – see zero-sum rule below. If the entry also sets available_thinking_levels, its default_thinking_level must be a reasoning level — see Judge Models Must Reason by Default
  • multimodal_capable / supports_vision / supports_doc_extraction – see multimodal rules below
  • reasoning_capable – for thinking/reasoning models. Default new models to reasoning_capable=False unless the model always emits its reasoning (see Reasoning Capable Default)
  • temp_top_p_exclusive – Anthropic models that can't have both temp and top_p
  • parser / formatter – for models needing special parsing (e.g. R1-style thinking)

2c. Multimodal capabilities

If the model supports non-text inputs, configure:

  • multimodal_capable=True and supports_doc_extraction=True if it supports any MIME types
  • supports_vision=True if it supports images
  • multimodal_requires_pdf_as_image=True if vision-capable but no native PDF support (also add KilnMimeType.PDF to MIME list). Always set this on OpenRouter providers — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.
  • Always include KilnMimeType.TXT and KilnMimeType.MD on any multimodal_capable model

Strategy: start broad, narrow based on test failures. Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.

Full MIME superset (Gemini uses all):

# documents
KilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD
# images
KilnMimeType.JPG, KilnMimeType.PNG
# audio
KilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG
# video
KilnMimeType.MP4, KilnMimeType.MOV

3c. Ordering — the list IS the UI

The order of built_in_models is exactly the order users see: model dropdowns group by provider and list each provider's models in list order, and the model library page lists models in list order. A misplaced entry ships a scrambled dropdown to every client via the remote config. Rules:

  1. Families are contiguous. Every model of a family sits in one block. Never append a new family member at the bottom of the file or after another family — that strands it (past bugs: Mistral Small 4/3 ended up inside the Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).

  2. Within a family, versions run newest → oldest, top to bottom. 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the family block. A new model of an existing version goes inside that version's group — NOT at the top of the family and NOT below older versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7 landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).

  3. Within a version, big → small. Always.

    • Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite; Large > Medium > Small.
    • Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and VL Thinking blocks) stay intact, ordered descending internally.
    • Tier-grouped families follow the same rule for their tier blocks: Claude runs Fable, then Opus > Sonnet > Haiku, versions descending inside each tier. (It once ran Haiku-first, and several families ran sizes ascending — those were bugs, not conventions. Do not preserve an ascending run because it's "what the family already does.")
  4. A net-new family's block goes at the END of built_in_models (this is the existing convention — the newest niche vendors sit at the bottom of the list). The "place before predecessor" rule only applies within an existing family; a new family has no predecessor.

  5. Verify after editing — print the order you just shipped and eyeball the affected family:

    uv run python -c "
    from kiln_ai.adapters.ml_model_list import built_in_models
    for m in built_in_models: print(m.family, '|', m.friendly_name)"
    

3d. suggested_for_evals / suggested_for_data_gen

Only set these if the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).

Zero-sum rule: When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. Ask the user to confirm the swap before making changes.

3e. ModelFamily enum (only if needed)

Only add a new family if the vendor is completely new.

3f. Thinking Levels (available_thinking_levels / default_thinking_level)

If the model supports configurable reasoning effort (not just on/off), add available_thinking_levels and default_thinking_level to each provider entry. See Thinking Levels Reference for the full lookup chain and existing constants.

Quick rules:

  • Reuse an existing _THINKING_LEVELS constant if the levels match exactly
  • Create a new constant only if levels differ; name it {MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS
  • default_thinking_level must be one of the values in available_thinking_levels
  • If the entry sets suggested_for_evals=True, default_thinking_level must be "medium" (or "low" if the model has no medium level) — never "none". See Judge Models Must Reason by Default

3g. Agent notes and tags

For each new model, write agent_notes and the model tags, then run the current procedure and the set-review procedure. For each new provider entry, including one added to an existing model, set its tags (provider-entry tags are empty today; do not use model tags on a provider entry). Follow Agent notes and tags.

Every new model always gets at least:

  • A minimal note with Description: and Lineup: from the lab's own model page or announcement. These two lines need no benchmark source.
  • The current tag, when it is the newest release at its lab's price level. current is mechanical: it needs only the release date, not a benchmark.
  • A Superseded by: line on the predecessor, when the current tag moves.

Other note lines (Strengths, Behavior, etc.) and set tags (recommended, top_value, etc.) need an independent source. When no source supports a set tag or a judgment line, leave it out and say so in the PR.

Steps for each new model:

  1. Write the model's agent_notes — at minimum Description: + Lineup:.
  2. Run the current procedure for its lab and price level (always).
  3. Write set tags and judgment note lines where a source supports them.
  4. Run the set-review procedure for each set the model could join.

3h. PR description — set changes

When any set changed (a model added to or removed from a set tag), add a section at the top of the PR description that lists each addition and removal, per set, with a short reason for each. Example:

## Set changes
- **recommended**: +GPT-6.1 Terra (strong general quality, current); −GPT-5.6 Terra (superseded)
- **top_value**: +Claude Haiku 5.5 (best quality at its price)

Agent notes and tags

KilnModel.agent_notes, KilnModel.tags, KilnModelProvider.tags, and the notes and tags of each item in built_in_providers (libs/core/kiln_ai/adapters/ml_provider_list.py) are for agents that choose or configure a model, such as the Harness Tuner optimizer and the Kiln assistant. recommended is an agent's main candidate list for new agents, and a model without the current tag is not chosen for a new run config unless asked. Agents outside Kiln read these from the published remote config; Kiln clients do not load the provider list from it. editorial_notes is different: it is short text for people, shown on the models page of the Kiln UI. Do not copy one into the other.

Note format

A model's agent_notes is one string of labeled lines, joined by \n, in this order:

LabelRequiredContent
Description:yesOne line: what this model is.
Lineup:yesIts place on the lab's ladder, and what it replaced. Explains names that do not show the order.
Superseded by:when not current, except on a serving variant of a current modelThe model that replaced it, and why in a few words. Not present on a current model.
Same weights:on a serving variant and on its standard modelNames the other model that serves the same weights, and how it differs.
Useful for:noCases where it is a good choice, when it is not the general pick for its level.
Strengths:noIts character, not its rank: "strong at agentic coding, weak at factual recall".
Behavior:noSettings that help or hurt, output quirks, verbosity, tool-call reliability.
Deployment:noFacts that change a hosting decision: dense or MoE, runs on a laptop, fits affordable GPUs.

Each label is used at most once. A line has one or a few sentences. In ml_model_list.py, write a note as adjacent string literals, one per labeled line, each but the last ending in \n, so each line is one source line.

Never in a note. Anything that Kiln or models.dev already keeps: price, batch and cache discounts, and claims like "most expensive"; context window and maximum output; release date and knowledge cutoff; open weights; which providers host the model; support for function calling, structured output or vision; the available and default thinking levels. A note can say how the model behaves when a setting is used. It does not say which settings exist.

No words relative to time. A note must not need edits as it ages. Do not use "current", "latest", "newest", "now", "previous", "frontier", "state of the art", "best" or "peers". To compare, name an exact model or generation: "near Claude Opus 4.6 level, with open weights", "a step down from GPT-6.1 Sol", "in the GPT-6.1 generation". The current tag and the Superseded by: line, added once, are the only places that show age.

Approved examples

These show the format. Their facts are illustrative, not checked. Follow the same style.

GPT-6.1 Terra · tags: current, recommended

Description: OpenAI's mid-price general model in the GPT-6.1 generation.
Lineup: OpenAI's ladder is Luna < Terra < Sol < Astra. A large step up from GPT-6 Luna, a step down from GPT-6.1 Sol. Replaced GPT-5.6 Terra.
Strengths: Steady quality in multi-turn tool use; GPT-6.1 Sol is better on hard reasoning.
Behavior: High effort helps on hard problems; xhigh adds latency for little gain on routine tasks.

GPT-5.4 Nano · tags: none

Description: The smallest model in the GPT-5.4 generation.
Lineup: Below GPT-5.4 Mini.
Superseded by: GPT-6 Luna, which is stronger at a lower price.

Gemini 4 Argon · tags: current, recommended

Description: Google's Pro-level reasoning model in the Gemini 4 generation.
Lineup: Gemini 4 uses element names, not size names. Argon is the Pro-level model. Replaced the Gemini 3.x Pro previews.
Strengths: Strong multimodal reasoning over charts, video and documents; mixed on long autonomous coding runs.
Behavior: Very verbose: uses far more output tokens than GPT-6.1 Sol or Claude Opus 5.5 on the same tasks, so cost per task is high. Slow to the first token at high effort.

Claude Haiku 5.5 · tags: current, recommended, top_value, top_fast

Description: Anthropic's smallest and fastest model in the 5.5 generation.
Lineup: Anthropic's ladder is Haiku < Sonnet < Opus < Fable. A large step down from Claude Sonnet 5.5.
Useful for: Subagents, routing, classification and extraction at high volume.
Behavior: Reliable tool calls for its size; quality drops on long multi-step reasoning.

Mistral Large 4 · tags: current (not recommended)

Description: Mistral's open-weight flagship of the Large 4 generation.
Lineup: Mistral's ladder is Ministral < Small < Medium < Large. Replaced Mistral Large 3.
Useful for: Teams that need a European lab.
Strengths: Strong at coding and terminal tasks; near GPT-5.6 Terra level on general intelligence.

Qwen 3.8 27B · tags: current, top_open

Description: Qwen's dense mid-size open-weight model in the 3.8 generation.
Lineup: Below the Qwen 3.8 2.4T A95B open MoE. The closed Qwen 3.8 Max and Flash API models are a separate line.
Useful for: Self-hosted or on-prem work where you manage your own GPUs.
Strengths: Close to Gemma 4 31B at reasoning and coding.
Deployment: Dense, so it runs on one affordable GPU more easily than MoE models of similar quality.

Gemma 4 2B · tags: current, top_on_device

Description: The smallest open-weight model in Google's Gemma 4 generation.
Lineup: Gemma 4 sizes are 2B < 4B < 27B MoE < 31B dense. Gemma is Google's open-weight line, separate from Gemini.
Useful for: Local or on-device agents, when data must stay on the machine.
Behavior: Follows simple instructions well; unreliable at multi-step tool use.
Deployment: Runs on consumer laptops and high-end phones.

Provider note format

A provider's notes in built_in_providers follow the same rules, with these labels, in this order, each optional but at least one present:

LabelContent
Data retention:Whether it logs or trains on prompts; whether a zero-retention option exists, and how to get it.
Rate limits:How limits are set (account tier, per model); free-tier limits that break agents.
Routing:Whether it sends requests to more than one upstream host, so output or quality can change.
Gotchas:Setup or runtime problems: model access approval, region locks, project or quota setup.

Never in a provider note: pricing, the models it hosts (the model list has that) and the friendly name.

OpenRouter (provider note example)

Data retention: Depends on the upstream host. Account settings can block hosts that log or train on prompts.
Routing: Routes each request to one of several upstream hosts; quality and quantization can differ between them.
Gotchas: Some models are free variants with low rate limits; do not use them for production agents.

Tags

There are two tag kinds:

  • Lineage tag. There is one: current. It moves along a lab's line of models. See The current tag.
  • Set tag. Every other tag. A set tag names a short list of models that share a judgment, such as "the best open-weight models". Each set has a size limit. See Set tags.

The top_ prefix marks a set that is the best few models with a property, not every model with it: top_open is not "has open weights".

Read each tag's meaning, kind and limit in libs/core/kiln_ai/adapters/model_agent_info.py. This skill keeps no copy of the list, so the list and its meanings exist in one place. There are three maps:

  • MODEL_TAGS for KilnModel.tags: true of the model on every provider.
  • PROVIDER_ENTRY_TAGS for KilnModelProvider.tags: true of one provider entry only.
  • PROVIDER_LIST_TAGS for the tags of a built_in_providers item.

Use a tag only from the map for that field. A tag in the wrong map fails the tests. Format ^[a-z0-9_]{1,40}$ (TAG_PATTERN), at most 12 tags on one entry (MAX_TAGS_PER_ENTRY), and no tag twice.

The current tag

Price level is a step on the lab's own ladder, as the note's Lineup: line describes it: for example Luna < Terra < Sol < Astra, or Haiku < Sonnet < Opus < Fable. For open-weight labs, a size class.

Procedure when a model is added:

  1. Find the current model from the same lab at the same price level.
  2. Remove its current tag.
  3. Add a Superseded by: line to its note, naming the new model and why in a few words.
  4. Tag the new model current.
  5. Skip this if the new model is at a different price level. No other records change.

Rules:

  • Each lab has one current model at each price level: the newest release of that line, by release date, whatever its name.
  • current is mechanical: it needs only the release date and the lab's product line. No benchmark source is needed.
  • Snapshots and aliases. When a plain alias and a dated snapshot serve the same weights, decide with the information available and state the assumption in the PR body. For example, if the lab says the alias tracks the snapshot, the alias is current. If not clear, treat both as separate entries — the snapshot gets current when it is newer.
  • A variant that is a separate product (Pro, Omni) is a line of its own. It is current when it is the newest model of that variant line.
  • Unclear price levels. When a lab's naming does not make the price level clear (for example, Qwen 3.8 Max versus Qwen 3.8 Max Prime), decide with the information available and state the assumption in the PR body. For example, treat a variant with its own name as its own line unless the lab says it is an alias.
  • One current per weight release. A serving variant serves the same weights as another model, under another name, for example a faster hosted version (GLM 5.3 Fast, Kimi K3 Fast, MiMo-V2.6-Pro-UltraSpeed). A serving variant is never current; only the standard model is. Both notes get a Same weights: line that names the other: "GLM 5.3 Fast is a faster hosted version" and "GLM 5.3 is the standard version." A serving variant can still be in a set other than recommended (for example top_fast). When the standard model is superseded, its serving variants get a Superseded by: line too.
  • Deprecations. A model whose provider entries are all deprecated loses all its tags. A successor-slug migration moves current to the new entry. Tests skip fully deprecated models for the Superseded by: rule.

Local-only models. A model whose non-deprecated provider entries are all on local providers (Ollama, Docker Model Runner) is local-only. Local-only models are selected mainly so they can be judged for top_on_device (and top_open); they are hard to experiment with, so they are never recommended, and their notes can be short. They still follow the current procedure for their lab and size class.

Set tags

Procedure when a model could be in a set:

  1. Load the whole set (every model with that tag across built_in_models).
  2. Decide whether the new model joins the set.
  3. If it joins, review every member and remove each one that no longer earns its place. That can be an older model at the same level, or a model from any lab or level that the new model beats for the set's purpose.
  4. A set never goes over its limit (max_models in the tag map). To add a model to a full set, remove one.

Rules:

  • The bar for a top_ set: when an agent optimizes for that property, it should know about this model. So top_fast can hold a serving variant that is not among the fastest models overall, but is the faster way to run a strong model; its note says so.
  • Every recommended model is also current. When a model loses current, it also leaves recommended.
  • The recommended bar. recommended is not one model per lab or per level; current already gives that. A model is recommended only when Artificial Analysis (or, without an AA page, another independent benchmark) shows its quality is high for its cost, compared with the models across all labs at a similar cost. "The lab's flagship" or "the lab's smallest model" is not a reason. A model available only on local providers (Ollama, Docker Model Runner) is never recommended: an optimizer cannot easily test it.
  • A model in a set has a note, so a reviewer can read the reason.
  • A set tag is a claim the maintainer would stand behind. Apply it only when a source supports it, as for notes.

Sources

  • Artificial Analysis first for comparisons across labs: model pages (https://artificialanalysis.ai/models/<slug>) give the intelligence index, cost per task, output speed, time to first token and output tokens used.
  • A lab's own release benchmarks are chosen to look good. They can support a claim but are never its only source.
  • Common sense.

Sources log

When a set changes, update the sources log at specs/projects/model-agent-info/backfill_sources.md so a reviewer can answer "why is this model in this set?" without redoing the research:

  • Per set: one row per member with the AA intelligence index, AA cost per task (or blended price when AA has no cost per task), AA output speed, and the AA page URL. A member with no AA page cites the URL of the source used instead. The reason column states the comparison ("index 52 at $1.10/task; next best at that cost is 44"), not a description ("strong flagship").
  • Per model: the source URL behind each line of its note other than Description: and Lineup: when those come from the lab's own model page, which is cited once.
  • Per provider: the source URL behind each line of its note.
  • Every figure in the log is checked against its source; a price is labeled input or output.

Adding a tag

Add it to the right map in model_agent_info.py with its meaning, kind and limit, and add it to test_starting_tags in test_model_agent_info.py, in the same change that first uses it. A new set tag that names "the best" of something uses the top_ prefix. An unsupported claim is left out; the PR says so.

How the rules are checked

Kiln does not validate tags or notes when it loads the remote config, so an older client keeps loading a file that uses newer tags. Tests check the built-in data instead. Run them after every change to notes, tags or the provider list:

uv run pytest -q libs/core/kiln_ai/adapters/test_model_agent_info.py libs/core/kiln_ai/adapters/test_ml_model_list.py libs/core/kiln_ai/adapters/test_ml_provider_list.py

Phase 4 – Run Tests

Tests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.

Vertex AI authentication: Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run gcloud auth application-default login before trying. These failures are auth issues, not model config problems.

-k filter syntax: Always use bracket notation for model+provider filtering, never and:

  • Good: -k "test_name[glm_5-fireworks_ai]" or -k "glm_5"
  • Bad: -k "glm_5 and fireworks" — and is a pytest keyword expression that can match wrong tests

4.0 — If the test env can't build (blocked git dependency)

Kiln's core lib pins together to a git fork (libs/core/pyproject.toml: together = { git = "https://github.com/scosman/together-python" }). In a sandboxed environment whose GitHub access is scoped to kiln-ai/kiln only (e.g. Claude Code Web), uv sync fails to fetch that fork with a 403 from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only kiln-ai/kiln resolves.

Workaround to build the venv for testing (the together fork isn't exercised by model-integration tests, which route through LiteLLM):

  1. Do NOT use uv sync --no-sources — it strips the workspace workspace = true sources too and makes kiln-root → kiln-ai unsatisfiable.
  2. Instead, temporarily comment out ONLY the together = { git = ... } line in libs/core/pyproject.toml, then run uv sync (this resolves together from PyPI).
  3. Run the tests.
  4. Revert the workaround with git checkout -- libs/core/pyproject.toml uv.lock — this restores the commented-out together git-fork pin in libs/core/pyproject.toml (NOT the repo-root pyproject.toml, which the workaround never touches) and reverts uv.lock if it changed. Only ml_model_list.py (and any intended test-file edits) should remain modified.

Note: the PyPI together may pull slightly different transitive deps (e.g. a newer starlette), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your -k filters to the model files and ignore those.

4a. Parallel testing + API keys

Parallel testing is already on. There is no pytest.ini — pytest config lives in [tool.pytest.ini_options] in the root pyproject.toml, and addopts = "-n auto" is active. No edit and no revert are needed. (Only override to -n 8 if a provider rate-limits you.)

Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings. conftest.py has an autouse use_temp_settings_dir fixture that points Config.settings_path at a temp dir, so ~/.kiln_ai/settings.yaml is deliberately ignored during tests. A key the user added through the app's provider page will not be seen — the test fails with "Attempted to use X without an API key set", which looks like a config bug but isn't.

Bridge the key from the user's settings into the test environment without ever printing it:

export FEATHERLESS_AI_API_KEY="$(uv run python -c \
  "from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)" 2>/dev/null | tail -1)"

Use the provider's env_var name from libs/core/kiln_ai/utils/config.py. Some providers need more than one value: Cloudflare needs both CLOUDFLARE_API_KEY and CLOUDFLARE_ACCOUNT_ID (and optionally CLOUDFLARE_AI_GATEWAY_ID). To check which keys are available before running, print booleans only — never the values:

uv run python -c "from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))"

Many paid tests also carry the ollama marker, so --runpaid alone silently skips them. Always pass --runpaid --ollama together, and use -rs to see skip reasons when a test you expected to run reports as skipped.

4b. Smoke test — verify slug works

Run a single test+provider combo first:

uv run pytest --runpaid --ollama -k "test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]"

If it fails, fix the slug/config before proceeding. Use --collect-only to find exact parameter IDs if unsure.

4c. Full test suite

uv run pytest --runpaid --ollama -k "MODEL_ENUM" -v 2>&1 | grep -E "PASSED|FAILED|ERROR|short test|=====|collected"

If tests fail — debug one at a time:

  1. Pick ONE failing test, run it with -v for full output
  2. Fix the config
  3. Re-run that single test to verify
  4. Only re-run the full suite once the single test passes

Anthropic API key gotcha: if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as KILN_ANTHROPIC_API_KEY instead of ANTHROPIC_API_KEY (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't export it globally:

ANTHROPIC_API_KEY="$KILN_ANTHROPIC_API_KEY" uv run pytest --runpaid ...

4d. Extraction tests (if supports_doc_extraction=True)

Tests are in libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py.

# See what will run:
uv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM

# Run them:
uv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k "MODEL_ENUM"

If a provider rejects a data type (400 error), remove that KilnMimeType and re-run.

4e. Confirm failures are actually yours

Before treating a failure as a problem with your change, check whether the same test already fails for an existing provider of that model. Several assertions are provider-independent and fail regardless.

Known example: test_structured_input_cot_prompt_builder asserts len(trace) == 5 unconditionally, which is incompatible with any provider setting reasoning_capable=True (that selects the single-call strategy, producing 3 messages). It fails for gpt_oss_120b on fireworks_ai on a clean tree.

uv run pytest --runpaid --ollama -q "path::test_name[MODEL-OTHER_PROVIDER]"

If it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.

4f. Test output format

Collect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:

  • ✅ for passed tests
  • ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason
  • ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason
  • List every test using the full pytest parametrize ID, grouped by provider
  • Include extraction tests (Phase 4d) if they were run

Phase 5 – Create Pull Request

5.0 — Important context about Claude Code Web's stop hook

This skill is often run via Claude Code Web (Slack connector). That environment has a non-user-configurable stop hook which, at end of session, will:

  • Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
  • Instruct the agent to commit and push any local work before stopping
  • Explicitly tell the agent NOT to create a PR unless the user asked for one

The problems this causes:

  1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned add-model/* branches on the remote.
  2. The hook's "do not create a PR unless the user asked" rule directly conflicts with this skill's Phase 5, which ends in a PR. Running this skill is the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.

The user's desires, in priority order:

  1. Ask before you push. If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
  2. No abandoned branches. Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
  3. If the user says to abandon: revert your local changes (git restore / git clean the specific files you touched) and delete any branch you created (git checkout main && git branch -D add-model/MODEL_NAME) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.
  4. On a successful run, push and open the PR as described in 5a/5b. Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.

5.1 — Gate before pushing

Do NOT commit, push, or create a branch if any of the following are true:

  • Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
  • The smoke test (4b) failed and wasn't resolved
  • Any step in Phases 2–4 was skipped or incomplete
  • You are unsure whether a ⚠️ flake is actually a real failure

If any of the above apply, stop and ask the user what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.

After all tests pass, commit the changes and open a PR against main.

5a. Commit and push

  1. Create a new branch named add-model/MODEL_NAME (e.g. add-model/glm-5-1)
  2. Stage only the changed files (typically just ml_model_list.py)
  3. Commit with a concise message (e.g. "Add GLM 5.1 to model list (together_ai, siliconflow_cn)")
  4. Push the branch

5b. Create the PR

Use gh pr create against main. Follow Rule 0, Rule 1 and Step 3 of the open-pr skill (.agents/skills/open-pr/SKILL.md) for the human header, the CLA and the title:

  • The title is WIP: chore: followed by the model-list subject, for example WIP: chore: add GLM 5.1 to model list. Use chore, because adding a model is maintenance work, not a new feature. Only a human removes the WIP: prefix.
  • Copy everything above # Agentic PR Summary from .github/pull_request_template.md with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the ## Contributor License Agreement section only when the PR author's GitHub username is scosman, sfierro, leonardmq, tawnymanticore, or chiang-daniel. Keep it, unsigned, for any other author.

Replace the `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` placeholder under # Agentic PR Summary with the test results, in this exact format:

## What does this PR do?

 Test Results

[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]

[Model Name] ([provider]):
- [N] passed, [N] skipped[, [N] failed]
- [Any notable failures or flakes]

[Repeat for each model+provider combo]

---
[Model Name] ([provider]):
✅ test_data_gen_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]
✅ test_all_built_in_models_llm_as_judge[model_enum-provider]
✅ test_all_built_in_models_structured_output[model_enum-provider]
✅ test_all_built_in_models_structured_input[model_enum-provider]
✅ test_structured_output_cot_prompt_builder[model_enum-provider]
✅ test_all_models_providers_plaintext[model_enum-provider]
✅ test_cot_prompt_builder[model_enum-provider]
⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason
❌ test_name[model_enum-provider] — brief reason

[Repeat for each model+provider combo]

Rules for the PR body:

  • Every test that ran must appear in the per-test dump, using the full pytest parametrize ID
  • Group tests by [Model Name] ([provider]): headers
  • The summary section at the top gives a quick pass/skip/fail count per model+provider
  • The detailed section below the --- inside the summary (not the ---- above # Agentic PR Summary) lists every individual test result
  • Use ⚠️ for content quality flakes (not real failures), ❌ for real errors

Checklist

  • ModelName enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)
  • KilnModel entry added to built_in_models per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)
  • Printed the resulting list order and eyeballed the affected family (3c verification command)
  • friendly_name matches the naming pattern of sibling models in the same family
  • If the provider is net-new: followed Adding a Net-New Provider, including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)
  • ModelFamily enum updated (only if new family)
  • All provider slugs verified from authoritative sources
  • Flags inherited from predecessor and adjusted for quirks
  • reasoning_capable defaulted to False for adaptive-reasoning models (only True for always-emits-reasoning models — see Reasoning Capable Default)
  • Thinking levels configured if model supports reasoning effort (see Thinking Levels Reference)
  • Cloudflare and OpenRouter: check_thinking_levels.py reports no ❌ finding (see Cloudflare thinking levels and OpenRouter (general))
  • Every suggested_for_evals entry with thinking levels defaults to "medium" (or "low"), never "none" (see Judge Models Must Reason by Default)
  • Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)
  • Zero-sum applied if model is suggested for evals/data gen
  • agent_notes always written (at minimum Description: + Lineup:); set tags written where a source supports them, or left out and the PR says so (see Agent notes and tags)
  • RAG config templates updated if the new model replaces one used in app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts
  • API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)
  • Smoke test passed
  • Full test suite passed
  • Failures cross-checked against an existing provider before being called regressions (see 4e)
  • PR created against main with a WIP: title, the untouched human header, and test results in the Agentic PR Summary

Reasoning Capable Default

Default newly-added models to reasoning_capable=False, even when the catalog reports supports_reasoning: true.

Most recent models use adaptive reasoning and sometimes return no reasoning at all. Kiln raises RuntimeError("Reasoning is required for this model, but no reasoning was returned.") whenever a provider has reasoning_capable=True and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.

reasoning_capable is orthogonal to available_thinking_levels — you keep the important behavior with it set to False:

  • Thinking-level / effort selection still works (every GPT-5.x entry sets available_thinking_levels with reasoning_capable unset).
  • Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).
  • The test_thinking_level_reasoning_content paid test still runs and still asserts reasoning is present — it is gated on available_thinking_levels, not reasoning_capable. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels. OpenAI-direct providers are the exception: OpenAI returns reasoning only as a summary, so that test checks only that the run succeeds, and test_openai_reasoning_summaries_paid (same file) checks the summary on providers that set openai_reasoning_summary.

What you give up with reasoning_capable=False:

  • The conditional "reasoning is present in intermediate_outputs" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).
  • When a user attaches a chain-of-thought prompt, the model uses the two-call two_message_cot strategy instead of the single-call native single_turn_r1_thinking.

Keep reasoning_capable=True only for models that always emit reasoning in a native <think> format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.

Narrower alternative: if a model reliably reasons but you only hit the error on structured output, set reasoning_optional_for_structured_output=True (requires reasoning_capable=True) instead of disabling reasoning entirely.


Adding a Net-New Provider

Adding a provider Kiln has never supported is a bigger job than adding a model, and it has a hard sequencing constraint. Follow this section end to end.

The release-gating rule (do NOT skip)

The remote config is generated from main's built_in_models and reaches all existing clients immediately — but provider support (name map, connect flow, API key handling) only reaches users through a client app release. If model entries for a brand-new provider land on main before a client release with the provider plumbing is live, every deployed client shows the raw provider ID (e.g. featherless_ai) in the model library and offers models nobody can connect to. This happened with Featherless in Aug 2026 and the entries had to be rolled back.

Sequence it in two PRs:

  1. PR 1 — provider plumbing only. The ModelProviderName enum member and everything else in the checklist below EXCEPT the model catalog. Merge whenever ready. The provider's item in built_in_providers belongs here, because it holds the friendly name.
  2. PR 2 — model catalog. ALL ml_model_list.py changes: ModelName members, ModelFamily (if the vendor is new), and the built_in_models entries. Open it, but merge only after a client release containing PR 1 is live. (Kiln clients load only the model lists from the remote config. The provider list is published too, but only agents outside Kiln read it, so its item is safe in PR 1. The enums belong in the same PR as the entries they exist for.)

Touchpoint checklist (from the Featherless integration, #1618)

First, identify the provider's authentication model — the checklist below describes the common single-API-key pattern, but not every provider fits it. Existing variants to crib from: Bedrock stores an access key + secret pair, Fireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project ID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base URL with no credentials. Follow the closest existing analog's plumbing through config.py, provider_warnings, provider_api.py, and the connect page — the credential fields, validation call, and UI steps all change with the auth model.

libs/core:

  • ModelProviderName enum — libs/core/kiln_ai/datamodel/datamodel_enums.py
  • Credential storage — libs/core/kiln_ai/utils/config.py (for API-key providers: a new key with env_var; otherwise whatever fields the auth model needs)
  • LiteLLM provider mapping — libs/core/kiln_ai/utils/litellm.py
  • libs/core/kiln_ai/adapters/provider_tools.py — two spots: provider_warnings (missing-credential message, required_config_keys), and the adapter config (credential / base URL / headers plumbing)
  • Provider list — a KilnProviderInfo item in built_in_providers (libs/core/kiln_ai/adapters/ml_provider_list.py), at the same position as the enum member: id, friendly_name (the one Python source of the friendly name; provider_name_from_id reads it), models_dev_id (the provider's key in https://models.dev/api.json, checked there, or None when models.dev has no entry for it), and notes and tags when a source supports them (see Agent notes and tags)
  • Model catalog — libs/core/kiln_ai/adapters/ml_model_list.py: ModelName members, ModelFamily if needed, and the built_in_models entries (PR 2 only, see gating rule)

Desktop server (app/desktop/studio_server/provider_api.py):

  • connect_<provider> credential-validation endpoint (find a cheap authenticated call; see the Featherless connect function for a pattern when the provider has no authenticated GET to ping)
  • Disconnect handling (clear the stored credentials)
  • Tests in test_provider_api.py

Web UI (app/web_ui):

  • src/lib/stores.ts — provider_name_map entry (friendly name)
  • src/lib/ui/provider_image.ts + SVG in static/images/ — icon must match the monochrome convention: bare glyph, currentColor, no background tile, viewBox cropped to the artwork
  • Connect page — src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte: provider card (name, description, and the auth flow — api_key_steps / api_key_fields for key providers, or the custom flow the auth model needs) plus connected-status wiring from settings keys. A setting chosen at connect time is an api_key_options checkbox; its value reaches connect_api_key in the payload's options dict, and the provider's disconnect case resets it, because the dialog always opens unchecked (see OpenAI's reasoning summaries)
  • src/lib/api_schema.d.ts — regenerate with make schema (needs the server running on :8757)

Tests & tooling:

  • libs/core/kiln_ai/adapters/test_provider_tools.py, test_adapter_registry.py, model_adapters/test_litellm_adapter.py — extend the per-provider parametrized cases
  • .agents/scripts/provider_utils.py — add the provider's model-catalog endpoint so agent tooling can enumerate its models
  • Thinking levels — find out how the provider takes reasoning effort, and where it publishes the values each model accepts. Add a quirks section that names that source and says thinking levels are required wherever it lists values. If the provider silently remaps unsupported values, add the provider to check_thinking_levels.py, because a paid test cannot see a remap.
  • model_adapters/test_thinking_level_paid.py — add the provider to the provider list of test_thinking_level_reasoning_content, so each of its thinking levels gets a paid test case

Known gap: the generated Copilot API client (app/desktop/studio_server/api_client/) mirrors the remote Copilot service's schema — it can't learn the new provider until that service updates. Note it in the PR rather than hand-editing generated code.

Friendly names everywhere

The raw enum value must never be user-visible. When you add the provider, verify a friendly name exists in both name maps: the Python provider list (built_in_providers in ml_provider_list.py, which provider_name_from_id reads) and the web provider_name_map. When the enum gains a member, test_one_item_per_provider_in_enum_order in test_ml_provider_list.py fails until the provider list has its item, and TypeScript forces the web map. That is why the plumbing PR must not skip them.


Provider Quirks Reference

Anthropic

  • Newer models (Opus 4.1+, Sonnet 4.5+) need temp_top_p_exclusive=True
  • Opus 4.5+ uses json_schema; older Opus uses function_calling
  • Extended thinking models: anthropic_extended_thinking=True + reasoning_capable=True

OpenAI

  • Most GPT models use json_schema for structured output
  • GPT-5.x models support available_thinking_levels — see Thinking Levels Reference
  • Reasoning summaries: set openai_reasoning_summary="auto" on the openai provider of GPT-5.4+ and GPT-6+ reasoning models, and of the Pro models LiteLLM already sends to /v1/responses (GPT-5.2 Pro, GPT-5.4 Pro). Kiln passes the value to OpenAI as reasoning.summary unchanged, and only sends it when the user turned on the reasoning summaries checkbox when connecting OpenAI. Leave it unset on every other provider, OpenRouter included, and on models without reasoning.
  • Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort
  • o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)

Google/Gemini

  • gemini_reasoning_enabled=True for reasoning-capable models
  • Gemini 3.x models support available_thinking_levels — see Thinking Levels Reference
  • Rich multimodal support (audio, video, images, documents)

DeepSeek

  • R1 models: parser=ModelParserID.r1_thinking + reasoning_capable=True
  • V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN
  • Some need r1_openrouter_options=True + require_openrouter_reasoning=True

OpenRouter (general)

  • Slugs: vendor/model-name
  • Never add a :free slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as :free, wait for a paid slug.
  • Reasoning models: may need require_openrouter_reasoning=True
  • Some models: openrouter_skip_required_parameters=True
  • Logprobs: logprobs_openrouter_options=True if supported
  • Always multimodal_requires_pdf_as_image=True (OpenRouter's PDF routing breaks LiteLLM)
  • Thinking levels: offer only values in the model's reasoning.supported_efforts (see Thinking Levels Reference). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no reasoning.supported_efforts, OpenRouter turns the effort into a thinking-token budget for Claude, so CLAUDE_OPENROUTER_THINKING_LEVELS (with Minimal) fits only those Claude models. check_thinking_levels.py checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.

Fireworks AI and Together AI: thinking levels

Neither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes xhigh, max and minimal on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.

Featherless AI

Serverless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:

  • json_instructions is the only usable structured output mode. LiteLLM's featherless_ai provider rejects response_format outright (UnsupportedParamsError), so json_schema, json_mode, and json_instruction_and_object are all unavailable. Routing it as a custom openai provider bypasses that gate, but was tested and is not reliable: GLM 5.2 accepts json_schema and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.
  • Gated models return HTTP 403 model_gated_needs_oauth and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. Always filter is_gated before adding. Note all 20 official meta-llama/* repos are gated, so no Llama variant is usable.
  • No cost reporting. Featherless models aren't in LiteLLM's price map (only two legacy Qwerky entries, on main too — not a version issue), and Featherless doesn't return cost in the usage object. Runs record tokens with cost: null.
  • Not in models.dev or the LiteLLM catalog, so their /v1/models endpoint is the only authoritative source. See Lagging Providers.
  • Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.

TypeSafe AI

Serves TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:

  • Adapter-backed, not LiteLLM-backed. Every provider entry must set adapter=ModelAdapterId.jev; JevAdapter calls POST /v1/systemone directly. lite_llm_core_config_for_provider and get_litellm_provider_info raise for typesafe on purpose, so the LiteLLM parametrized tests exclude it.
  • Structured output only. No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set supports_structured_output=True, supports_data_gen=False, supports_function_calling=False, and structured_output_mode=StructuredOutputMode.json_schema.
  • No logprobs. supports_logprobs=False. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.
  • Not in models.dev or the LiteLLM catalog, and GET /v1/models cannot confirm a model_id. It lists aliases (jev-latest, jev-preview), not the pinned versions Kiln registers, and it returns {"models": [{"name", ...}]} rather than the OpenAI {"data": [{"id", ...}]} shape, so the shared fetch_openai_compat path cannot read it either. That is why typesafe is in SKIP_PROVIDERS in .agents/scripts/provider_utils.py rather than in PROVIDER_CONFIG. The authoritative check for a pinned id is a POST /v1/systemone round trip that the API accepts — that is how jev-1.13.0 was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)

Cloudflare (Workers AI)

Runs through LiteLLM's generic openai/ route against Cloudflare's OpenAI-compatible endpoint (/accounts/{id}/ai/v1/chat/completions), with an optional AI Gateway header. Kiln does not use LiteLLM's cloudflare/ provider. Read this before adding or changing any entry:

  • Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models. Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to 256 output tokens (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.

  • How to tell them apart: look at the model's input schema in the catalog JSON. OpenAI-format models list max_completion_tokens (and tool_choice); older models list prompt, raw and lora instead, and no max_completion_tokens:

    jq -r '.models[] | select(.task.name=="Text Generation")
      | [.name, (if (.schema.input | tostring | test("max_completion_tokens")) then "openai-format" else "OLDER - skip" end)] | @tsv' /tmp/cf_models.json
    
  • Also skip: coding models (kimi-*-code, qwen*-coder-*), llama-guard-* (classifier), -lora bases, and models Kiln has no ModelName for unless the user asks for a new model.

  • Usually a new provider on an existing model. Most Cloudflare models are already in Kiln under other providers; add a KilnModelProvider(name=ModelProviderName.cloudflare, ...) to the existing KilnModel.

  • Setting flags from the catalog's properties[] (confirm every flag with the paid tests — the catalog has no JSON-mode flag):

    Cloudflare propertyKiln setting
    function_calling: "true"leave supports_function_calling at its default; set False if the tool tests fail
    vision: "true"supports_vision, multimodal_capable, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set supports_doc_extraction: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway
    reasoning: "true"keep reasoning_capable=False (see Reasoning Capable Default); LiteLLM already reads reasoning_content
    reasoning_effort.supported_efforts presentrequired: set available_thinking_levels to exactly these values, and default_thinking_level to default_effort. See Cloudflare thinking levels
    reasoning_effort without supported_effortsno thinking levels; the catalog does not say which values work
    require_workers_paid: "true"no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR
    planned_deprecation_date presentdo not add the model
  • Thinking levels come from the catalog, and only from the catalog. Kiln sends the level as reasoning_effort. For each model, reasoning_effort in the catalog lists supported_efforts, a default_effort, and normalizes_to: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps medium to max, so a "Medium" option runs the most expensive effort.

    • Offer only values in supported_efforts, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, none still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.

    • Labels: none → "Off", low → "Low", medium → "Medium", high → "High", xhigh → "Extra High", max → "Max". List them from least to most effort.

    • default_thinking_level is default_effort, unless the entry is suggested for evals (see Judge Models Must Reason by Default).

    • mandatory: true means none is not in the list; do not add an Off level.

    • Reuse a constant when the values match exactly; else name a new one {MODEL}_CLOUDFLARE_THINKING_LEVELS.

    • Cloudflare edits these lists. After every change to a Cloudflare entry, and on every run of this skill, run the checker below. It must report no ❌ finding:

      uv run python .agents/skills/claude-maintain-models/scripts/check_thinking_levels.py
      

      It compares every Cloudflare and OpenRouter entry in this checkout with the provider's catalog, and exits 1 on a level or default outside supported_efforts. For Cloudflare it also exits 1 on a missing level list, or levels on a model whose catalog lists none. --provider cloudflare or --provider openrouter checks one provider. A note: line is information only.

    • Then run the paid test_thinking_level_reasoning_content for each new level. If -k "<model_enum>" collects no cloudflare/... case, cloudflare is missing from the provider list in libs/core/kiln_ai/adapters/model_adapters/test_thinking_level_paid.py; add it.

  • Structured output: these models take the OpenAI response_format envelope, so start from json_schema and fall back only on test failures. Cloudflare's JSON mode doesn't support streaming.

  • Don't send max_tokens. Live tests showed no truncation on the OpenAI-format models without it.

  • Workers Paid: many of the newer models need the Workers Paid plan. Paid tests need an account on it.

  • Rate limits: 300 requests/minute for most text models, but only 20/minute per model for Workers Paid-only models. Don't run the paid suite with many workers against one model; use -n 4 or lower.

  • Slow models: Qwen 3.8 27B runs at about 23 tokens/s. Long non-streaming generations hit Cloudflare's server-side timeout (HTTP 408, code 3046, after about 2 minutes).

  • No cost reporting. Kiln sends bare @cf/... ids through LiteLLM's openai/ route, which never match LiteLLM's price map (its few entries are keyed cloudflare/@cf/...), so every Cloudflare run records tokens with cost: null.

  • No Model Substitution. When Cloudflare retires a model it may silently alias the old ID to a different, often more expensive, model (Kimi K2.5 → K2.6 in May 2026). The response still echoes the requested ID, so Kiln can't detect it at runtime. Cloudflare announces retirements by adding planned_deprecation_date to the model (notice has been 13–22 days). Remove or mark deprecated every Kiln Cloudflare entry that gains a planned_deprecation_date, before that date. The kiln-check-deprecation skill reports these dates.

Qwen3 / Thinking Models

  • Thinking variants: reasoning_capable=True, parser=ModelParserID.r1_thinking
  • No-thinking variants: formatter=ModelFormatterID.qwen3_style_no_think
  • SiliconFlow may need siliconflow_enable_thinking=True/False

Thinking Levels Reference

Two providers publish the thinking levels per model, and their list is the only source for their entries: Cloudflare (see Cloudflare thinking levels) and OpenRouter (reasoning.supported_efforts in its models API, see OpenRouter (general)). For every other provider the levels must be sourced by hand. Use this lookup chain in priority order.

Lookup Chain

  1. Vendor model page (most authoritative)

    • OpenAI: Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL: https://developers.openai.com/api/docs/models/{model-id}
    • Anthropic: The effort docs list levels per model. Opus 4.6 supports low, medium, high, max; Sonnet 4.6 supports low, medium, high.
    • Google Gemini: The models API returns thinking: true/false (boolean only). Levels come from docs.
  2. Vercel AI Gateway docs — clean structured tables per provider:

    • https://vercel.com/docs/ai-gateway/capabilities/reasoning/openai
    • https://vercel.com/docs/ai-gateway/capabilities/reasoning/anthropic
    • https://vercel.com/docs/ai-gateway/capabilities/reasoning/google
  3. Inherit from predecessor — if the same family/tier model has a _THINKING_LEVELS dict, the new model very likely uses the same or a superset.

  4. OpenRouter reasoning — most models list their efforts:

    curl -s https://openrouter.ai/api/v1/models | jq '.data[] | select(.id == "SLUG") | {reasoning, supported_parameters}'
    

    reasoning.supported_efforts lists the efforts (highest first) and reasoning.default_effort the default. If reasoning is absent and supported_parameters has no reasoning, the model does not support effort levels — skip thinking levels entirely. OpenRouter documents that it maps an unsupported effort to the nearest supported one for some models, so an extra level may silently run a different effort.

  5. Smoke test — as a last resort, send a request with an invalid effort level and check the error message, which often enumerates the valid values.

Judge Models Must Reason by Default

If a provider entry sets both suggested_for_evals=True and available_thinking_levels, set default_thinking_level to "medium" when the model offers a medium level, else "low". Never leave it at "none" or unset.

The V2 LLM judge runner builds the judge model's run config without a thinking_level, so the adapter falls back to the provider entry's default_thinking_level. A default of "none" therefore ships a judge that never reasons, and the eval results page shows no thinking for it.

This applies only to judge-tagged entries. A non-judge entry of the same model may keep "none" — the two are set independently per provider entry.

test_judge_models_with_thinking_levels_default_to_reasoning in libs/core/kiln_ai/adapters/test_ml_model_list.py enforces this across the whole list.

Important Distinctions

  • Effort-level models (GPT-5.x, Claude 4.x, Gemini 3.x) → add available_thinking_levels dicts
  • R1-style thinking models (DeepSeek R1, Qwen thinking variants) → on/off thinking, NOT effort levels. Use reasoning_capable=True + parser=ModelParserID.r1_thinking. Do NOT add thinking level dicts.
  • Chat/instant models (e.g. GPT-5.3 Instant) → may not support reasoning effort at all. Verify on the vendor model page.

Existing Constants

Reuse when levels match exactly. Create a new constant only if levels differ. This is not an exhaustive list.

ConstantLevelsDefaultUsed by
GPT_5_4_OPENAI_THINKING_LEVELSnone, low, medium, high, xhighnoneGPT-5.4
GPT_5_4_PRO_OPENAI_THINKING_LEVELSmedium, high, xhighmediumGPT-5.4 Pro
GPT_5_2_OPENAI_THINKING_LEVELSnone, low, medium, high, xhighnoneGPT-5.2, GPT-5.2 Chat
GPT_5_2_PRO_OPENAI_THINKING_LEVELSmedium, high, xhighmediumGPT-5.2 Pro
GPT_5_1_OPENAI_THINKING_LEVELSnone, low, medium, highnoneGPT-5.1
GPT_5_OPENAI_THINKING_LEVELSminimal, low, medium, highmediumGPT-5, GPT-5 Mini, GPT-5 Nano, GPT-5 Chat
GEMINI_3_PRO_THINKING_LEVELSlow, medium, highhighGemini 3 Pro, Gemini 3.1 Pro
GEMINI_3_FLASH_THINKING_LEVELSminimal, low, medium, highhighGemini 3 Flash, Gemini 3.1 Flash Lite
CLAUDE_ANTHROPIC_EFFORT_THINKING_LEVELSlow, medium, highhighClaude (Anthropic direct)
CLAUDE_OPENROUTER_THINKING_LEVELSnone, minimal, low, medium, high, xhighnoneClaude (OpenRouter)

Sources That Do NOT Work

These were investigated and confirmed to lack thinking level data:

  • LiteLLM catalog — only supports_reasoning: true/false
  • Google Gemini models API — only thinking: true/false
  • OpenAI /v1/models endpoint — minimal object with no capability fields

Slug Lookup Reference

Use both LiteLLM and models.dev when looking up slugs — they complement each other. LiteLLM gives you the exact slugs Kiln will use (since Kiln runs on LiteLLM), while models.dev often has broader coverage of newer or niche models with pricing, context limits, and capability details.

LiteLLM Model Catalog (https://api.litellm.ai/model_catalog)

100 free requests/day, no key needed. Supports server-side filtering: model= (substring match), provider=, mode=, supports_vision=true, supports_reasoning=true, page_size=500.

# Find all variants of a model across providers:
curl -s 'https://api.litellm.ai/model_catalog?model=MODEL_NAME&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[] | {id, provider, mode, max_input_tokens, supports_vision, supports_reasoning, supports_function_calling}'

# List all models for a provider:
curl -s 'https://api.litellm.ai/model_catalog?provider=PROVIDER&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[].id'

models.dev (https://models.dev/api.json)

Mega JSON covering 50+ providers with model IDs, pricing, context limits, capabilities, and release dates. Large file — always use curl+jq, never WebFetch.

# Search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"

# List all model IDs for a specific provider:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models | keys[]'

# Get full details for a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

Other verified sources

Lagging Providers

Fireworks, Together, SiliconFlow and Cloudflare typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, always cross-check directly — both when adding a new model and when running the Phase 1B backfill check.

Fireworks AI — model pages are the most current source. WebFetch directly:

WebFetch https://fireworks.ai/models/fireworks/{model-slug}

Or browse the catalog at https://fireworks.ai/models. Kiln slug format: accounts/fireworks/models/{model-slug}.

Together AI — the /v1/models endpoint requires an API key. The key is set in the user's shell as TOGETHERAI_API_KEY (note the AI — this matches LiteLLM's together_ai env var; the plain TOGETHER_API_KEY is usually NOT set):

# List all Together model IDs matching a term:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"

# Full record for a specific slug:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | select(.id == "SLUG")'

If TOGETHERAI_API_KEY isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.

Featherless AI — appears in neither models.dev nor the LiteLLM catalog (the catalog has only two stale Qwerky entries), so /v1/models is the sole authoritative source. It needs no API key to enumerate.

The response is enormous — ~22,000 models, the vast majority community fine-tunes. Never WebFetch it and never dump it raw. Save it once, then filter with jq:

curl -s https://api.featherless.ai/v1/models -o /tmp/feath.json
jq '.data | length' /tmp/feath.json

Restrict to official vendor orgs, or you'll drown in forks like DavidAU/Gemma3-27B-it-vl-GLM-4.7-Uncensored-Heretic:

# newest official models, excluding gated ones (gated = unusable, see Provider Quirks)
jq -r '.data[]
  | select(.is_gated | not)
  | select(.id | test("^(deepseek-ai|zai-org|Qwen|moonshotai|MiniMaxAI|mistralai|openai|google|microsoft)/"))
  | "\(.id)\tctx=\(.context_length)\t$\(.pricing.input)/\(.pricing.output)\ttools=\(.features.tool_use // false)\timg=\(.features.image_input // false)"' \
  /tmp/feath.json | sort

Look up one model, including its capability flags and exact pricing:

jq '.data[] | select(.id == "zai-org/GLM-5.2")' /tmp/feath.json

The highest-precision way to find backfill candidates is to check whether a slug Kiln already ships for together_ai / siliconflow_cn exists verbatim here — those providers use the same HuggingFace repo-id convention, so a hit needs no guessing:

jq -r '.data[] | select(.is_gated | not) | .id' /tmp/feath.json | sort > /tmp/feath_ids.txt
grep -Fx "zai-org/GLM-5.2" /tmp/feath_ids.txt   # exit 0 = safe to add

Slugs are case-sensitive (google/gemma-4-31B-it, not -31b-it). Always confirm is_gated is false and cross-check features.tool_use / features.image_input against the flags you set — don't claim capabilities the endpoint doesn't advertise.

Cloudflare Workers AI — the authoritative list is Cloudflare's public, unauthenticated catalog JSON. It is about 900 KB and can return a transient 502, so save it to a file (retry on 502) and filter with jq; never WebFetch it:

curl -s https://ai-cloudflare-com.pages.dev/api/models -o /tmp/cf_models.json

# Text-generation models, newest first, with Kiln-relevant flags
jq -r '.models[] | select(.task.name=="Text Generation")
  | (reduce .properties[] as $p ({}; .[$p.property_id]=$p.value)) as $p
  | [.name, .created_at[0:10], ($p.context_window//"-"),
     (if $p.function_calling=="true" then "tools" else "" end),
     (if $p.vision=="true" then "vision" else "" end),
     (if $p.reasoning=="true" then "reasoning" else "" end),
     (if $p.require_workers_paid=="true" then "paid" else "" end),
     (if $p.lora=="true" then "lora" else "" end),
     ($p.planned_deprecation_date//"")] | @tsv' /tmp/cf_models.json | sort -t$'\t' -k2 -r

# Full record for one model (drop the large schema)
jq '.models[] | select(.name=="@cf/zai-org/glm-5.3") | del(.schema)' /tmp/cf_models.json

Model IDs are the name field (@cf/<author>/<model>). With credentials (CLOUDFLARE_API_KEY with Workers AI access, plus CLOUDFLARE_ACCOUNT_ID), the official API returns the same records; add &include_deprecated=true to see models retired in the last three months:

curl -s "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/models/search?task=Text%20Generation&per_page=100" \
  -H "Authorization: Bearer $CLOUDFLARE_API_KEY" | jq -r '.result[].name'
  • Cross-check: models.dev, provider key cloudflare-workers-ai (curl -s https://models.dev/api.json | jq '.["cloudflare-workers-ai"].models | keys[]'). It syncs hourly and is good for spotting additions, but it no longer deletes retired models, so never use it to decide a model still exists.
  • Never use LiteLLM's catalog for Cloudflare. Its cloudflare/ entries are a one-time import from June 2026: they miss newer models and still list retired ones.
  • Fallback: the per-model JSON files in the cloudflare/cloudflare-docs GitHub repo, src/content/workers-ai-models/*.json (same shape as the catalog).
  • Also scan the Workers AI changelog for launches in the last two weeks, in case the catalog lags.

See Cloudflare (Workers AI) in Provider Quirks for the inclusion rule and flags.

SiliconFlow — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:

WebFetch https://siliconflow.com/models
WebFetch https://siliconflow.com/models/{vendor}/{model}

When you find a new reliable slug source, append it here.

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall