Kiln-AI/Kiln / .agents/skills/claude-maintain-models
claude-maintain-models
VerifiedAdd new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, wants to add a net-new AI provider to Kiln, or wants to write or fix the agent notes and tags of a model, a provider entry or a provider.
Add a New AI Model to Kiln
Branch check first: if the request involves a provider Kiln does not support yet, start at Adding a Net-New Provider. That workflow spans core, server, UI, tests, and tooling, and is gated on a client release — the model-entry steps below are only its final, gated piece.
For a new model on an already-supported provider, integrating it into
libs/core/kiln_ai/adapters/ml_model_list.py requires:
ModelNameenum – add an enum memberbuilt_in_modelslist – add aKilnModel(...)entry with providersModelFamilyenum – only if the vendor is brand-new- Agent notes and tags – on the new
KilnModeland its provider entries, when a source supports them (see Agent notes and tags)
After code changes, run paid integration tests, then draft a Discord post.
Global Rules
These apply throughout the entire workflow.
- Slug verification: NEVER guess or infer model slugs from naming patterns. Every
model_idmust come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it. - Date awareness: These models are often released very recently. Web search for current info before assuming you know the details.
Phase 1 – Model Discovery (only when asked to find new/missing models)
If the user asks you to find new models, do NOT just web search "new AI models this week" — that only surfaces major releases. Instead, systematically check each family against both the LiteLLM catalog and models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.
-
Read the
ModelFamilyandModelNameenums to know what we already have. -
Query both catalogs for each family (run in parallel where possible):
LiteLLM catalog — filters out mirror providers to avoid duplicates:
curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != "openrouter" and .provider != "bedrock" and .provider != "bedrock_converse" and .provider != "vertex_ai-anthropic_models" and .provider != "azure") | .id] | unique | .[]'models.dev — search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"For details on a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]' -
Search terms (one query per term):
claude,gpt,o1,o3,o4(OpenAI reasoning),gemini,llama,deepseek,qwen,qwq,mistral,grok,kimi,glm,minimax,hunyuan,ernie,phi,gemma,seed,step,pangu -
Union and cross-reference results from both catalogs against
ModelName. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). Skip pure coding models (e.g.codestral,deepseek-coder,qwen-coder). -
Run targeted web searches per family to catch very fresh releases not yet in either catalog:
"[family] new model [current year]""[family] release [current month] [current year]"
-
Present findings as a summary. Let the user decide which to add.
Phase 1B – Lagging-Provider Backfill Check (every run)
Some providers — Fireworks AI, Together AI, SiliconFlow, Cloudflare — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now and miss the window to backfill recently-added models whose provider support has since grown.
Run this check on every invocation of the skill, regardless of whether you're in discovery mode or adding a specific model.
-
Pull the 10 most recently added models from git history. List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:
git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20 -
For the model you're adding (if any) AND each of those 10 models, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the Lagging Providers Reference. Do NOT trust
models.dev/ LiteLLM as the final word for these providers. -
If a lagging provider now supports a recently-added model that isn't yet in its
KilnModelentry, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it. -
Run the Cloudflare thinking-level checker (see Cloudflare thinking levels). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.
Phase 2 – Gather Context
-
Read the predecessor model in
ml_model_list.py(e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it. -
Query the LiteLLM catalog for the new model. This is the primary slug source since Kiln uses LiteLLM. See the Slug Lookup Reference for query syntax and all verified sources.
-
Get the OpenRouter slug via:
curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"- Fallback: WebSearch for
openrouter [model name] model id
-
Get the direct-provider slug (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the Slug Lookup Reference for provider-specific URLs.
-
Identify quirks — check the Provider Quirks Reference for the relevant provider, and web search for any new quirks:
- Structured output mode (JSON schema vs function calling)?
- Reasoning model (needs
reasoning_capable, parsers, OpenRouter options)? - Vision/multimodal support? Which MIME types?
- Provider-specific flags (
temp_top_p_exclusive, etc.)? - Rate limit concerns (
max_parallel_requests)?
-
Determine thinking levels — does the model support configurable reasoning effort? See Thinking Levels Reference for the full lookup chain. Key quick checks:
- Check the vendor model page (e.g. OpenAI model pages say "Reasoning.effort supports: X, Y, Z")
- Check OpenRouter
supported_parameters— ifreasoningis absent, skip thinking levels - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts
Phase 3 – Code Changes
All changes go in libs/core/kiln_ai/adapters/ml_model_list.py.
3a. ModelName enum
- snake_case:
claude_opus_4_6 = "claude_opus_4_6" - Place before predecessor (newer first within group). If the vendor is brand-new there is no predecessor — start a new group at the end of the enum
- Follow existing grouping (all claude together, all gpt together, etc.)
3b. KilnModel entry in built_in_models
- Place per the ordering rules in 3c — this placement is user-visible, get it right
- Copy predecessor's structure and modify:
name,friendly_name,model_idper provider, flags friendly_namemust follow the existing naming pattern of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use"Claude {version} Sonnet"(e.g. "Claude 4.5 Sonnet"), not"Claude Sonnet {version}". Do NOT use the vendor's marketing name if it differs from Kiln's established convention.
Provider model_id formats:
| Provider | Format | Notes |
|---|---|---|
openrouter | vendor/model-name | Always verify via API |
openai | Bare model name | Verify via OpenAI docs |
anthropic | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |
gemini_api | Bare name | Verify via Google AI Studio docs |
fireworks_ai | accounts/fireworks/models/... | Verify via Fireworks docs |
together_ai | Vendor path format | Verify via Together docs |
vertex | Usually same as gemini_api | Verify via Vertex docs |
siliconflow_cn | Vendor/model format | Verify via SiliconFlow docs |
featherless_ai | HuggingFace repo id, case-sensitive (zai-org/GLM-5.2) | Verify via their /v1/models — see Featherless |
typesafe | Dotted System One version (jev-1.13.0) | Their /v1/models lists aliases only — verify by POST /v1/systemone, see TypeSafe AI |
cloudflare | Cloudflare's own @cf/<author>/<model> id, verbatim (@cf/zai-org/glm-5.3) — no cloudflare/ or openai/ prefix | Verify via Cloudflare's model JSON — see Cloudflare |
Every single model_id must be verified from an authoritative source. No exceptions.
Setting flags — use catalog data + predecessor as dual signals:
The LiteLLM catalog and models.dev responses include capability flags (supports_vision, supports_function_calling, supports_reasoning, etc.). Use these as the primary signal for what to enable on the new model:
- If the catalog says
supports_vision: true→ enablesupports_vision,multimodal_capable, and vision MIME types (see 2c) - If the catalog says
supports_function_calling: true→ useStructuredOutputMode.json_schema(orfunction_callingdepending on provider norms — check predecessor) - If the catalog says
supports_reasoning: true→ the model can reason, but do NOT reflexively setreasoning_capable=True— default toreasoning_capable=False(see Reasoning Capable Default). Still addavailable_thinking_levelsif it supports effort levels, and check parser/formatter flags.
Then cross-check against the predecessor. The predecessor tells you how Kiln configures a similar model (which structured_output_mode, which provider-specific flags, etc.). The catalog tells you what the model can do. Use both:
- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.
- Predecessor has
temp_top_p_exclusivebut nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track. - Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.
Common flags:
structured_output_mode– how the model handles JSON outputsuggested_for_evals/suggested_for_data_gen– see zero-sum rule below. If the entry also setsavailable_thinking_levels, itsdefault_thinking_levelmust be a reasoning level — see Judge Models Must Reason by Defaultmultimodal_capable/supports_vision/supports_doc_extraction– see multimodal rules belowreasoning_capable– for thinking/reasoning models. Default new models toreasoning_capable=Falseunless the model always emits its reasoning (see Reasoning Capable Default)temp_top_p_exclusive– Anthropic models that can't have both temp and top_pparser/formatter– for models needing special parsing (e.g. R1-style thinking)
2c. Multimodal capabilities
If the model supports non-text inputs, configure:
multimodal_capable=Trueandsupports_doc_extraction=Trueif it supports any MIME typessupports_vision=Trueif it supports imagesmultimodal_requires_pdf_as_image=Trueif vision-capable but no native PDF support (also addKilnMimeType.PDFto MIME list). Always set this on OpenRouter providers — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.- Always include
KilnMimeType.TXTandKilnMimeType.MDon anymultimodal_capablemodel
Strategy: start broad, narrow based on test failures. Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.
Full MIME superset (Gemini uses all):
# documents
KilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD
# images
KilnMimeType.JPG, KilnMimeType.PNG
# audio
KilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG
# video
KilnMimeType.MP4, KilnMimeType.MOV
3c. Ordering — the list IS the UI
The order of built_in_models is exactly the order users see: model dropdowns
group by provider and list each provider's models in list order, and the model
library page lists models in list order. A misplaced entry ships a scrambled
dropdown to every client via the remote config. Rules:
-
Families are contiguous. Every model of a family sits in one block. Never append a new family member at the bottom of the file or after another family — that strands it (past bugs: Mistral Small 4/3 ended up inside the Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).
-
Within a family, versions run newest → oldest, top to bottom. 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the family block. A new model of an existing version goes inside that version's group — NOT at the top of the family and NOT below older versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7 landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).
-
Within a version, big → small. Always.
- Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite; Large > Medium > Small.
- Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and VL Thinking blocks) stay intact, ordered descending internally.
- Tier-grouped families follow the same rule for their tier blocks: Claude runs Fable, then Opus > Sonnet > Haiku, versions descending inside each tier. (It once ran Haiku-first, and several families ran sizes ascending — those were bugs, not conventions. Do not preserve an ascending run because it's "what the family already does.")
-
A net-new family's block goes at the END of
built_in_models(this is the existing convention — the newest niche vendors sit at the bottom of the list). The "place before predecessor" rule only applies within an existing family; a new family has no predecessor. -
Verify after editing — print the order you just shipped and eyeball the affected family:
uv run python -c " from kiln_ai.adapters.ml_model_list import built_in_models for m in built_in_models: print(m.family, '|', m.friendly_name)"
3d. suggested_for_evals / suggested_for_data_gen
Only set these if the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).
Zero-sum rule: When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. Ask the user to confirm the swap before making changes.
3e. ModelFamily enum (only if needed)
Only add a new family if the vendor is completely new.
3f. Thinking Levels (available_thinking_levels / default_thinking_level)
If the model supports configurable reasoning effort (not just on/off), add available_thinking_levels and default_thinking_level to each provider entry. See Thinking Levels Reference for the full lookup chain and existing constants.
Quick rules:
- Reuse an existing
_THINKING_LEVELSconstant if the levels match exactly - Create a new constant only if levels differ; name it
{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS default_thinking_levelmust be one of the values inavailable_thinking_levels- If the entry sets
suggested_for_evals=True,default_thinking_levelmust be"medium"(or"low"if the model has no medium level) — never"none". See Judge Models Must Reason by Default
3g. Agent notes and tags
For each new model, write agent_notes and the model tags, then run the current procedure and the set-review procedure. For each new provider entry, including one added to an existing model, set its tags (provider-entry tags are empty today; do not use model tags on a provider entry). Follow Agent notes and tags.
Every new model always gets at least:
- A minimal note with
Description:andLineup:from the lab's own model page or announcement. These two lines need no benchmark source. - The
currenttag, when it is the newest release at its lab's price level.currentis mechanical: it needs only the release date, not a benchmark. - A
Superseded by:line on the predecessor, when thecurrenttag moves.
Other note lines (Strengths, Behavior, etc.) and set tags (recommended, top_value, etc.) need an independent source. When no source supports a set tag or a judgment line, leave it out and say so in the PR.
Steps for each new model:
- Write the model's
agent_notes— at minimumDescription:+Lineup:. - Run the
currentprocedure for its lab and price level (always). - Write set tags and judgment note lines where a source supports them.
- Run the set-review procedure for each set the model could join.
3h. PR description — set changes
When any set changed (a model added to or removed from a set tag), add a section at the top of the PR description that lists each addition and removal, per set, with a short reason for each. Example:
## Set changes
- **recommended**: +GPT-6.1 Terra (strong general quality, current); −GPT-5.6 Terra (superseded)
- **top_value**: +Claude Haiku 5.5 (best quality at its price)
Agent notes and tags
KilnModel.agent_notes, KilnModel.tags, KilnModelProvider.tags, and the notes and tags of each item in built_in_providers (libs/core/kiln_ai/adapters/ml_provider_list.py) are for agents that choose or configure a model, such as the Harness Tuner optimizer and the Kiln assistant. recommended is an agent's main candidate list for new agents, and a model without the current tag is not chosen for a new run config unless asked. Agents outside Kiln read these from the published remote config; Kiln clients do not load the provider list from it. editorial_notes is different: it is short text for people, shown on the models page of the Kiln UI. Do not copy one into the other.
Note format
A model's agent_notes is one string of labeled lines, joined by \n, in this order:
| Label | Required | Content |
|---|---|---|
Description: | yes | One line: what this model is. |
Lineup: | yes | Its place on the lab's ladder, and what it replaced. Explains names that do not show the order. |
Superseded by: | when not current, except on a serving variant of a current model | The model that replaced it, and why in a few words. Not present on a current model. |
Same weights: | on a serving variant and on its standard model | Names the other model that serves the same weights, and how it differs. |
Useful for: | no | Cases where it is a good choice, when it is not the general pick for its level. |
Strengths: | no | Its character, not its rank: "strong at agentic coding, weak at factual recall". |
Behavior: | no | Settings that help or hurt, output quirks, verbosity, tool-call reliability. |
Deployment: | no | Facts that change a hosting decision: dense or MoE, runs on a laptop, fits affordable GPUs. |
Each label is used at most once. A line has one or a few sentences. In ml_model_list.py, write a note as adjacent string literals, one per labeled line, each but the last ending in \n, so each line is one source line.
Never in a note. Anything that Kiln or models.dev already keeps: price, batch and cache discounts, and claims like "most expensive"; context window and maximum output; release date and knowledge cutoff; open weights; which providers host the model; support for function calling, structured output or vision; the available and default thinking levels. A note can say how the model behaves when a setting is used. It does not say which settings exist.
No words relative to time. A note must not need edits as it ages. Do not use "current", "latest", "newest", "now", "previous", "frontier", "state of the art", "best" or "peers". To compare, name an exact model or generation: "near Claude Opus 4.6 level, with open weights", "a step down from GPT-6.1 Sol", "in the GPT-6.1 generation". The current tag and the Superseded by: line, added once, are the only places that show age.
Approved examples
These show the format. Their facts are illustrative, not checked. Follow the same style.
GPT-6.1 Terra · tags: current, recommended
Description: OpenAI's mid-price general model in the GPT-6.1 generation.
Lineup: OpenAI's ladder is Luna < Terra < Sol < Astra. A large step up from GPT-6 Luna, a step down from GPT-6.1 Sol. Replaced GPT-5.6 Terra.
Strengths: Steady quality in multi-turn tool use; GPT-6.1 Sol is better on hard reasoning.
Behavior: High effort helps on hard problems; xhigh adds latency for little gain on routine tasks.
GPT-5.4 Nano · tags: none
Description: The smallest model in the GPT-5.4 generation.
Lineup: Below GPT-5.4 Mini.
Superseded by: GPT-6 Luna, which is stronger at a lower price.
Gemini 4 Argon · tags: current, recommended
Description: Google's Pro-level reasoning model in the Gemini 4 generation.
Lineup: Gemini 4 uses element names, not size names. Argon is the Pro-level model. Replaced the Gemini 3.x Pro previews.
Strengths: Strong multimodal reasoning over charts, video and documents; mixed on long autonomous coding runs.
Behavior: Very verbose: uses far more output tokens than GPT-6.1 Sol or Claude Opus 5.5 on the same tasks, so cost per task is high. Slow to the first token at high effort.
Claude Haiku 5.5 · tags: current, recommended, top_value, top_fast
Description: Anthropic's smallest and fastest model in the 5.5 generation.
Lineup: Anthropic's ladder is Haiku < Sonnet < Opus < Fable. A large step down from Claude Sonnet 5.5.
Useful for: Subagents, routing, classification and extraction at high volume.
Behavior: Reliable tool calls for its size; quality drops on long multi-step reasoning.
Mistral Large 4 · tags: current (not recommended)
Description: Mistral's open-weight flagship of the Large 4 generation.
Lineup: Mistral's ladder is Ministral < Small < Medium < Large. Replaced Mistral Large 3.
Useful for: Teams that need a European lab.
Strengths: Strong at coding and terminal tasks; near GPT-5.6 Terra level on general intelligence.
Qwen 3.8 27B · tags: current, top_open
Description: Qwen's dense mid-size open-weight model in the 3.8 generation.
Lineup: Below the Qwen 3.8 2.4T A95B open MoE. The closed Qwen 3.8 Max and Flash API models are a separate line.
Useful for: Self-hosted or on-prem work where you manage your own GPUs.
Strengths: Close to Gemma 4 31B at reasoning and coding.
Deployment: Dense, so it runs on one affordable GPU more easily than MoE models of similar quality.
Gemma 4 2B · tags: current, top_on_device
Description: The smallest open-weight model in Google's Gemma 4 generation.
Lineup: Gemma 4 sizes are 2B < 4B < 27B MoE < 31B dense. Gemma is Google's open-weight line, separate from Gemini.
Useful for: Local or on-device agents, when data must stay on the machine.
Behavior: Follows simple instructions well; unreliable at multi-step tool use.
Deployment: Runs on consumer laptops and high-end phones.
Provider note format
A provider's notes in built_in_providers follow the same rules, with these labels, in this order, each optional but at least one present:
| Label | Content |
|---|---|
Data retention: | Whether it logs or trains on prompts; whether a zero-retention option exists, and how to get it. |
Rate limits: | How limits are set (account tier, per model); free-tier limits that break agents. |
Routing: | Whether it sends requests to more than one upstream host, so output or quality can change. |
Gotchas: | Setup or runtime problems: model access approval, region locks, project or quota setup. |
Never in a provider note: pricing, the models it hosts (the model list has that) and the friendly name.
OpenRouter (provider note example)
Data retention: Depends on the upstream host. Account settings can block hosts that log or train on prompts.
Routing: Routes each request to one of several upstream hosts; quality and quantization can differ between them.
Gotchas: Some models are free variants with low rate limits; do not use them for production agents.
Tags
There are two tag kinds:
- Lineage tag. There is one:
current. It moves along a lab's line of models. See Thecurrenttag. - Set tag. Every other tag. A set tag names a short list of models that share a judgment, such as "the best open-weight models". Each set has a size limit. See Set tags.
The top_ prefix marks a set that is the best few models with a property, not every model with it: top_open is not "has open weights".
Read each tag's meaning, kind and limit in libs/core/kiln_ai/adapters/model_agent_info.py. This skill keeps no copy of the list, so the list and its meanings exist in one place. There are three maps:
MODEL_TAGSforKilnModel.tags: true of the model on every provider.PROVIDER_ENTRY_TAGSforKilnModelProvider.tags: true of one provider entry only.PROVIDER_LIST_TAGSfor thetagsof abuilt_in_providersitem.
Use a tag only from the map for that field. A tag in the wrong map fails the tests. Format ^[a-z0-9_]{1,40}$ (TAG_PATTERN), at most 12 tags on one entry (MAX_TAGS_PER_ENTRY), and no tag twice.
The current tag
Price level is a step on the lab's own ladder, as the note's Lineup: line describes it: for example Luna < Terra < Sol < Astra, or Haiku < Sonnet < Opus < Fable. For open-weight labs, a size class.
Procedure when a model is added:
- Find the
currentmodel from the same lab at the same price level. - Remove its
currenttag. - Add a
Superseded by:line to its note, naming the new model and why in a few words. - Tag the new model
current. - Skip this if the new model is at a different price level. No other records change.
Rules:
- Each lab has one
currentmodel at each price level: the newest release of that line, by release date, whatever its name. currentis mechanical: it needs only the release date and the lab's product line. No benchmark source is needed.- Snapshots and aliases. When a plain alias and a dated snapshot serve the same weights, decide with the information available and state the assumption in the PR body. For example, if the lab says the alias tracks the snapshot, the alias is
current. If not clear, treat both as separate entries — the snapshot getscurrentwhen it is newer. - A variant that is a separate product (Pro, Omni) is a line of its own. It is
currentwhen it is the newest model of that variant line. - Unclear price levels. When a lab's naming does not make the price level clear (for example, Qwen 3.8 Max versus Qwen 3.8 Max Prime), decide with the information available and state the assumption in the PR body. For example, treat a variant with its own name as its own line unless the lab says it is an alias.
- One
currentper weight release. A serving variant serves the same weights as another model, under another name, for example a faster hosted version (GLM 5.3 Fast, Kimi K3 Fast, MiMo-V2.6-Pro-UltraSpeed). A serving variant is nevercurrent; only the standard model is. Both notes get aSame weights:line that names the other: "GLM 5.3 Fast is a faster hosted version" and "GLM 5.3 is the standard version." A serving variant can still be in a set other thanrecommended(for exampletop_fast). When the standard model is superseded, its serving variants get aSuperseded by:line too. - Deprecations. A model whose provider entries are all deprecated loses all its tags. A successor-slug migration moves
currentto the new entry. Tests skip fully deprecated models for theSuperseded by:rule.
Local-only models. A model whose non-deprecated provider entries are all on local providers (Ollama, Docker Model Runner) is local-only. Local-only models are selected mainly so they can be judged for top_on_device (and top_open); they are hard to experiment with, so they are never recommended, and their notes can be short. They still follow the current procedure for their lab and size class.
Set tags
Procedure when a model could be in a set:
- Load the whole set (every model with that tag across
built_in_models). - Decide whether the new model joins the set.
- If it joins, review every member and remove each one that no longer earns its place. That can be an older model at the same level, or a model from any lab or level that the new model beats for the set's purpose.
- A set never goes over its limit (
max_modelsin the tag map). To add a model to a full set, remove one.
Rules:
- The bar for a
top_set: when an agent optimizes for that property, it should know about this model. Sotop_fastcan hold a serving variant that is not among the fastest models overall, but is the faster way to run a strong model; its note says so. - Every
recommendedmodel is alsocurrent. When a model losescurrent, it also leavesrecommended. - The
recommendedbar.recommendedis not one model per lab or per level;currentalready gives that. A model isrecommendedonly when Artificial Analysis (or, without an AA page, another independent benchmark) shows its quality is high for its cost, compared with the models across all labs at a similar cost. "The lab's flagship" or "the lab's smallest model" is not a reason. A model available only on local providers (Ollama, Docker Model Runner) is neverrecommended: an optimizer cannot easily test it. - A model in a set has a note, so a reviewer can read the reason.
- A set tag is a claim the maintainer would stand behind. Apply it only when a source supports it, as for notes.
Sources
- Artificial Analysis first for comparisons across labs: model pages (
https://artificialanalysis.ai/models/<slug>) give the intelligence index, cost per task, output speed, time to first token and output tokens used. - A lab's own release benchmarks are chosen to look good. They can support a claim but are never its only source.
- Common sense.
Sources log
When a set changes, update the sources log at specs/projects/model-agent-info/backfill_sources.md so a reviewer can answer "why is this model in this set?" without redoing the research:
- Per set: one row per member with the AA intelligence index, AA cost per task (or blended price when AA has no cost per task), AA output speed, and the AA page URL. A member with no AA page cites the URL of the source used instead. The reason column states the comparison ("index 52 at $1.10/task; next best at that cost is 44"), not a description ("strong flagship").
- Per model: the source URL behind each line of its note other than
Description:andLineup:when those come from the lab's own model page, which is cited once. - Per provider: the source URL behind each line of its note.
- Every figure in the log is checked against its source; a price is labeled input or output.
Adding a tag
Add it to the right map in model_agent_info.py with its meaning, kind and limit, and add it to test_starting_tags in test_model_agent_info.py, in the same change that first uses it. A new set tag that names "the best" of something uses the top_ prefix. An unsupported claim is left out; the PR says so.
How the rules are checked
Kiln does not validate tags or notes when it loads the remote config, so an older client keeps loading a file that uses newer tags. Tests check the built-in data instead. Run them after every change to notes, tags or the provider list:
uv run pytest -q libs/core/kiln_ai/adapters/test_model_agent_info.py libs/core/kiln_ai/adapters/test_ml_model_list.py libs/core/kiln_ai/adapters/test_ml_provider_list.py
Phase 4 – Run Tests
Tests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.
Vertex AI authentication: Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run gcloud auth application-default login before trying. These failures are auth issues, not model config problems.
-k filter syntax: Always use bracket notation for model+provider filtering, never and:
- Good:
-k "test_name[glm_5-fireworks_ai]"or-k "glm_5" - Bad:
-k "glm_5 and fireworks"—andis a pytest keyword expression that can match wrong tests
4.0 — If the test env can't build (blocked git dependency)
Kiln's core lib pins together to a git fork (libs/core/pyproject.toml: together = { git = "https://github.com/scosman/together-python" }). In a sandboxed environment whose GitHub access is scoped to kiln-ai/kiln only (e.g. Claude Code Web), uv sync fails to fetch that fork with a 403 from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only kiln-ai/kiln resolves.
Workaround to build the venv for testing (the together fork isn't exercised by model-integration tests, which route through LiteLLM):
- Do NOT use
uv sync --no-sources— it strips the workspaceworkspace = truesources too and makeskiln-root → kiln-aiunsatisfiable. - Instead, temporarily comment out ONLY the
together = { git = ... }line inlibs/core/pyproject.toml, then runuv sync(this resolvestogetherfrom PyPI). - Run the tests.
- Revert the workaround with
git checkout -- libs/core/pyproject.toml uv.lock— this restores the commented-outtogethergit-fork pin inlibs/core/pyproject.toml(NOT the repo-rootpyproject.toml, which the workaround never touches) and revertsuv.lockif it changed. Onlyml_model_list.py(and any intended test-file edits) should remain modified.
Note: the PyPI together may pull slightly different transitive deps (e.g. a newer starlette), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your -k filters to the model files and ignore those.
4a. Parallel testing + API keys
Parallel testing is already on. There is no pytest.ini — pytest config lives in [tool.pytest.ini_options] in the root pyproject.toml, and addopts = "-n auto" is active. No edit and no revert are needed. (Only override to -n 8 if a provider rate-limits you.)
Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings. conftest.py has an autouse use_temp_settings_dir fixture that points Config.settings_path at a temp dir, so ~/.kiln_ai/settings.yaml is deliberately ignored during tests. A key the user added through the app's provider page will not be seen — the test fails with "Attempted to use X without an API key set", which looks like a config bug but isn't.
Bridge the key from the user's settings into the test environment without ever printing it:
export FEATHERLESS_AI_API_KEY="$(uv run python -c \
"from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)" 2>/dev/null | tail -1)"
Use the provider's env_var name from libs/core/kiln_ai/utils/config.py. Some providers need more than one value: Cloudflare needs both CLOUDFLARE_API_KEY and CLOUDFLARE_ACCOUNT_ID (and optionally CLOUDFLARE_AI_GATEWAY_ID). To check which keys are available before running, print booleans only — never the values:
uv run python -c "from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))"
Many paid tests also carry the ollama marker, so --runpaid alone silently skips them. Always pass --runpaid --ollama together, and use -rs to see skip reasons when a test you expected to run reports as skipped.
4b. Smoke test — verify slug works
Run a single test+provider combo first:
uv run pytest --runpaid --ollama -k "test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]"
If it fails, fix the slug/config before proceeding. Use --collect-only to find exact parameter IDs if unsure.
4c. Full test suite
uv run pytest --runpaid --ollama -k "MODEL_ENUM" -v 2>&1 | grep -E "PASSED|FAILED|ERROR|short test|=====|collected"
If tests fail — debug one at a time:
- Pick ONE failing test, run it with
-vfor full output - Fix the config
- Re-run that single test to verify
- Only re-run the full suite once the single test passes
Anthropic API key gotcha: if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as KILN_ANTHROPIC_API_KEY instead of ANTHROPIC_API_KEY (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't export it globally:
ANTHROPIC_API_KEY="$KILN_ANTHROPIC_API_KEY" uv run pytest --runpaid ...
4d. Extraction tests (if supports_doc_extraction=True)
Tests are in libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py.
# See what will run:
uv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM
# Run them:
uv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k "MODEL_ENUM"
If a provider rejects a data type (400 error), remove that KilnMimeType and re-run.
4e. Confirm failures are actually yours
Before treating a failure as a problem with your change, check whether the same test already fails for an existing provider of that model. Several assertions are provider-independent and fail regardless.
Known example: test_structured_input_cot_prompt_builder asserts len(trace) == 5 unconditionally, which is incompatible with any provider setting reasoning_capable=True (that selects the single-call strategy, producing 3 messages). It fails for gpt_oss_120b on fireworks_ai on a clean tree.
uv run pytest --runpaid --ollama -q "path::test_name[MODEL-OTHER_PROVIDER]"
If it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.
4f. Test output format
Collect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:
- ✅ for passed tests
- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason
- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason
- List every test using the full pytest parametrize ID, grouped by provider
- Include extraction tests (Phase 4d) if they were run
Phase 5 – Create Pull Request
5.0 — Important context about Claude Code Web's stop hook
This skill is often run via Claude Code Web (Slack connector). That environment has a non-user-configurable stop hook which, at end of session, will:
- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
- Instruct the agent to commit and push any local work before stopping
- Explicitly tell the agent NOT to create a PR unless the user asked for one
The problems this causes:
- When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned
add-model/*branches on the remote. - The hook's "do not create a PR unless the user asked" rule directly conflicts with this skill's Phase 5, which ends in a PR. Running this skill is the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.
The user's desires, in priority order:
- Ask before you push. If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
- No abandoned branches. Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
- If the user says to abandon: revert your local changes (
git restore/git cleanthe specific files you touched) and delete any branch you created (git checkout main && git branch -D add-model/MODEL_NAME) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch. - On a successful run, push and open the PR as described in 5a/5b. Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.
5.1 — Gate before pushing
Do NOT commit, push, or create a branch if any of the following are true:
- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
- The smoke test (4b) failed and wasn't resolved
- Any step in Phases 2–4 was skipped or incomplete
- You are unsure whether a ⚠️ flake is actually a real failure
If any of the above apply, stop and ask the user what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.
After all tests pass, commit the changes and open a PR against main.
5a. Commit and push
- Create a new branch named
add-model/MODEL_NAME(e.g.add-model/glm-5-1) - Stage only the changed files (typically just
ml_model_list.py) - Commit with a concise message (e.g. "Add GLM 5.1 to model list (together_ai, siliconflow_cn)")
- Push the branch
5b. Create the PR
Use gh pr create against main. Follow Rule 0, Rule 1 and Step 3 of the open-pr skill (.agents/skills/open-pr/SKILL.md) for the human header, the CLA and the title:
- The title is
WIP: chore:followed by the model-list subject, for exampleWIP: chore: add GLM 5.1 to model list. Usechore, because adding a model is maintenance work, not a new feature. Only a human removes theWIP:prefix. - Copy everything above
# Agentic PR Summaryfrom.github/pull_request_template.mdwith no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the## Contributor License Agreementsection only when the PR author's GitHub username isscosman,sfierro,leonardmq,tawnymanticore, orchiang-daniel. Keep it, unsigned, for any other author.
Replace the `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` placeholder under # Agentic PR Summary with the test results, in this exact format:
## What does this PR do?
Test Results
[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]
[Model Name] ([provider]):
- [N] passed, [N] skipped[, [N] failed]
- [Any notable failures or flakes]
[Repeat for each model+provider combo]
---
[Model Name] ([provider]):
✅ test_data_gen_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]
✅ test_all_built_in_models_llm_as_judge[model_enum-provider]
✅ test_all_built_in_models_structured_output[model_enum-provider]
✅ test_all_built_in_models_structured_input[model_enum-provider]
✅ test_structured_output_cot_prompt_builder[model_enum-provider]
✅ test_all_models_providers_plaintext[model_enum-provider]
✅ test_cot_prompt_builder[model_enum-provider]
⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason
❌ test_name[model_enum-provider] — brief reason
[Repeat for each model+provider combo]
Rules for the PR body:
- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID
- Group tests by
[Model Name] ([provider]):headers - The summary section at the top gives a quick pass/skip/fail count per model+provider
- The detailed section below the
---inside the summary (not the----above# Agentic PR Summary) lists every individual test result - Use ⚠️ for content quality flakes (not real failures), ❌ for real errors
Checklist
-
ModelNameenum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor) -
KilnModelentry added tobuilt_in_modelsper the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list) - Printed the resulting list order and eyeballed the affected family (3c verification command)
-
friendly_namematches the naming pattern of sibling models in the same family - If the provider is net-new: followed Adding a Net-New Provider, including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)
-
ModelFamilyenum updated (only if new family) - All provider slugs verified from authoritative sources
- Flags inherited from predecessor and adjusted for quirks
-
reasoning_capabledefaulted toFalsefor adaptive-reasoning models (onlyTruefor always-emits-reasoning models — see Reasoning Capable Default) - Thinking levels configured if model supports reasoning effort (see Thinking Levels Reference)
- Cloudflare and OpenRouter:
check_thinking_levels.pyreports no ❌ finding (see Cloudflare thinking levels and OpenRouter (general)) - Every
suggested_for_evalsentry with thinking levels defaults to"medium"(or"low"), never"none"(see Judge Models Must Reason by Default) - Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)
- Zero-sum applied if model is suggested for evals/data gen
-
agent_notesalways written (at minimumDescription:+Lineup:); set tags written where a source supports them, or left out and the PR says so (see Agent notes and tags) - RAG config templates updated if the new model replaces one used in
app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts - API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)
- Smoke test passed
- Full test suite passed
- Failures cross-checked against an existing provider before being called regressions (see 4e)
- PR created against
mainwith aWIP:title, the untouched human header, and test results in the Agentic PR Summary
Reasoning Capable Default
Default newly-added models to reasoning_capable=False, even when the catalog reports supports_reasoning: true.
Most recent models use adaptive reasoning and sometimes return no reasoning at all. Kiln raises RuntimeError("Reasoning is required for this model, but no reasoning was returned.") whenever a provider has reasoning_capable=True and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.
reasoning_capable is orthogonal to available_thinking_levels — you keep the important behavior with it set to False:
- Thinking-level / effort selection still works (every GPT-5.x entry sets
available_thinking_levelswithreasoning_capableunset). - Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).
- The
test_thinking_level_reasoning_contentpaid test still runs and still asserts reasoning is present — it is gated onavailable_thinking_levels, notreasoning_capable. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels. OpenAI-direct providers are the exception: OpenAI returns reasoning only as a summary, so that test checks only that the run succeeds, andtest_openai_reasoning_summaries_paid(same file) checks the summary on providers that setopenai_reasoning_summary.
What you give up with reasoning_capable=False:
- The conditional "reasoning is present in
intermediate_outputs" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped). - When a user attaches a chain-of-thought prompt, the model uses the two-call
two_message_cotstrategy instead of the single-call nativesingle_turn_r1_thinking.
Keep reasoning_capable=True only for models that always emit reasoning in a native <think> format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.
Narrower alternative: if a model reliably reasons but you only hit the error on structured output, set reasoning_optional_for_structured_output=True (requires reasoning_capable=True) instead of disabling reasoning entirely.
Adding a Net-New Provider
Adding a provider Kiln has never supported is a bigger job than adding a model, and it has a hard sequencing constraint. Follow this section end to end.
The release-gating rule (do NOT skip)
The remote config is generated from main's built_in_models and reaches all
existing clients immediately — but provider support (name map, connect
flow, API key handling) only reaches users through a client app release. If
model entries for a brand-new provider land on main before a client release
with the provider plumbing is live, every deployed client shows the raw
provider ID (e.g. featherless_ai) in the model library and offers models
nobody can connect to. This happened with Featherless in Aug 2026 and the
entries had to be rolled back.
Sequence it in two PRs:
- PR 1 — provider plumbing only. The
ModelProviderNameenum member and everything else in the checklist below EXCEPT the model catalog. Merge whenever ready. The provider's item inbuilt_in_providersbelongs here, because it holds the friendly name. - PR 2 — model catalog. ALL
ml_model_list.pychanges:ModelNamemembers,ModelFamily(if the vendor is new), and thebuilt_in_modelsentries. Open it, but merge only after a client release containing PR 1 is live. (Kiln clients load only the model lists from the remote config. The provider list is published too, but only agents outside Kiln read it, so its item is safe in PR 1. The enums belong in the same PR as the entries they exist for.)
Touchpoint checklist (from the Featherless integration, #1618)
First, identify the provider's authentication model — the checklist below
describes the common single-API-key pattern, but not every provider fits it.
Existing variants to crib from: Bedrock stores an access key + secret pair,
Fireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project
ID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base
URL with no credentials. Follow the closest existing analog's plumbing through
config.py, provider_warnings, provider_api.py, and the connect page —
the credential fields, validation call, and UI steps all change with the auth
model.
libs/core:
ModelProviderNameenum —libs/core/kiln_ai/datamodel/datamodel_enums.py- Credential storage —
libs/core/kiln_ai/utils/config.py(for API-key providers: a new key withenv_var; otherwise whatever fields the auth model needs) - LiteLLM provider mapping —
libs/core/kiln_ai/utils/litellm.py libs/core/kiln_ai/adapters/provider_tools.py— two spots:provider_warnings(missing-credential message,required_config_keys), and the adapter config (credential / base URL / headers plumbing)- Provider list — a
KilnProviderInfoitem inbuilt_in_providers(libs/core/kiln_ai/adapters/ml_provider_list.py), at the same position as the enum member:id,friendly_name(the one Python source of the friendly name;provider_name_from_idreads it),models_dev_id(the provider's key inhttps://models.dev/api.json, checked there, orNonewhen models.dev has no entry for it), andnotesandtagswhen a source supports them (see Agent notes and tags) - Model catalog —
libs/core/kiln_ai/adapters/ml_model_list.py:ModelNamemembers,ModelFamilyif needed, and thebuilt_in_modelsentries (PR 2 only, see gating rule)
Desktop server (app/desktop/studio_server/provider_api.py):
connect_<provider>credential-validation endpoint (find a cheap authenticated call; see the Featherless connect function for a pattern when the provider has no authenticated GET to ping)- Disconnect handling (clear the stored credentials)
- Tests in
test_provider_api.py
Web UI (app/web_ui):
src/lib/stores.ts—provider_name_mapentry (friendly name)src/lib/ui/provider_image.ts+ SVG instatic/images/— icon must match the monochrome convention: bare glyph,currentColor, no background tile, viewBox cropped to the artwork- Connect page —
src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte: provider card (name, description, and the auth flow —api_key_steps/api_key_fieldsfor key providers, or the custom flow the auth model needs) plus connected-status wiring fromsettingskeys. A setting chosen at connect time is anapi_key_optionscheckbox; its value reachesconnect_api_keyin the payload'soptionsdict, and the provider's disconnect case resets it, because the dialog always opens unchecked (see OpenAI's reasoning summaries) src/lib/api_schema.d.ts— regenerate withmake schema(needs the server running on :8757)
Tests & tooling:
libs/core/kiln_ai/adapters/test_provider_tools.py,test_adapter_registry.py,model_adapters/test_litellm_adapter.py— extend the per-provider parametrized cases.agents/scripts/provider_utils.py— add the provider's model-catalog endpoint so agent tooling can enumerate its models- Thinking levels — find out how the provider takes reasoning effort, and
where it publishes the values each model accepts. Add a quirks section
that names that source and says thinking levels are required wherever
it lists values. If the provider silently remaps unsupported values,
add the provider to
check_thinking_levels.py, because a paid test cannot see a remap. model_adapters/test_thinking_level_paid.py— add the provider to the provider list oftest_thinking_level_reasoning_content, so each of its thinking levels gets a paid test case
Known gap: the generated Copilot API client
(app/desktop/studio_server/api_client/) mirrors the remote Copilot service's
schema — it can't learn the new provider until that service updates. Note it in
the PR rather than hand-editing generated code.
Friendly names everywhere
The raw enum value must never be user-visible. When you add the provider,
verify a friendly name exists in both name maps: the Python provider list
(built_in_providers in ml_provider_list.py, which provider_name_from_id
reads) and the web provider_name_map. When the enum gains a member,
test_one_item_per_provider_in_enum_order in test_ml_provider_list.py
fails until the provider list has its item, and TypeScript forces the web
map. That is why the plumbing PR must not skip them.
Provider Quirks Reference
Anthropic
- Newer models (Opus 4.1+, Sonnet 4.5+) need
temp_top_p_exclusive=True - Opus 4.5+ uses
json_schema; older Opus usesfunction_calling - Extended thinking models:
anthropic_extended_thinking=True+reasoning_capable=True
OpenAI
- Most GPT models use
json_schemafor structured output - GPT-5.x models support
available_thinking_levels— see Thinking Levels Reference - Reasoning summaries: set
openai_reasoning_summary="auto"on theopenaiprovider of GPT-5.4+ and GPT-6+ reasoning models, and of the Pro models LiteLLM already sends to/v1/responses(GPT-5.2 Pro, GPT-5.4 Pro). Kiln passes the value to OpenAI asreasoning.summaryunchanged, and only sends it when the user turned on the reasoning summaries checkbox when connecting OpenAI. Leave it unset on every other provider, OpenRouter included, and on models without reasoning. - Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort
- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)
Google/Gemini
gemini_reasoning_enabled=Truefor reasoning-capable models- Gemini 3.x models support
available_thinking_levels— see Thinking Levels Reference - Rich multimodal support (audio, video, images, documents)
DeepSeek
- R1 models:
parser=ModelParserID.r1_thinking+reasoning_capable=True - V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN
- Some need
r1_openrouter_options=True+require_openrouter_reasoning=True
OpenRouter (general)
- Slugs:
vendor/model-name - Never add a
:freeslug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as:free, wait for a paid slug. - Reasoning models: may need
require_openrouter_reasoning=True - Some models:
openrouter_skip_required_parameters=True - Logprobs:
logprobs_openrouter_options=Trueif supported - Always
multimodal_requires_pdf_as_image=True(OpenRouter's PDF routing breaks LiteLLM) - Thinking levels: offer only values in the model's
reasoning.supported_efforts(see Thinking Levels Reference). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has noreasoning.supported_efforts, OpenRouter turns the effort into a thinking-token budget for Claude, soCLAUDE_OPENROUTER_THINKING_LEVELS(with Minimal) fits only those Claude models.check_thinking_levels.pychecks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.
Fireworks AI and Together AI: thinking levels
Neither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes xhigh, max and minimal on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.
Featherless AI
Serverless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:
json_instructionsis the only usable structured output mode. LiteLLM'sfeatherless_aiprovider rejectsresponse_formatoutright (UnsupportedParamsError), sojson_schema,json_mode, andjson_instruction_and_objectare all unavailable. Routing it as a customopenaiprovider bypasses that gate, but was tested and is not reliable: GLM 5.2 acceptsjson_schemaand silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.- Gated models return HTTP 403
model_gated_needs_oauthand cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. Always filteris_gatedbefore adding. Note all 20 officialmeta-llama/*repos are gated, so no Llama variant is usable. - No cost reporting. Featherless models aren't in LiteLLM's price map (only two legacy
Qwerkyentries, onmaintoo — not a version issue), and Featherless doesn't return cost in theusageobject. Runs record tokens withcost: null. - Not in models.dev or the LiteLLM catalog, so their
/v1/modelsendpoint is the only authoritative source. See Lagging Providers. - Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.
TypeSafe AI
Serves TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:
- Adapter-backed, not LiteLLM-backed. Every provider entry must set
adapter=ModelAdapterId.jev;JevAdaptercallsPOST /v1/systemonedirectly.lite_llm_core_config_for_providerandget_litellm_provider_inforaise fortypesafeon purpose, so the LiteLLM parametrized tests exclude it. - Structured output only. No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set
supports_structured_output=True,supports_data_gen=False,supports_function_calling=False, andstructured_output_mode=StructuredOutputMode.json_schema. - No logprobs.
supports_logprobs=False. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval. - Not in models.dev or the LiteLLM catalog, and
GET /v1/modelscannot confirm amodel_id. It lists aliases (jev-latest,jev-preview), not the pinned versions Kiln registers, and it returns{"models": [{"name", ...}]}rather than the OpenAI{"data": [{"id", ...}]}shape, so the sharedfetch_openai_compatpath cannot read it either. That is whytypesafeis inSKIP_PROVIDERSin.agents/scripts/provider_utils.pyrather than inPROVIDER_CONFIG. The authoritative check for a pinned id is aPOST /v1/systemoneround trip that the API accepts — that is howjev-1.13.0was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)
Cloudflare (Workers AI)
Runs through LiteLLM's generic openai/ route against Cloudflare's OpenAI-compatible endpoint (/accounts/{id}/ai/v1/chat/completions), with an optional AI Gateway header. Kiln does not use LiteLLM's cloudflare/ provider. Read this before adding or changing any entry:
-
Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models. Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to 256 output tokens (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.
-
How to tell them apart: look at the model's input schema in the catalog JSON. OpenAI-format models list
max_completion_tokens(andtool_choice); older models listprompt,rawandlorainstead, and nomax_completion_tokens:jq -r '.models[] | select(.task.name=="Text Generation") | [.name, (if (.schema.input | tostring | test("max_completion_tokens")) then "openai-format" else "OLDER - skip" end)] | @tsv' /tmp/cf_models.json -
Also skip: coding models (
kimi-*-code,qwen*-coder-*),llama-guard-*(classifier),-lorabases, and models Kiln has noModelNamefor unless the user asks for a new model. -
Usually a new provider on an existing model. Most Cloudflare models are already in Kiln under other providers; add a
KilnModelProvider(name=ModelProviderName.cloudflare, ...)to the existingKilnModel. -
Setting flags from the catalog's
properties[](confirm every flag with the paid tests — the catalog has no JSON-mode flag):Cloudflare property Kiln setting function_calling: "true"leave supports_function_callingat its default; setFalseif the tool tests failvision: "true"supports_vision,multimodal_capable, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't setsupports_doc_extraction: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gatewayreasoning: "true"keep reasoning_capable=False(see Reasoning Capable Default); LiteLLM already readsreasoning_contentreasoning_effort.supported_effortspresentrequired: set available_thinking_levelsto exactly these values, anddefault_thinking_leveltodefault_effort. See Cloudflare thinking levelsreasoning_effortwithoutsupported_effortsno thinking levels; the catalog does not say which values work require_workers_paid: "true"no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR planned_deprecation_datepresentdo not add the model -
Thinking levels come from the catalog, and only from the catalog. Kiln sends the level as
reasoning_effort. For each model,reasoning_effortin the catalog listssupported_efforts, adefault_effort, andnormalizes_to: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 mapsmediumtomax, so a "Medium" option runs the most expensive effort.-
Offer only values in
supported_efforts, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2,nonestill returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider. -
Labels:
none→"Off",low→"Low",medium→"Medium",high→"High",xhigh→"Extra High",max→"Max". List them from least to most effort. -
default_thinking_levelisdefault_effort, unless the entry is suggested for evals (see Judge Models Must Reason by Default). -
mandatory: truemeansnoneis not in the list; do not add an Off level. -
Reuse a constant when the values match exactly; else name a new one
{MODEL}_CLOUDFLARE_THINKING_LEVELS. -
Cloudflare edits these lists. After every change to a Cloudflare entry, and on every run of this skill, run the checker below. It must report no ❌ finding:
uv run python .agents/skills/claude-maintain-models/scripts/check_thinking_levels.pyIt compares every Cloudflare and OpenRouter entry in this checkout with the provider's catalog, and exits 1 on a level or default outside
supported_efforts. For Cloudflare it also exits 1 on a missing level list, or levels on a model whose catalog lists none.--provider cloudflareor--provider openrouterchecks one provider. Anote:line is information only. -
Then run the paid
test_thinking_level_reasoning_contentfor each new level. If-k "<model_enum>"collects nocloudflare/...case,cloudflareis missing from the provider list inlibs/core/kiln_ai/adapters/model_adapters/test_thinking_level_paid.py; add it.
-
-
Structured output: these models take the OpenAI
response_formatenvelope, so start fromjson_schemaand fall back only on test failures. Cloudflare's JSON mode doesn't support streaming. -
Don't send
max_tokens. Live tests showed no truncation on the OpenAI-format models without it. -
Workers Paid: many of the newer models need the Workers Paid plan. Paid tests need an account on it.
-
Rate limits: 300 requests/minute for most text models, but only 20/minute per model for Workers Paid-only models. Don't run the paid suite with many workers against one model; use
-n 4or lower. -
Slow models: Qwen 3.8 27B runs at about 23 tokens/s. Long non-streaming generations hit Cloudflare's server-side timeout (HTTP 408, code 3046, after about 2 minutes).
-
No cost reporting. Kiln sends bare
@cf/...ids through LiteLLM'sopenai/route, which never match LiteLLM's price map (its few entries are keyedcloudflare/@cf/...), so every Cloudflare run records tokens withcost: null. -
No Model Substitution. When Cloudflare retires a model it may silently alias the old ID to a different, often more expensive, model (Kimi K2.5 → K2.6 in May 2026). The response still echoes the requested ID, so Kiln can't detect it at runtime. Cloudflare announces retirements by adding
planned_deprecation_dateto the model (notice has been 13–22 days). Remove or mark deprecated every Kiln Cloudflare entry that gains aplanned_deprecation_date, before that date. Thekiln-check-deprecationskill reports these dates.
Qwen3 / Thinking Models
- Thinking variants:
reasoning_capable=True,parser=ModelParserID.r1_thinking - No-thinking variants:
formatter=ModelFormatterID.qwen3_style_no_think - SiliconFlow may need
siliconflow_enable_thinking=True/False
Thinking Levels Reference
Two providers publish the thinking levels per model, and their list is the only source for their entries: Cloudflare (see Cloudflare thinking levels) and OpenRouter (reasoning.supported_efforts in its models API, see OpenRouter (general)). For every other provider the levels must be sourced by hand. Use this lookup chain in priority order.
Lookup Chain
-
Vendor model page (most authoritative)
- OpenAI: Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL:
https://developers.openai.com/api/docs/models/{model-id} - Anthropic: The effort docs list levels per model. Opus 4.6 supports
low, medium, high, max; Sonnet 4.6 supportslow, medium, high. - Google Gemini: The models API returns
thinking: true/false(boolean only). Levels come from docs.
- OpenAI: Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL:
-
Vercel AI Gateway docs — clean structured tables per provider:
https://vercel.com/docs/ai-gateway/capabilities/reasoning/openaihttps://vercel.com/docs/ai-gateway/capabilities/reasoning/anthropichttps://vercel.com/docs/ai-gateway/capabilities/reasoning/google
-
Inherit from predecessor — if the same family/tier model has a
_THINKING_LEVELSdict, the new model very likely uses the same or a superset. -
OpenRouter
reasoning— most models list their efforts:curl -s https://openrouter.ai/api/v1/models | jq '.data[] | select(.id == "SLUG") | {reasoning, supported_parameters}'reasoning.supported_effortslists the efforts (highest first) andreasoning.default_effortthe default. Ifreasoningis absent andsupported_parametershas noreasoning, the model does not support effort levels — skip thinking levels entirely. OpenRouter documents that it maps an unsupported effort to the nearest supported one for some models, so an extra level may silently run a different effort. -
Smoke test — as a last resort, send a request with an invalid effort level and check the error message, which often enumerates the valid values.
Judge Models Must Reason by Default
If a provider entry sets both suggested_for_evals=True and available_thinking_levels, set default_thinking_level to "medium" when the model offers a medium level, else "low". Never leave it at "none" or unset.
The V2 LLM judge runner builds the judge model's run config without a thinking_level, so the adapter falls back to the provider entry's default_thinking_level. A default of "none" therefore ships a judge that never reasons, and the eval results page shows no thinking for it.
This applies only to judge-tagged entries. A non-judge entry of the same model may keep "none" — the two are set independently per provider entry.
test_judge_models_with_thinking_levels_default_to_reasoning in libs/core/kiln_ai/adapters/test_ml_model_list.py enforces this across the whole list.
Important Distinctions
- Effort-level models (GPT-5.x, Claude 4.x, Gemini 3.x) → add
available_thinking_levelsdicts - R1-style thinking models (DeepSeek R1, Qwen thinking variants) → on/off thinking, NOT effort levels. Use
reasoning_capable=True+parser=ModelParserID.r1_thinking. Do NOT add thinking level dicts. - Chat/instant models (e.g. GPT-5.3 Instant) → may not support reasoning effort at all. Verify on the vendor model page.
Existing Constants
Reuse when levels match exactly. Create a new constant only if levels differ. This is not an exhaustive list.
| Constant | Levels | Default | Used by |
|---|---|---|---|
GPT_5_4_OPENAI_THINKING_LEVELS | none, low, medium, high, xhigh | none | GPT-5.4 |
GPT_5_4_PRO_OPENAI_THINKING_LEVELS | medium, high, xhigh | medium | GPT-5.4 Pro |
GPT_5_2_OPENAI_THINKING_LEVELS | none, low, medium, high, xhigh | none | GPT-5.2, GPT-5.2 Chat |
GPT_5_2_PRO_OPENAI_THINKING_LEVELS | medium, high, xhigh | medium | GPT-5.2 Pro |
GPT_5_1_OPENAI_THINKING_LEVELS | none, low, medium, high | none | GPT-5.1 |
GPT_5_OPENAI_THINKING_LEVELS | minimal, low, medium, high | medium | GPT-5, GPT-5 Mini, GPT-5 Nano, GPT-5 Chat |
GEMINI_3_PRO_THINKING_LEVELS | low, medium, high | high | Gemini 3 Pro, Gemini 3.1 Pro |
GEMINI_3_FLASH_THINKING_LEVELS | minimal, low, medium, high | high | Gemini 3 Flash, Gemini 3.1 Flash Lite |
CLAUDE_ANTHROPIC_EFFORT_THINKING_LEVELS | low, medium, high | high | Claude (Anthropic direct) |
CLAUDE_OPENROUTER_THINKING_LEVELS | none, minimal, low, medium, high, xhigh | none | Claude (OpenRouter) |
Sources That Do NOT Work
These were investigated and confirmed to lack thinking level data:
- LiteLLM catalog — only
supports_reasoning: true/false - Google Gemini models API — only
thinking: true/false - OpenAI
/v1/modelsendpoint — minimal object with no capability fields
Slug Lookup Reference
Use both LiteLLM and models.dev when looking up slugs — they complement each other. LiteLLM gives you the exact slugs Kiln will use (since Kiln runs on LiteLLM), while models.dev often has broader coverage of newer or niche models with pricing, context limits, and capability details.
LiteLLM Model Catalog (https://api.litellm.ai/model_catalog)
100 free requests/day, no key needed. Supports server-side filtering: model= (substring match), provider=, mode=, supports_vision=true, supports_reasoning=true, page_size=500.
# Find all variants of a model across providers:
curl -s 'https://api.litellm.ai/model_catalog?model=MODEL_NAME&mode=chat&page_size=500' \
-H 'accept: application/json' | jq '.data[] | {id, provider, mode, max_input_tokens, supports_vision, supports_reasoning, supports_function_calling}'
# List all models for a provider:
curl -s 'https://api.litellm.ai/model_catalog?provider=PROVIDER&mode=chat&page_size=500' \
-H 'accept: application/json' | jq '.data[].id'
models.dev (https://models.dev/api.json)
Mega JSON covering 50+ providers with model IDs, pricing, context limits, capabilities, and release dates. Large file — always use curl+jq, never WebFetch.
# Search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"
# List all model IDs for a specific provider:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models | keys[]'
# Get full details for a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'
Other verified sources
- OpenRouter:
curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM" - Anthropic: https://docs.anthropic.com/en/api/models/list
- Cerebras: https://inference-docs.cerebras.ai/models/overview
Lagging Providers
Fireworks, Together, SiliconFlow and Cloudflare typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, always cross-check directly — both when adding a new model and when running the Phase 1B backfill check.
Fireworks AI — model pages are the most current source. WebFetch directly:
WebFetch https://fireworks.ai/models/fireworks/{model-slug}
Or browse the catalog at https://fireworks.ai/models. Kiln slug format: accounts/fireworks/models/{model-slug}.
Together AI — the /v1/models endpoint requires an API key. The key is set in the user's shell as TOGETHERAI_API_KEY (note the AI — this matches LiteLLM's together_ai env var; the plain TOGETHER_API_KEY is usually NOT set):
# List all Together model IDs matching a term:
curl -s https://api.together.xyz/v1/models \
-H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"
# Full record for a specific slug:
curl -s https://api.together.xyz/v1/models \
-H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | select(.id == "SLUG")'
If TOGETHERAI_API_KEY isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.
Featherless AI — appears in neither models.dev nor the LiteLLM catalog (the catalog has only two stale Qwerky entries), so /v1/models is the sole authoritative source. It needs no API key to enumerate.
The response is enormous — ~22,000 models, the vast majority community fine-tunes. Never WebFetch it and never dump it raw. Save it once, then filter with jq:
curl -s https://api.featherless.ai/v1/models -o /tmp/feath.json
jq '.data | length' /tmp/feath.json
Restrict to official vendor orgs, or you'll drown in forks like DavidAU/Gemma3-27B-it-vl-GLM-4.7-Uncensored-Heretic:
# newest official models, excluding gated ones (gated = unusable, see Provider Quirks)
jq -r '.data[]
| select(.is_gated | not)
| select(.id | test("^(deepseek-ai|zai-org|Qwen|moonshotai|MiniMaxAI|mistralai|openai|google|microsoft)/"))
| "\(.id)\tctx=\(.context_length)\t$\(.pricing.input)/\(.pricing.output)\ttools=\(.features.tool_use // false)\timg=\(.features.image_input // false)"' \
/tmp/feath.json | sort
Look up one model, including its capability flags and exact pricing:
jq '.data[] | select(.id == "zai-org/GLM-5.2")' /tmp/feath.json
The highest-precision way to find backfill candidates is to check whether a slug Kiln already ships for together_ai / siliconflow_cn exists verbatim here — those providers use the same HuggingFace repo-id convention, so a hit needs no guessing:
jq -r '.data[] | select(.is_gated | not) | .id' /tmp/feath.json | sort > /tmp/feath_ids.txt
grep -Fx "zai-org/GLM-5.2" /tmp/feath_ids.txt # exit 0 = safe to add
Slugs are case-sensitive (google/gemma-4-31B-it, not -31b-it). Always confirm is_gated is false and cross-check features.tool_use / features.image_input against the flags you set — don't claim capabilities the endpoint doesn't advertise.
Cloudflare Workers AI — the authoritative list is Cloudflare's public, unauthenticated catalog JSON. It is about 900 KB and can return a transient 502, so save it to a file (retry on 502) and filter with jq; never WebFetch it:
curl -s https://ai-cloudflare-com.pages.dev/api/models -o /tmp/cf_models.json
# Text-generation models, newest first, with Kiln-relevant flags
jq -r '.models[] | select(.task.name=="Text Generation")
| (reduce .properties[] as $p ({}; .[$p.property_id]=$p.value)) as $p
| [.name, .created_at[0:10], ($p.context_window//"-"),
(if $p.function_calling=="true" then "tools" else "" end),
(if $p.vision=="true" then "vision" else "" end),
(if $p.reasoning=="true" then "reasoning" else "" end),
(if $p.require_workers_paid=="true" then "paid" else "" end),
(if $p.lora=="true" then "lora" else "" end),
($p.planned_deprecation_date//"")] | @tsv' /tmp/cf_models.json | sort -t$'\t' -k2 -r
# Full record for one model (drop the large schema)
jq '.models[] | select(.name=="@cf/zai-org/glm-5.3") | del(.schema)' /tmp/cf_models.json
Model IDs are the name field (@cf/<author>/<model>). With credentials (CLOUDFLARE_API_KEY with Workers AI access, plus CLOUDFLARE_ACCOUNT_ID), the official API returns the same records; add &include_deprecated=true to see models retired in the last three months:
curl -s "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/models/search?task=Text%20Generation&per_page=100" \
-H "Authorization: Bearer $CLOUDFLARE_API_KEY" | jq -r '.result[].name'
- Cross-check: models.dev, provider key
cloudflare-workers-ai(curl -s https://models.dev/api.json | jq '.["cloudflare-workers-ai"].models | keys[]'). It syncs hourly and is good for spotting additions, but it no longer deletes retired models, so never use it to decide a model still exists. - Never use LiteLLM's catalog for Cloudflare. Its
cloudflare/entries are a one-time import from June 2026: they miss newer models and still list retired ones. - Fallback: the per-model JSON files in the
cloudflare/cloudflare-docsGitHub repo,src/content/workers-ai-models/*.json(same shape as the catalog). - Also scan the Workers AI changelog for launches in the last two weeks, in case the catalog lags.
See Cloudflare (Workers AI) in Provider Quirks for the inclusion rule and flags.
SiliconFlow — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:
WebFetch https://siliconflow.com/models
WebFetch https://siliconflow.com/models/{vendor}/{model}
When you find a new reliable slug source, append it here.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |