cogni-dao/cogni / .claude/skills/openrouter-api-key-expert
openrouter-api-key-expert
VerifiedEnd-to-end OpenRouter key + spend + outage playbook for Cogni. Use when investigating unexplained OpenRouter spend ('who used opus/model X?', 'why is my bill high'), a suspected leaked/compromised API key, a prod AI outage that surfaces as `provider_unavailable` / `402` / LiteLLM `Insufficient credits`, or when you must rotate/cap/kill an OpenRouter key across the fleet. Encodes the diagnostic decision tree (provider_unavailable = upstream, truth in `service=litellm` Loki; 402 = account balance dry; a model NOT in the deployed LiteLLM catalog can't come from Cogni infra), the attribution method that needs a MANAGEMENT key (/api/v1/{keys,activity,credits,key}), the rule-out-local sweep, the LiteLLM-is-Compose-on-VM + OpenBao-SSOT converge facts, the full break-glass rotation runbook (mint capped → OpenBao fleet-wide → deploy-infra converge → BEHAVIOR-verify → delete), and the CARDINAL rule: never leave a compromised key alive/uncapped — cap-or-disable on suspicion, delete the moment prod is verified on the replacement. Triggers: 'openrouter', 'who used opus', 'opus-4.8 spend', 'unexplained openrouter charges', 'leaked api key', 'compromised key', 'rotate openrouter key', 'provider_unavailable', '402 insufficient credits', 'prod AI down', 'litellm out of credits', 'openrouter activity', 'management key', 'cap the key', 'kill the leaked key', 'openrouter attribution', 'which key does prod use'.
OpenRouter API Key Expert
Route the work through operator-architect first; this skill adds the OpenRouter key and spend rules on top of it.
The playbook for OpenRouter spend investigation, leak/compromise response, outage diagnosis, and fleet key rotation. Battle-tested in the 2026-08-27→31 incident (leaked prod key cogni-template-prod-2 drained the account twice via direct off-infra opus-4.8 abuse → two prod AI outages). Read this BEFORE touching a key.
0. Prereq: you need a MANAGEMENT key (mint it, use it, revoke it)
- Inference and provisioning keys cannot read account attribution —
/api/v1/activityand/api/v1/keysreturn403 "Only management keys can fetch activity for an account". You need a management key. - The account owner (a human with OpenRouter dashboard access) mints one at https://openrouter.ai/settings/keys, hands it to you to save in
.env.localasOPENROUTER_MANAGEMENT_KEY=(gitignored — verifygit check-ignore .env.local), and revokes it the moment the job is done. It is ephemeral break-glass custody, not a standing credential — treat it like a root token, never commit or echo it. - Everything below that says "(mgmt)" needs it. Read it inline:
MK=$(grep -E '^OPENROUTER_MANAGEMENT_KEY=' .env.local | cut -d= -f2- | tr -d '"'"'"' \r').
1. Outage decision tree — provider_unavailable / 402
provider_unavailable is the app correctly reporting an upstream LLM failure (bug.5056 fix — FAULT_PARTY_BEFORE_BUCKET). It is NOT an app bug. The real cause lives in the litellm service logs, not the app/proxy:
scripts/loki-query.sh '{env="production",service="litellm"} |~ "(?i)(error|402|401|429|insufficient|exception)"' 20 40
402 "Insufficient credits"onopenrouter.ai/api/v1/chat/completions→ the OpenRouter ACCOUNT balance is $0. Fix is billing, not code/deploy: add credits at https://openrouter.ai/settings/credits → recovers instantly, LiteLLM picks it up on the next request (no restart). Confirm withcurl /api/v1/credits (mgmt)→total_usage ≈ total_credits. This is a per-account 402, so it 402s every key at once.401 "User not found"→ the key is bad/deleted/disabled — auth, not balance. Top-up won't help; fix the key.429→ a per-billing-account quota block at the Cogni tenant layer (a node/agent billing account out of Cogni credits), hit before the request reaches OpenRouter — not the OpenRouter key or account. Don't chase it as a key problem.
App-side symptom (masked, for reference): route="graphs.run.internal" error="provider_unavailable", errorCode="PROVIDER_UNAVAILABLE", LlmError ... status:402. See [[project_ai_outage_openrouter_402_masked]].
2. Spend attribution — "who used model X?"
Cardinal filter first: compare the model against the currently-deployed LiteLLM catalog (infra/compose/runtime/configs/litellm.config.yaml — read it live, the roster changes). If the model is not in that catalog (in the incident, opus-4.8 while the catalog topped at opus-4.5), it cannot have come from any Cogni path — LiteLLM would reject it, and there will be zero hits in {service="app"|"litellm"} Loki. That model reached OpenRouter directly, off-infra (leaked key or a local tool — see §3).
Then attribute with the management key:
curl -s https://openrouter.ai/api/v1/key -H "Authorization: Bearer $MK" # THIS key: usage, usage_daily/weekly/monthly, limit, creator_user_id, is_management_key
curl -s https://openrouter.ai/api/v1/credits -H "Authorization: Bearer $MK" # account: total_credits, total_usage
curl -s https://openrouter.ai/api/v1/keys -H "Authorization: Bearer $MK" # (mgmt) full roster: name, hash, per-key usage, disabled, limit
curl -s "https://openrouter.ai/api/v1/activity" -H "Authorization: Bearer $MK" # (mgmt) per-DAY per-MODEL per-provider usage, ~30d window
- Account-accounting by diff: sum every key's lifetime
usage(from/keys) and compare to account/credits.total_usage. A gap = deleted/rotated keys (their spend persists in the account total) — gap alone is NOT proof of a live leak. - Reconciliation (the money shot):
/activityis account-wide (per model/provider/date, no per-key field). Match its window total against each key'susage_monthlyfrom/keys. In the incident, activity-window total$37.88== the prod key'susage_monthlywhile all other keys were$0monthly → 100% of recent account spend was the prod key, of which opus-4.8 was 95%. - HARD LIMIT — source IP/DNS is NOT in the API.
/activityhas no origin/IP/app field;/api/v1/generation?id=gives per-request origin but detail is purged after ~24h (returns null). The only place a caller's "app"/referer survives is the dashboard Activity page — and even that is at most an app tag, never a raw IP. Say this plainly; don't promise an IP the API can't give.
3. Rule out a LOCAL caller (before concluding "external leak")
The key value lives in .env.production, OpenBao, the prod cluster, and wherever a human/tool copied it. Sweep the machine (report paths only, never the key):
PRODKEY=<value>
grep -rlF "$PRODKEY" /Users/*/conductor/workspaces /Users/*/dev ~/.config ~/.zshrc ~/.zshenv 2>/dev/null # Cursor file-History + deploy artifacts are common hits
grep -riE 'ANTHROPIC_BASE_URL|OPENAI_BASE_URL|OPENROUTER' ~/.zshrc ~/.claude/settings.json ~/.config 2>/dev/null # coding-agent base-url overrides → OpenRouter
Coding agents (Cursor/opencode/aider/cline) can be pointed at OpenRouter with any model — a whole-repo session shows as a huge single-day prompt-token burst (incident Aug-20: 9 reqs, 5M prompt tokens, $26.85). If nothing local matches AND no Cogni path can emit the model → it's an external leak of the key value.
4. Architecture facts you must know before rotating
- LiteLLM is Compose-on-VM, not a pod. Prod svc
operator-litellm-externalis anExternalName→cogni.vm.cognidao.org:4000. The OpenRouter call is made by the LiteLLM container readingOPENROUTER_API_KEYfrom its VM.env. ESO/OpenBao pod-side sync alone does not move the LLM path. - On ESO-established envs,
deploy-infrarenders the VM.envOPENROUTER from OpenBao, NOT the GitHub Actions secret.scripts/ci/deploy-infra.sh(OPENBAO_RUNTIME_SSOT=truewhenoperator-env-secretsexists) →source_openbao_runtime_key required OPENROUTER_API_KEY operator .... The GH Actionssecrets.OPENROUTER_API_KEYis only for bootstrap/non-ESO envs. So OpenBaocogni/<env>/operatoris the SSOT for the VM LiteLLM. - The key is shared fleet-wide via catalog
inheritFrom: operator— operator's canonical value fans to EVERY inheriting node — derive the node set —for f in infra/catalog/*.yaml; do grep -q '^type: node' $f && basename $f .yaml; done— never a hand-typed list (Doltoperator-node-catalog: "the roster is LIVE STATE"). Deleting it breaks the whole fleet's AI, not one node. Converge all before deleting. - Behavior-test which key prod uses by exercising LiteLLM from an app pod (needs account credits or you'll only see 402):
POD=$(kubectl -n cogni-production get pods | grep 'operator-node-app.*Running' | head -1 | awk '{print $1}') kubectl -n cogni-production exec $POD -c app -- sh -c \ 'curl -s "$LITELLM_BASE_URL/v1/chat/completions" -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ -H content-type:application/json -d "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"user\",\"content\":\"say ok\"}],\"max_tokens\":180}"' # then read /api/v1/key usage on old vs new key — whichever's usage MOVES is the key prod uses.
5. Rotation runbook
Clean path (preferred, per https://cognidao.org/knowledge/break-glass-secret-write): self-serve API POST /api/v1/nodes/<id>/secrets {env,key:"OPENROUTER_API_KEY",value,op:"rotate"} with COGNI_API_KEY_<ENV>. A 401 = the key is expired → STOP and ask a human to refresh it; do NOT fall back to spelunking. (In the incident COGNI_API_KEY_PROD was expired, forcing break-glass.)
Break-glass (mgmt key + prod OpenBao root + kubeconfig from ~/dev/cogni-template/.local/provision-creds/production/):
- Mint the new key WITH A CAP:
POST /api/v1/keys {name:"cogni-template-prod-N", limit:50}(mgmt). Validate:GET /api/v1/keyon it → 200. - Write it to the SSOT fleet-wide:
bao kv patch cogni/production/<n> OPENROUTER_API_KEY=-for everytype: nodecatalog row — derive the node set —for f in infra/catalog/*.yaml; do grep -q '^type: node' $f && basename $f .yaml; done— never a hand-typed list (Doltoperator-node-catalog: "the roster is LIVE STATE"). A hand-typed list here is an OUTAGE: this skill previously named 4 of 7 nodes, so a rotation following it would have left the rest holding a dead key. (viakubectl -n openbao exec openbao-0). Force-sync ESO:kubectl annotate externalsecret -n cogni-production <n>-env-secrets force-sync=$(date +%s) --overwrite. - Converge the VM LiteLLM (the ONLY way to move the actual OpenRouter caller): dispatch
promote-and-deploy.ymlfor prod withskip_infra=false deploy_infra_mode=full, pinned to the CURRENT prod sha (curl https://cognidao.org/version→source_sha+build_sha) so the app does NOT roll. Note the input isdeploy_infra_modeon main (notinfra_mode). It runs on the VM self-hosted runner (why laptop SSH is a dead end). Adeploy-infra"failure" is usually a premature LiteLLM readiness timeout during container recreate — verify in Loki that LiteLLM actually came back (Started server process→readiness 200) before treating it as a real failure. - BEHAVIOR-verify prod is on the new key: fire a completion (§4) and confirm the new key's
usagemoves and the old key's does NOT. Deploy-proof ≠ function. - Delete the old key (mgmt
DELETE /api/v1/keys/{hash}) only after step 4 passes.
6. CARDINAL cleanup rule (the lesson that cost a weekend)
Never leave a compromised/leaked key alive or uncapped "temporarily." The moment you suspect compromise:
- Immediately
PATCH {disabled:true}(reversible) or cap to ~$1/day. Do NOT "raise the cap so users can use" on a leaked key — mint the new key for that. - Finish the rotation in ONE session. A disabled-but-not-deleted or (worse) uncapped leaked key WILL be abused. In this incident the leaked key was left uncapped over a weekend → external opus abuse re-drained the account → second prod outage. Half-done is worse than not-started.
- Cap the live prod key at a generous monthly limit (
{limit:50,limit_reset:"monthly"}, ~25× MVP usage) as account-insurance — OpenRouter has no per-key model allowlist, so a spend cap is the only bound against a future leak draining the whole balance. Bump it when onboarding a paying user. - Purge stale copies (Cursor
~/Library/Application Support/Cursor/User/History/*.production,.contextdeploy artifacts) — though a deleted key value is already inert.
Related
- [[project_ai_outage_openrouter_402_masked]] — the 402-masking +
FAULT_PARTY_BEFORE_BUCKETfix (bug.5056). - [[reference_openrouter_preview_rotation]] — the LiteLLM-Compose-on-VM
.envconverge recipe (preview; same shape for prod). https://cognidao.org/knowledge/break-glass-secret-write— the clean self-serve rotation lane + the401 = STOPrule.scripts/ci/deploy-infra.sh(source_openbao_runtime_key) — proof the VM.envrenders OPENROUTER from OpenBao SSOT..claude/skills/cicd-secrets-expert/— OpenBao/ESO custody +pnpm secrets:setlanes.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |