ahstn/oceans-llm / .agents/skills/verify-oceans-admin
verify-oceans-admin
VerifiedVerify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.
features/live-llm-requests.md
View on GitHub# Live LLM requests
Live LLM requests prove that a changed request path reaches a real upstream provider and returns through the gateway. They are paid integration checks, not part of the read-only admin baseline.
## Sub-features
- `live-openrouter-generic` sends one short Chat Completions request through the preferred generic OpenRouter route.
- `live-anthropic-messages` sends one short Anthropic Messages request to `/v1/messages` for Messages-format changes.
- Thinking, display, effort, native metadata and general Messages regressions additionally follow [Anthropic Messages](./anthropic-messages.md); its offline matrix is required even when the public live smoke passes.
- `live-messages-tool-loop` sends a two-turn Anthropic Messages tool exchange to `/v1/messages` for changes to Messages tool translation.
- `live-bedrock-mantle` sends one short Responses request through Bedrock Mantle for Bedrock-specific changes.
- `live-decisions` sends one OpenAI-format request to `/v1/decisions` and one System One request to `/v1/systemone` through the `jev` alpha route for Decisions changes.
- `live-count-tokens` counts a short Messages prompt through `/v1/messages/count_tokens` and confirms that a route without an upstream count API returns `501 not_implemented_error`.
- `live-request-log` confirms the selected provider, route, upstream model, status, and usage in the admin request log.
- `live-key-cleanup` revokes the temporary gateway key used by the canary.
## How to get to it (user POV)
- Sign in, choose `API Keys`, and create a unique temporary key with access only to the selected canary model.
- For a generic request-path change, call `/v1/chat/completions` with gateway model `deepseek-v4.1-flash`.
- For an Anthropic Messages change, call `/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01` and gateway model `claude-haiku-5.5`.
- For a Bedrock-specific change, call `/v1/responses` with gateway model `gpt-oss-120b-bedrock`.
- For a Decisions change, call `/v1/decisions` (OpenAI format, `input`) and `/v1/systemone` (System One format, `state`) with gateway model `jev`.
- For a token-counting change, call `/v1/messages/count_tokens` or `/messages/count_tokens` with `x-api-key` and gateway model `claude-sonnet-4.6` (Bedrock Runtime Anthropic invoke). Call it again with `deepseek-v4.1-flash` to see the 501 path.
- For every canary except a token count, choose `Request Logs` and open the matching request after the call completes. A token count creates no request log, so skip this step for it.
## Driving it with control-oceans-admin
Preconditions:
- `control-oceans-admin doctor` passes.
- The task affects behavior that needs real upstream proof. UI-only, documentation-only, seed-only, and unrelated configuration changes do not qualify.
- `OPENROUTER_API_KEY` resolves through fnox for the launched stack (see Launch in [SKILL.md](../SKILL.md)) for the generic and Messages canaries, or the AWS default credential chain can call Bedrock Mantle for a Bedrock-specific canary. A Decisions canary needs a working credential for the `jev` alpha route.
- The operator accepts one short paid request. A Decisions canary makes two paid calls, one per format, and the operator must accept both. Its mixed-format 400 check is free. Use synthetic input and do not include repository content, user data, or secrets.
- Create a unique temporary gateway API key through the API Keys recipe with exactly the selected model grant and only the `inference` capability. Keep it in memory (or only the current shell as `OCEANS_VERIFY_API_KEY`). Do not write it to evidence or command history.
- **Select one provider.** Prefer OpenRouter unless the change is Bedrock-specific. Do not call both providers only to increase test count.
- **Token cap.** Every canary request sets `max_tokens` (or `max_output_tokens`) to 4096 or less. Use the smallest value that lets the model finish: reasoning models spend part of the budget before any output, and a truncated answer or empty tool arguments prove nothing.
- **OpenRouter request.** Send one non-streaming request with `model: deepseek-v4.1-flash`, a short synthetic prompt, and `max_tokens` no greater than 4096. Expect a successful Chat Completions response.
- **Anthropic Messages request.** For Messages-format changes only, send one request to `/v1/messages` with `model: claude-haiku-5.5`, a short synthetic prompt, and `max_tokens` no greater than 4096. Expect a `message` response with `usage.input_tokens` and `usage.output_tokens`. Add `stream: true` for Messages streaming work; the stream should open with `message_start` and end with `message_stop`.
- **Messages tool loop.** For changes to Messages tool translation, send turn 1 to `/v1/messages` with one deterministic tool (for example `get_weather` with `city` limited to `Paris`) and `tool_choice: {"type": "any"}`. Expect `stop_reason: tool_use`. Send turn 2 with the same `tools`, `tool_choice` omitted (or `{"type": "auto"}`, because a forced choice makes the model call the tool again), the original user message, the assistant `content` from turn 1, and a user turn that holds one `tool_result` per `tool_use` id. Give the result block-array content with a synthetic code the model cannot guess. Expect `stop_reason: end_turn`, no `tool_use` block, and text that repeats the code. Repeat turn 2 with `stream: true` when streaming is in scope. Choose `deepseek-v4.1-flash` or `claude-haiku-5.5`, which both reach OpenRouter through `openai_compat`; use both only when cross-model parity is explicitly in scope and the extra calls are authorized. Record it with `--file messages-tool-loop-canary-proof.json`. Do not record the prompt, the code, or response text.
- **Bedrock request.** Send one non-streaming request with `model: gpt-oss-120b-bedrock`, a short synthetic input, `max_output_tokens` no greater than 4096, and `store: false`. Expect a successful Responses response.
- **Decisions requests.** For Decisions changes only, send one OpenAI-format body to `/v1/decisions`. Use `model: jev`, a short synthetic `input`, and one named `predicate` question. Expect `answers[0].type == "predicate"` and integer `usage.input_tokens_details.cached_tokens`. Then send one System One body to `/v1/systemone` with a synthetic `state` and one `noul` question. Expect a map of `answers` and `usage.input_tokens`. Also send one body that contains both `input` and `state` to `/v1/decisions`, and confirm it returns 400 without a request log or provider attempt. This last call is free.
- **Changed behavior.** Add only the smallest option needed to exercise the changed path, such as `stream: true` for streaming work or one deterministic function tool for tool translation. Do not broaden the prompt.
- **Token count request.** Send one Messages body with a short synthetic prompt to `/v1/messages/count_tokens` using model `claude-sonnet-4.6`. Expect HTTP 200 and `{"input_tokens": N}` with `N` greater than zero. Send the same body with model `deepseek-v4.1-flash`. Expect HTTP 501, `error.type: not_implemented_error`, and a message naming the route's provider key and upstream model. Neither call adds a Request Logs row. Record one `--check` each for the count, the 501 envelope, and the unchanged request-log list.
- **Request-log proof.** For canaries that create a request log, which excludes token counts, locate the canary in `Request Logs` by time and gateway model. Confirm success, the expected provider and configured upstream model, and non-zero usage when the provider supplies it. Capture the list and detail states with bounded, sanitized payload evidence and credentials redacted. Record it with `control-oceans-admin record live-llm-requests --file openrouter-canary-proof.json` (or `bedrock-canary-proof.json`). Use one `--check` each for the response status, provider and upstream model, usage, and request-log match, with the gateway model and request-log ID in the notes, plus the captured list and detail screenshots as `--artifact`. Do not include prompts, responses, or credentials.
- **Cleanup.** Revoke the temporary gateway API key and confirm it can no longer call `/v1/models`. Then run the normal stack cleanup and `control-oceans-admin evidence live-llm`.
## Gotchas
- The Messages tool-loop procedure is an expanded paid proof: two calls per selected model, or three with the streaming continuation. Run one relevant model unless cross-model parity is explicitly in scope. The default single-canary authorization does not cover this expansion; count and authorize the calls before running it.
- For live Request Logs screenshots, mask `request-log-payloads`; omit the same section from ARIA/text artifacts. Capture route, metrics and attempts without copying complete admin detail responses.
- A configured model, present credential, healthy gateway, or generated client configuration does not prove a live provider call.
- OpenRouter is the preferred low-cost generic path. `deepseek/deepseek-v4.1-flash` is priced from the pricing catalog (about $0.016 input and $0.89 output per million tokens when this was written), but OpenRouter can route among upstream hosts. The older `deepseek-v4-flash-0731` route stays configured; use it only to compare against an earlier run.
- `claude-haiku-5.5` reaches `anthropic/claude-haiku-5.5` through OpenRouter's native Messages endpoint. Its current price comes from the catalog. This proves the native OpenRouter path; Vertex and Bedrock still need their own integration checks when platform-specific behavior changes. For rolling-alias regressions, follow the run-local alias scenario in [Anthropic Messages](./anthropic-messages.md).
- Bedrock Mantle uses model ID `openai.gpt-oss-120b` and the `/v1/responses` path. The configured standard rate in `us-east-1` is $0.15 input and $0.60 output per million tokens. The Bedrock Runtime route (`gpt-oss-120b`, `runtime_openai_chat`, upstream `openai.gpt-oss-120b-1:0`) is a separate Chat Completions path that does not accept `/v1/responses`. Do not use it for Mantle proof. Use it with `/v1/chat/completions` when the Bedrock Runtime OpenAI-chat path is under test. Bedrock Runtime rejects the short ID `gpt-oss-120b` with "The provided model identifier is invalid".
- `/v1/decisions` and `/v1/systemone` (model `jev`, upstream `typesafe/jev-1.13` on the alpha route) are a separate paid path. The route speaks System One, so an OpenAI-format call proves translation in both directions. Request logs show operation `Decisions` and store the caller-format response. Decisions skips inference guardrails. Chat and Responses canaries do not prove it.
- No local route speaks the OpenAI Decisions format upstream (`compatibility.openai_compat.decisions_api: openai`). The provider round-trip tests cover that path. Do not add an OpenAI route only for a canary.
- Token counting is never logged as a request. Prove it from the HTTP response and the `anthropic count_tokens request completed` gateway log event, not from Request Logs or usage totals.
- The seeded `claude-sonnet-4.6` route uses the cross-Region profile `us.anthropic.claude-sonnet-4-6`, so its count is sent to `anthropic.claude-sonnet-4-6`. A Bedrock `ValidationException` on a count is an upstream rejection, not a gateway failure. It has two common causes: the foundation model has no Runtime `CountTokens` support in the provider's Region, or the route uses an ID Bedrock does not publish. Check the model card's token-counting support for that Region before changing the ID; if `/v1/messages` succeeds on the same route, the ID is valid and the model is not countable there.
- Bedrock `CountTokens` accepts foundation model IDs only. The gateway strips a cross-Region profile prefix such as `us.`, including inside a system-defined `inference-profile/` ARN; an application inference profile ARN is rejected upstream.
- The unversioned aliases (`/chat/completions`, `/responses`, `/embeddings`, `/images`, `/images/generations`, `/decisions`, `/models`) reuse the `/v1` handlers. A canary on an alias proves routing only, not provider behavior.
- Image generation (`/v1/images` and chat image output) has its own paid recipe: [Image generation](./image-generation.md).
- Service tier changes (`service_tier`, `speed`, tier pricing, `-fast` routes) have their own paid A/B recipe: [Service tiers](./service-tiers.md). A generic canary does not prove the tier reached the provider.
- Bedrock stored responses default to retention. Keep `store: false` unless storage behavior is the test target.
- A provider outage, missing model access, expired credential, quota, or network block is an integration failure with a specific cause. It does not invalidate separate local UI proof.
- Never record raw API keys, authorization headers, full environment dumps, or provider response content that can contain submitted data.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |