ahstn/oceans-llm / .agents/skills/verify-oceans-admin
verify-oceans-admin
VerifiedVerify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.
features/models.md
View on GitHub# Models
Models lets a signed-in user review configured model IDs, routing status, provider details, pricing, sourced benchmark scores, capabilities, and generated client configuration choices, 30 models per page. Platform administrators also see the `Allow List` column and access rules, and can refresh catalogues.
## Sub-features
- `models-open` opens Models from the Control Plane sidebar.
- `models-list` shows the configured model count and desktop or mobile list, including the always-visible Intelligence Index. A model with context pricing tiers lists its base rate with a `from` prefix in `Cost / 1M tokens`.
- `models-info` opens overview, routing, economics (base rates plus one `Above N tokens` section per context pricing tier), benchmarks, and (for platform admins) access details for one model.
- `models-columns` toggles the optional context-window and capability columns.
- `models-client-config` opens client configuration for a configurable model via the row `Config` button.
- `models-select-config` selects several configurable models with `Select model <id>` and opens one `Client config` dialog through `Generate config`.
- `models-pagination` pages 30 models at a time with `Previous` and `Next`.
- `models-litellm-info` serves the LiteLLM-compatible `GET /model_group/info` list to gateway API keys, so clients such as omp can use `discovery.type: litellm`.
## How to get to it (user POV)
- Sign in and choose `Models` under `Control Plane` in the sidebar.
- Open `/admin/models`; an unauthenticated user is sent to `/admin/login` first.
## Driving it with control-oceans-admin
Preconditions:
- Sign in as the seeded `admin@local` platform administrator so the `Access` model information section is available.
- The recorded stack is healthy and contains the `gpt-6-astra` demo model from `gateway.yaml`.
- The browser viewport is at least 768 pixels wide for the desktop table.
- `control-oceans-admin doctor` passes.
- **Automated proof.** Run `control-oceans-admin drive models`. It ensures Playwright Chromium is installed, signs in, follows the `Models` link, reads the `Model pagination` range (`first–last of total`, next to `Rows per page`), checks the displayed range size against rendered rows and the total against `/api/v1/admin/models?page=1&page_size=100&include_aliases=false` (the same direct-model view the page loads), checks the always-visible Intelligence Index against that API response, checks all four platform-admin information sections (`Overview`, including `Pricing and limits`, `Routing`, `Benchmarks`, and `Access`), compares the benchmark state with that API response, checks the Artificial Analysis attribution below the model list, enables the optional Context window and Capabilities columns, and opens client configuration for `gpt-6-astra`.
- **Known model.** Locate `models-desktop-cell-gpt-6-astra`. The cell shows `gpt-6-astra` and a status indicator.
- **Model detail.** In the same table row, choose `Info`. A dialog headed `Model info` contains `gpt-6-astra` and navigation named `Model info sections`.
- **Pricing tiers.** The `gpt-6-astra` row's `Cost / 1M tokens` cell reads `Input from $10.00` and `Output from $50.00`. In its `Model info` dialog, `Pricing and limits` shows the base `Input cost` `$10.00` and `Output cost` `$50.00`, then the sentence `Tier rates apply to the whole request when the prompt, including cached tokens, is larger than the threshold.` and a region named `Above 272k tokens` with `Input cost` `$20.00`, `Output cost` `$75.00`, `Cache read` `$2.00`, and `Cache write` `$25.00`. These values come from the bundled catalog; a runtime refresh can change them, so compare against `pricing_tiers` in the `/api/v1/admin/models` row rather than the literals when they differ. A model without tiers shows no `from` prefix, no `Above` section, and no whole-request sentence.
- **Columns.** Intelligence Index is always visible. Choose `Columns`, then select `Context window` or `Capabilities`. The matching optional table header becomes visible without a route change.
- **Service tiers.** In the `gpt-6-luna` `Model info` dialog, `Overview` has a `Service tiers` row with `Standard (default)`, `Flex`, `Fast (priority)` and `Ultrafast` badges. A model that serves only the default tier reads `Default only`, and a model with no chat route reads `No available service tiers`. A model with an allow-list adds `Callers may request …`: in the `deepseek-v4.1-flash` dialog, the row reads `Callers may request Flex.` `Pricing and limits` shows per-tier rates in `model-service-tier-pricing` when the catalog has them. Then close the dialog and open `Model info` for `gpt-6-luna-fast`. In `Routing`, its route summary has a `Fast (priority)` badge. Expand it (`Details for …, route 1`); its `Service tier` row reads `Fast (priority) on every request`. Routes of other models read `The tier the caller requests`. Compare with `service_tiers`, `available_service_tiers` and `pricing_service_tiers` in `/api/v1/admin/models`. The [Service tiers](./service-tiers.md) driver checks the allow-list and forced-route rows.
- **Client configuration.** Choose the button named `Generate client config for gpt-6-astra`. A dialog headed `Client config` appears. Do not treat generated configuration as live-provider proof.
- **Claude Code configuration.** Select a Claude model and open the `Claude Code` tab. The `Gateway model settings` block uses `ANTHROPIC_DEFAULT_HAIKU_MODEL` for background work, not the deprecated `ANTHROPIC_SMALL_FAST_MODEL`. A compatible selected Haiku takes priority; otherwise it uses the first selected model that can turn thinking off. An all-always-thinking selection shows a warning. Setup points to user settings or an ignored `.claude/settings.local.json`, not the shared project file. The optional `Lower token usage settings` block keeps `ENABLE_TOOL_SEARCH=auto` and does not disable experimental betas or change fullscreen rendering. Compare the copied JSON with the client-config API response.
- **Decisions-only model.** The `jev` row shows `—` in its Config cell, and `Select model jev` is disabled.
- **LiteLLM model info.** Create a temporary user API key through [API keys](./api-keys.md) and keep it in memory. `GET /model_group/info` with `Authorization: Bearer <key>` returns `200`. The set of `data[].model_group` IDs equals the set of `data[].id` from `GET /v1/models` with the same key. Every row has a non-empty lowercase `providers` array, `mode` (`chat` or `embedding`), a boolean `supports_vision`, a boolean `supports_function_calling`, and a `supported_openai_params` array. A priced row has `input_cost_per_token` and `output_cost_per_token`, plus `cache_read_input_token_cost` and `cache_creation_input_token_cost` when the cache prices are known; an unpriced row has none of them. The response must not contain a provider base URL or credential. This check makes no provider call. `drive models` does not run it, so report it beside the driver proof. Revoke the key afterwards.
- **Proof.** Retain the Models screenshots and ARIA snapshots, including `03-model-benchmarks` (plus one `05-model-client-config-<key>` pair per client config), and `models-proof.json` in the run evidence directory.
## Gotchas
- The mobile list replaces `models-desktop-table` below the `md` breakpoint.
- Startup requires `env.*` credential references to be configured and present, but it does not validate credential correctness or upstream connectivity. A healthy Models page does not prove upstream access.
- `Refresh pricing` can call the pricing refresh boundary and is outside the read-only baseline. It refreshes models.dev, OpenRouter, and LiteLLM and shows one toast: `Pricing refreshed: models.dev <outcome>, OpenRouter <outcome>, LiteLLM <outcome>` (each outcome `updated` or `unchanged`) when every source succeeds, or the warning `Pricing refresh failed for <sources>; <remaining summary>` when any source fails. `Pricing refreshed, but the model list did not reload` is an error toast for a failed list reload after a successful refresh.
- Some aliases and decisions-only models such as `jev` have no client configuration. Use `gpt-6-astra` for the baseline detail proof.
- Demo model IDs follow `gateway.yaml`. When a proof times out waiting for a model cell, check whether the model was renamed there before suspecting the UI.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |