Skip to content

ahstn/oceans-llm / .agents/skills/verify-oceans-admin

verify-oceans-admin

Verified

Verify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.

FileKindSize
SKILL.mdskill md29.0 KB
examples/mcp-candidates.jsonconfig3.0 KB
features/README.mdmarkdown10.0 KB
features/account-onboarding.mdmarkdown4.3 KB
features/agent-harnesses.mdmarkdown4.3 KB
features/agent-plugins.mdmarkdown1005 B
features/agent-sessions.mdmarkdown3.9 KB
features/anthropic-messages.mdmarkdown20.7 KB
features/api-keys.mdmarkdown6.5 KB
features/attribution.mdmarkdown13.9 KB
features/audit-log.mdmarkdown4.9 KB
features/backend-gateway.mdmarkdown5.2 KB
features/batch-requests.mdmarkdown4.2 KB
features/design-system.mdmarkdown4.6 KB
features/guardrails.mdmarkdown7.0 KB
features/harness-integration.mdmarkdown7.8 KB
features/identity.mdmarkdown4.2 KB
features/image-generation.mdmarkdown7.3 KB
features/leaderboard.mdmarkdown3.3 KB
features/live-llm-requests.mdmarkdown12.6 KB
features/mcp-client-oauth.mdmarkdown9.9 KB
features/mcp-ownership.mdmarkdown7.4 KB
features/mcp.mdmarkdown19.6 KB
features/models.mdmarkdown8.2 KB
features/password-sign-in.mdmarkdown8.3 KB
features/profile.mdmarkdown2.5 KB
features/rate-limits.mdmarkdown6.0 KB
features/request-logs.mdmarkdown8.6 KB
features/review-agent.mdmarkdown3.7 KB
features/routing.jsonconfig33.6 KB
features/service-tiers.mdmarkdown6.3 KB
features/skills.mdmarkdown9.8 KB
features/spend-controls.mdmarkdown7.3 KB
features/usage-costs.mdmarkdown3.7 KB
features/workspace-connections.mdmarkdown6.1 KB
scripts/check-messages.mjsexecutablescript2.4 KB
scripts/check-messages.test.mjsscript3.0 KB
scripts/control-oceans-adminexecutablescript25.3 KB
scripts/control-oceans-admin.test.mjsscript7.6 KB
scripts/drive-backend-gateway.mjsexecutablescript12.5 KB
scripts/drive-harness-integration.mjsscript6.6 KB
scripts/drive-mcp.mjsscript13.0 KB
scripts/drive-models.mjsscript12.9 KB
scripts/drive-observability.mjsscript16.8 KB
scripts/drive-profile.mjsscript8.2 KB
scripts/drive-service-tiers.mjsexecutablescript20.1 KB
scripts/drive-skills.mjsscript27.2 KB
scripts/mcp-browser.mjsscript26.5 KB
scripts/mcp-canary.mjsscript18.8 KB
scripts/mcp-candidates.mjsscript3.5 KB
scripts/mcp-verification.test.mjsscript10.2 KB
scripts/messages-regressions.mjsscript15.8 KB
scripts/plan-verification.mjsscript5.2 KB
scripts/plan-verification.test.mjsscript6.1 KB
scripts/proof-cli.mjsscript11.1 KB
scripts/proof.mjsscript9.2 KB
scripts/proof.test.mjsscript10.4 KB

features/models.md

View on GitHub
# Models

Models lets a signed-in user review configured model IDs, routing status, provider details, pricing, sourced benchmark scores, capabilities, and generated client configuration choices, 30 models per page. Platform administrators also see the `Allow List` column and access rules, and can refresh catalogues.

## Sub-features

- `models-open` opens Models from the Control Plane sidebar.
- `models-list` shows the configured model count and desktop or mobile list, including the always-visible Intelligence Index. A model with context pricing tiers lists its base rate with a `from` prefix in `Cost / 1M tokens`.
- `models-info` opens overview, routing, economics (base rates plus one `Above N tokens` section per context pricing tier), benchmarks, and (for platform admins) access details for one model.
- `models-columns` toggles the optional context-window and capability columns.
- `models-client-config` opens client configuration for a configurable model via the row `Config` button.
- `models-select-config` selects several configurable models with `Select model <id>` and opens one `Client config` dialog through `Generate config`.
- `models-pagination` pages 30 models at a time with `Previous` and `Next`.
- `models-litellm-info` serves the LiteLLM-compatible `GET /model_group/info` list to gateway API keys, so clients such as omp can use `discovery.type: litellm`.

## How to get to it (user POV)

- Sign in and choose `Models` under `Control Plane` in the sidebar.
- Open `/admin/models`; an unauthenticated user is sent to `/admin/login` first.

## Driving it with control-oceans-admin

Preconditions:

- Sign in as the seeded `admin@local` platform administrator so the `Access` model information section is available.
- The recorded stack is healthy and contains the `gpt-6-astra` demo model from `gateway.yaml`.
- The browser viewport is at least 768 pixels wide for the desktop table.
- `control-oceans-admin doctor` passes.

- **Automated proof.** Run `control-oceans-admin drive models`. It ensures Playwright Chromium is installed, signs in, follows the `Models` link, reads the `Model pagination` range (`first–last of total`, next to `Rows per page`), checks the displayed range size against rendered rows and the total against `/api/v1/admin/models?page=1&page_size=100&include_aliases=false` (the same direct-model view the page loads), checks the always-visible Intelligence Index against that API response, checks all four platform-admin information sections (`Overview`, including `Pricing and limits`, `Routing`, `Benchmarks`, and `Access`), compares the benchmark state with that API response, checks the Artificial Analysis attribution below the model list, enables the optional Context window and Capabilities columns, and opens client configuration for `gpt-6-astra`.
- **Known model.** Locate `models-desktop-cell-gpt-6-astra`. The cell shows `gpt-6-astra` and a status indicator.
- **Model detail.** In the same table row, choose `Info`. A dialog headed `Model info` contains `gpt-6-astra` and navigation named `Model info sections`.
- **Pricing tiers.** The `gpt-6-astra` row's `Cost / 1M tokens` cell reads `Input from $10.00` and `Output from $50.00`. In its `Model info` dialog, `Pricing and limits` shows the base `Input cost` `$10.00` and `Output cost` `$50.00`, then the sentence `Tier rates apply to the whole request when the prompt, including cached tokens, is larger than the threshold.` and a region named `Above 272k tokens` with `Input cost` `$20.00`, `Output cost` `$75.00`, `Cache read` `$2.00`, and `Cache write` `$25.00`. These values come from the bundled catalog; a runtime refresh can change them, so compare against `pricing_tiers` in the `/api/v1/admin/models` row rather than the literals when they differ. A model without tiers shows no `from` prefix, no `Above` section, and no whole-request sentence.
- **Columns.** Intelligence Index is always visible. Choose `Columns`, then select `Context window` or `Capabilities`. The matching optional table header becomes visible without a route change.
- **Service tiers.** In the `gpt-6-luna` `Model info` dialog, `Overview` has a `Service tiers` row with `Standard (default)`, `Flex`, `Fast (priority)` and `Ultrafast` badges. A model that serves only the default tier reads `Default only`, and a model with no chat route reads `No available service tiers`. A model with an allow-list adds `Callers may request …`: in the `deepseek-v4.1-flash` dialog, the row reads `Callers may request Flex.` `Pricing and limits` shows per-tier rates in `model-service-tier-pricing` when the catalog has them. Then close the dialog and open `Model info` for `gpt-6-luna-fast`. In `Routing`, its route summary has a `Fast (priority)` badge. Expand it (`Details for …, route 1`); its `Service tier` row reads `Fast (priority) on every request`. Routes of other models read `The tier the caller requests`. Compare with `service_tiers`, `available_service_tiers` and `pricing_service_tiers` in `/api/v1/admin/models`. The [Service tiers](./service-tiers.md) driver checks the allow-list and forced-route rows.
- **Client configuration.** Choose the button named `Generate client config for gpt-6-astra`. A dialog headed `Client config` appears. Do not treat generated configuration as live-provider proof.
- **Claude Code configuration.** Select a Claude model and open the `Claude Code` tab. The `Gateway model settings` block uses `ANTHROPIC_DEFAULT_HAIKU_MODEL` for background work, not the deprecated `ANTHROPIC_SMALL_FAST_MODEL`. A compatible selected Haiku takes priority; otherwise it uses the first selected model that can turn thinking off. An all-always-thinking selection shows a warning. Setup points to user settings or an ignored `.claude/settings.local.json`, not the shared project file. The optional `Lower token usage settings` block keeps `ENABLE_TOOL_SEARCH=auto` and does not disable experimental betas or change fullscreen rendering. Compare the copied JSON with the client-config API response.
- **Decisions-only model.** The `jev` row shows `—` in its Config cell, and `Select model jev` is disabled.
- **LiteLLM model info.** Create a temporary user API key through [API keys](./api-keys.md) and keep it in memory. `GET /model_group/info` with `Authorization: Bearer <key>` returns `200`. The set of `data[].model_group` IDs equals the set of `data[].id` from `GET /v1/models` with the same key. Every row has a non-empty lowercase `providers` array, `mode` (`chat` or `embedding`), a boolean `supports_vision`, a boolean `supports_function_calling`, and a `supported_openai_params` array. A priced row has `input_cost_per_token` and `output_cost_per_token`, plus `cache_read_input_token_cost` and `cache_creation_input_token_cost` when the cache prices are known; an unpriced row has none of them. The response must not contain a provider base URL or credential. This check makes no provider call. `drive models` does not run it, so report it beside the driver proof. Revoke the key afterwards.
- **Proof.** Retain the Models screenshots and ARIA snapshots, including `03-model-benchmarks` (plus one `05-model-client-config-<key>` pair per client config), and `models-proof.json` in the run evidence directory.

## Gotchas

- The mobile list replaces `models-desktop-table` below the `md` breakpoint.
- Startup requires `env.*` credential references to be configured and present, but it does not validate credential correctness or upstream connectivity. A healthy Models page does not prove upstream access.
- `Refresh pricing` can call the pricing refresh boundary and is outside the read-only baseline. It refreshes models.dev, OpenRouter, and LiteLLM and shows one toast: `Pricing refreshed: models.dev <outcome>, OpenRouter <outcome>, LiteLLM <outcome>` (each outcome `updated` or `unchanged`) when every source succeeds, or the warning `Pricing refresh failed for <sources>; <remaining summary>` when any source fails. `Pricing refreshed, but the model list did not reload` is an error toast for a failed list reload after a successful refresh.
- Some aliases and decisions-only models such as `jev` have no client configuration. Use `gpt-6-astra` for the baseline detail proof.
- Demo model IDs follow `gateway.yaml`. When a proof times out waiting for a model cell, check whether the model was renamed there before suspecting the UI.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall