ahstn/oceans-llm / .agents/skills/verify-oceans-admin
verify-oceans-admin
VerifiedVerify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.
features/README.md
View on GitHub# Oceans Admin verification map
This directory is the maintained source for verification of the user-facing Oceans LLM admin control plane. Read this index before driving the app, then use the matching feature file as the recipe.
## Baseline preconditions
- Start the real gateway and admin UI with `control-oceans-admin launch`, which runs `mise run dev-stack` with the recorded alternate ports exported as `GATEWAY_PORT` and `UI_PORT`.
- Use only the stack started for the current `OCEANS_VERIFY_RUN_ID`.
- Require `control-oceans-admin doctor` to report the recorded gateway PID, ready state, and gateway version.
- Use the seeded `admin@local` / `admin` platform-admin account unless a feature requires a narrower role.
- Verify Agent Sessions with the seeded platform administrator. The page needs `agent_sessions` in `gateway.yaml` plus agent-analysis access, which `dev-stack` grants through `AGENT_ANALYSIS_SHADOW_DIAGNOSTICS_ENABLED=true`.
- Expect `dev-stack` to refresh the demo data in this checkout's `gateway.db`. Skills verification is an explicit exception: `OCEANS_VERIFY_SKILLS=true` uses a private run-local database and native RustFS.
- Do not run a second stack from this checkout. It would share `gateway.db` even if its ports differ.
## Routing
[routing.json](./routing.json) maps repository paths to the features below. `control-oceans-admin plan` reads it.
- `features[]` pairs a feature `id` with its recipe `file`, its `proof` (`{"drive": "<target>"}` or `{"manual": true}`), an optional `paid` flag, and path globs.
- `shared[]` sends cross-cutting paths such as the admin shell, `gateway.yaml`, and the core inference path to a fixed smoke set of features.
- `skip[]` lists paths with no local runtime surface, each with a reason. Skip rules win over feature matches.
- Globs support `*`, `**`, `?`, and `{a,b}`.
When you add a recipe or a UI route, add its routing entry in the same change. `scripts/plan-verification.test.mjs` fails when a recipe or an admin route file is unrouted.
## Driving conventions
- Start every recipe from the baseline state unless its preconditions say otherwise.
- Prefer ARIA roles and accessible names over CSS selectors. Use an existing `data-testid` when the visible table has no unique accessible name.
- Treat each command and quoted label as literal.
- Use `control-oceans-admin drive models` for the automated Models proof.
- Use `control-oceans-admin drive observability` for the combined Leaderboard and Agent Harnesses proof.
- Use `control-oceans-admin drive backend-gateway` for the bounded OpenRouter, deterministic guardrail, generated-tool decision, and request-log proof.
- Use `control-oceans-admin drive service-tiers` for the paid OpenRouter service tier A/B proof, the free tier rejections, tier discovery, and request-log tier evidence.
- Use `control-oceans-admin drive harness-integration` for the paid Pi, OpenCode and packed-plugin trials in `tests/harness-integration`. It needs no launched stack.
- Use `control-oceans-admin drive mcp` for the MCP registry, Tool Sets workbench, grant checks, bounded live tool calls, and invocation proof. Read its credential preconditions before launch.
- Use `control-oceans-admin drive profile` for the Profile landing page, budget meter, heatmap, and create-key links.
- Use `control-oceans-admin drive skills` for browser uploads, immutable versions, archive digests, and owner access against native RustFS. Read its launch preconditions first.
- Extend the harness before reporting another path as automated. Manual Playwright steps in this map remain the contract for that extension.
- Read-only control-plane verification does not call upstream services. Use the live LLM recipe when changed model-request behavior warrants paid integration proof. Use the MCP recipe for authorized, reviewed read-only tool calls; these can consume upstream service quota. Neither recipe permits unrelated upstream mutations.
## Proof and skip reporting
- Capture the user entry action and resulting state, not only the final screen.
- UI proof includes an ARIA snapshot and a screenshot with `Oceans Gateway` or the page heading visible.
- Read-only data proof compares a visible list or detail with the production API used by the UI.
- Mutation proof must include a second read-only view of the saved value and cleanup of the created record.
- Every verified feature ends with a `*-proof.json` envelope: drivers write their own, and manual recipes use `control-oceans-admin record` with one `--check` per observable result and the captured screenshots as `--artifact`.
- Report an unreachable path as a `blocked` check whose note names the unmet precondition.
- Do not report a skipped entry point as verified through a different path.
- Keep proof artifacts after stack cleanup.
## Feature entry contract
Each feature file starts with an H1 title and one paragraph that describes user-visible behavior. It then uses exactly four H2 sections in this order.
1. `Sub-features` lists short IDs and one line for each behavior.
2. `How to get to it (user POV)` lists each user entry point.
3. `Driving it with control-oceans-admin` starts with `Preconditions:` and pairs each action with a stable handle and observable result.
4. `Gotchas` lists traps that can invalidate a verification run.
## Features
- [Models](./models.md) covers password sign-in, sidebar navigation, configured model listing, and model detail.
- [Leaderboard](./leaderboard.md) covers 7-day production API parity, top models, the most-used harness, and the 31-day range.
- [Agent Harnesses](./agent-harnesses.md) covers request and token aggregates, Mastra and Oh My Pi presentation, and the 31-day range.
- [Password sign-in](./password-sign-in.md) covers protected-route redirection, seeded credentials, authenticated identity, and sign-out.
- [API keys](./api-keys.md) covers the scoped key list, create and manage flows, one-time user-key secrets, and authorized service-account reveal controls.
- [Agent sessions](./agent-sessions.md) covers the seeded session list, filters, and detail sheet.
- [Request logs](./request-logs.md) covers the seeded request list, filters, and request detail.
- [Live LLM requests](./live-llm-requests.md) covers bounded paid canaries through OpenRouter or Bedrock and their request-log evidence.
- [Image generation](./image-generation.md) covers the Images API routes, chat image output, provider-reported image cost, image-safe request logs, and image output modalities in `/v1/models`.
- [Anthropic Messages](./anthropic-messages.md) covers `/v1/messages`, thinking/display/effort regressions, native transport boundaries, SSE, tool continuity, safeguards and one scoped public-endpoint canary.
- [Backend gateway](./backend-gateway.md) covers the OpenRouter route for `deepseek-v4.1-flash`, deterministic and generated-tool guardrails, request-log evidence, and temporary key cleanup.
- [Service tiers](./service-tiers.md) covers `service_tier` and `speed` requests, the default/priority/flex A/B through OpenRouter, a route-forced fast model, tier policy and routing rejections, tier discovery, and requested and served tiers in Request Logs.
- [Harness integration](./harness-integration.md) covers the real Pi and OpenCode clients and the packed OpenCode plugin: discovery, native SDK routing, variants, attachments, and token and cost parity with Request Logs.
- [Usage costs](./usage-costs.md) covers the spend KPIs, the 7- and 30-day windows, the owner filter, breakdowns, FOCUS export, and spend-report API parity.
- [Attribution](./attribution.md) covers the Git AI attribution KPIs, the 7-, 30-, and 90-day windows, the user and repository filters, the models, pull request, and change tables, the hook-install empty state, and attribution-report API parity.
- [Spend controls](./spend-controls.md) covers user, service-account, and model budgets, alert history, and a safe model-budget mutation with cleanup.
- [Rate limits](./rate-limits.md) covers the policy list, creating a per-key limit, enforcement headers and the `429` contract, pausing, deletion, and optional MCP tool-call limits.
- [Guardrails](./guardrails.md) covers the decision view's tiles, filters, and detail sheet, the policy as the `Guardrail Settings` YAML preview, and a free evaluate-endpoint decision.
- [Audit Log](./audit-log.md) covers admin audit entries from UI changes, redaction, filters, cursor paging, and API parity.
- [Batch requests](./batch-requests.md) covers the seeded batch list, filters, the responses sheet, and the cancel dialog without cancelling.
- [Identity](./identity.md) covers the Users, Teams, and Service Accounts lists, user management, team membership transfer, and read-only directories.
- [Account onboarding](./account-onboarding.md) covers password-user creation, the invite URL, `Account ready`, and voluntary password change.
- [Review agent](./review-agent.md) covers repository registration, review settings, workflow generation, run listing, and disable cleanup.
- [Workspace connections](./workspace-connections.md) covers the personal OAuth MCP connections page and its default empty state.
- [MCP ownership](./mcp-ownership.md) covers the team admin's scoped MCP area, owner and visibility in the registry, the owner filter, publishing policy, and platform ownership transfer.
- [MCP client OAuth](./mcp-client-oauth.md) covers the `401` challenge, discovery metadata, client registration, the consent page, token exchange, `oat_` MCP calls, refresh rotation, and revocation.
- [MCP](./mcp.md) covers registry discovery, Tool Sets, generated client configuration, explicit grants, direct and aggregate tool calls, upstream authentication failure, invocation records, and cleanup.
- [Skills](./skills.md) covers namespace claims, ZIP uploads, previews, immutable versions, browser downloads, shared reads, owner-only updates, service-account access, and scoped RustFS cleanup.
- [Profile](./profile.md) covers the post-sign-in landing page, budget meter API parity, usage heatmap, personal keys, and the prefilled create-key link.
- [Agent plugins](./agent-plugins.md) covers the placeholder Agent Plugins page and its empty catalog.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |