Skip to content

ahstn/oceans-llm / .agents/skills/verify-oceans-admin

verify-oceans-admin

Verified

Verify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.

FileKindSize
SKILL.mdskill md29.0 KB
examples/mcp-candidates.jsonconfig3.0 KB
features/README.mdmarkdown10.0 KB
features/account-onboarding.mdmarkdown4.3 KB
features/agent-harnesses.mdmarkdown4.3 KB
features/agent-plugins.mdmarkdown1005 B
features/agent-sessions.mdmarkdown3.9 KB
features/anthropic-messages.mdmarkdown20.7 KB
features/api-keys.mdmarkdown6.5 KB
features/attribution.mdmarkdown13.9 KB
features/audit-log.mdmarkdown4.9 KB
features/backend-gateway.mdmarkdown5.2 KB
features/batch-requests.mdmarkdown4.2 KB
features/design-system.mdmarkdown4.6 KB
features/guardrails.mdmarkdown7.0 KB
features/harness-integration.mdmarkdown7.8 KB
features/identity.mdmarkdown4.2 KB
features/image-generation.mdmarkdown7.3 KB
features/leaderboard.mdmarkdown3.3 KB
features/live-llm-requests.mdmarkdown12.6 KB
features/mcp-client-oauth.mdmarkdown9.9 KB
features/mcp-ownership.mdmarkdown7.4 KB
features/mcp.mdmarkdown19.6 KB
features/models.mdmarkdown8.2 KB
features/password-sign-in.mdmarkdown8.3 KB
features/profile.mdmarkdown2.5 KB
features/rate-limits.mdmarkdown6.0 KB
features/request-logs.mdmarkdown8.6 KB
features/review-agent.mdmarkdown3.7 KB
features/routing.jsonconfig33.6 KB
features/service-tiers.mdmarkdown6.3 KB
features/skills.mdmarkdown9.8 KB
features/spend-controls.mdmarkdown7.3 KB
features/usage-costs.mdmarkdown3.7 KB
features/workspace-connections.mdmarkdown6.1 KB
scripts/check-messages.mjsexecutablescript2.4 KB
scripts/check-messages.test.mjsscript3.0 KB
scripts/control-oceans-adminexecutablescript25.3 KB
scripts/control-oceans-admin.test.mjsscript7.6 KB
scripts/drive-backend-gateway.mjsexecutablescript12.5 KB
scripts/drive-harness-integration.mjsscript6.6 KB
scripts/drive-mcp.mjsscript13.0 KB
scripts/drive-models.mjsscript12.9 KB
scripts/drive-observability.mjsscript16.8 KB
scripts/drive-profile.mjsscript8.2 KB
scripts/drive-service-tiers.mjsexecutablescript20.1 KB
scripts/drive-skills.mjsscript27.2 KB
scripts/mcp-browser.mjsscript26.5 KB
scripts/mcp-canary.mjsscript18.8 KB
scripts/mcp-candidates.mjsscript3.5 KB
scripts/mcp-verification.test.mjsscript10.2 KB
scripts/messages-regressions.mjsscript15.8 KB
scripts/plan-verification.mjsscript5.2 KB
scripts/plan-verification.test.mjsscript6.1 KB
scripts/proof-cli.mjsscript11.1 KB
scripts/proof.mjsscript9.2 KB
scripts/proof.test.mjsscript10.4 KB

features/README.md

View on GitHub
# Oceans Admin verification map

This directory is the maintained source for verification of the user-facing Oceans LLM admin control plane. Read this index before driving the app, then use the matching feature file as the recipe.

## Baseline preconditions

- Start the real gateway and admin UI with `control-oceans-admin launch`, which runs `mise run dev-stack` with the recorded alternate ports exported as `GATEWAY_PORT` and `UI_PORT`.
- Use only the stack started for the current `OCEANS_VERIFY_RUN_ID`.
- Require `control-oceans-admin doctor` to report the recorded gateway PID, ready state, and gateway version.
- Use the seeded `admin@local` / `admin` platform-admin account unless a feature requires a narrower role.
- Verify Agent Sessions with the seeded platform administrator. The page needs `agent_sessions` in `gateway.yaml` plus agent-analysis access, which `dev-stack` grants through `AGENT_ANALYSIS_SHADOW_DIAGNOSTICS_ENABLED=true`.
- Expect `dev-stack` to refresh the demo data in this checkout's `gateway.db`. Skills verification is an explicit exception: `OCEANS_VERIFY_SKILLS=true` uses a private run-local database and native RustFS.
- Do not run a second stack from this checkout. It would share `gateway.db` even if its ports differ.

## Routing

[routing.json](./routing.json) maps repository paths to the features below. `control-oceans-admin plan` reads it.

- `features[]` pairs a feature `id` with its recipe `file`, its `proof` (`{"drive": "<target>"}` or `{"manual": true}`), an optional `paid` flag, and path globs.
- `shared[]` sends cross-cutting paths such as the admin shell, `gateway.yaml`, and the core inference path to a fixed smoke set of features.
- `skip[]` lists paths with no local runtime surface, each with a reason. Skip rules win over feature matches.
- Globs support `*`, `**`, `?`, and `{a,b}`.

When you add a recipe or a UI route, add its routing entry in the same change. `scripts/plan-verification.test.mjs` fails when a recipe or an admin route file is unrouted.

## Driving conventions

- Start every recipe from the baseline state unless its preconditions say otherwise.
- Prefer ARIA roles and accessible names over CSS selectors. Use an existing `data-testid` when the visible table has no unique accessible name.
- Treat each command and quoted label as literal.
- Use `control-oceans-admin drive models` for the automated Models proof.
- Use `control-oceans-admin drive observability` for the combined Leaderboard and Agent Harnesses proof.
- Use `control-oceans-admin drive backend-gateway` for the bounded OpenRouter, deterministic guardrail, generated-tool decision, and request-log proof.
- Use `control-oceans-admin drive service-tiers` for the paid OpenRouter service tier A/B proof, the free tier rejections, tier discovery, and request-log tier evidence.
- Use `control-oceans-admin drive harness-integration` for the paid Pi, OpenCode and packed-plugin trials in `tests/harness-integration`. It needs no launched stack.
- Use `control-oceans-admin drive mcp` for the MCP registry, Tool Sets workbench, grant checks, bounded live tool calls, and invocation proof. Read its credential preconditions before launch.
- Use `control-oceans-admin drive profile` for the Profile landing page, budget meter, heatmap, and create-key links.
- Use `control-oceans-admin drive skills` for browser uploads, immutable versions, archive digests, and owner access against native RustFS. Read its launch preconditions first.
- Extend the harness before reporting another path as automated. Manual Playwright steps in this map remain the contract for that extension.
- Read-only control-plane verification does not call upstream services. Use the live LLM recipe when changed model-request behavior warrants paid integration proof. Use the MCP recipe for authorized, reviewed read-only tool calls; these can consume upstream service quota. Neither recipe permits unrelated upstream mutations.

## Proof and skip reporting

- Capture the user entry action and resulting state, not only the final screen.
- UI proof includes an ARIA snapshot and a screenshot with `Oceans Gateway` or the page heading visible.
- Read-only data proof compares a visible list or detail with the production API used by the UI.
- Mutation proof must include a second read-only view of the saved value and cleanup of the created record.
- Every verified feature ends with a `*-proof.json` envelope: drivers write their own, and manual recipes use `control-oceans-admin record` with one `--check` per observable result and the captured screenshots as `--artifact`.
- Report an unreachable path as a `blocked` check whose note names the unmet precondition.
- Do not report a skipped entry point as verified through a different path.
- Keep proof artifacts after stack cleanup.

## Feature entry contract

Each feature file starts with an H1 title and one paragraph that describes user-visible behavior. It then uses exactly four H2 sections in this order.

1. `Sub-features` lists short IDs and one line for each behavior.
2. `How to get to it (user POV)` lists each user entry point.
3. `Driving it with control-oceans-admin` starts with `Preconditions:` and pairs each action with a stable handle and observable result.
4. `Gotchas` lists traps that can invalidate a verification run.

## Features

- [Models](./models.md) covers password sign-in, sidebar navigation, configured model listing, and model detail.
- [Leaderboard](./leaderboard.md) covers 7-day production API parity, top models, the most-used harness, and the 31-day range.
- [Agent Harnesses](./agent-harnesses.md) covers request and token aggregates, Mastra and Oh My Pi presentation, and the 31-day range.
- [Password sign-in](./password-sign-in.md) covers protected-route redirection, seeded credentials, authenticated identity, and sign-out.
- [API keys](./api-keys.md) covers the scoped key list, create and manage flows, one-time user-key secrets, and authorized service-account reveal controls.
- [Agent sessions](./agent-sessions.md) covers the seeded session list, filters, and detail sheet.
- [Request logs](./request-logs.md) covers the seeded request list, filters, and request detail.
- [Live LLM requests](./live-llm-requests.md) covers bounded paid canaries through OpenRouter or Bedrock and their request-log evidence.
- [Image generation](./image-generation.md) covers the Images API routes, chat image output, provider-reported image cost, image-safe request logs, and image output modalities in `/v1/models`.
- [Anthropic Messages](./anthropic-messages.md) covers `/v1/messages`, thinking/display/effort regressions, native transport boundaries, SSE, tool continuity, safeguards and one scoped public-endpoint canary.
- [Backend gateway](./backend-gateway.md) covers the OpenRouter route for `deepseek-v4.1-flash`, deterministic and generated-tool guardrails, request-log evidence, and temporary key cleanup.
- [Service tiers](./service-tiers.md) covers `service_tier` and `speed` requests, the default/priority/flex A/B through OpenRouter, a route-forced fast model, tier policy and routing rejections, tier discovery, and requested and served tiers in Request Logs.
- [Harness integration](./harness-integration.md) covers the real Pi and OpenCode clients and the packed OpenCode plugin: discovery, native SDK routing, variants, attachments, and token and cost parity with Request Logs.
- [Usage costs](./usage-costs.md) covers the spend KPIs, the 7- and 30-day windows, the owner filter, breakdowns, FOCUS export, and spend-report API parity.
- [Attribution](./attribution.md) covers the Git AI attribution KPIs, the 7-, 30-, and 90-day windows, the user and repository filters, the models, pull request, and change tables, the hook-install empty state, and attribution-report API parity.
- [Spend controls](./spend-controls.md) covers user, service-account, and model budgets, alert history, and a safe model-budget mutation with cleanup.
- [Rate limits](./rate-limits.md) covers the policy list, creating a per-key limit, enforcement headers and the `429` contract, pausing, deletion, and optional MCP tool-call limits.
- [Guardrails](./guardrails.md) covers the decision view's tiles, filters, and detail sheet, the policy as the `Guardrail Settings` YAML preview, and a free evaluate-endpoint decision.
- [Audit Log](./audit-log.md) covers admin audit entries from UI changes, redaction, filters, cursor paging, and API parity.
- [Batch requests](./batch-requests.md) covers the seeded batch list, filters, the responses sheet, and the cancel dialog without cancelling.
- [Identity](./identity.md) covers the Users, Teams, and Service Accounts lists, user management, team membership transfer, and read-only directories.
- [Account onboarding](./account-onboarding.md) covers password-user creation, the invite URL, `Account ready`, and voluntary password change.
- [Review agent](./review-agent.md) covers repository registration, review settings, workflow generation, run listing, and disable cleanup.
- [Workspace connections](./workspace-connections.md) covers the personal OAuth MCP connections page and its default empty state.
- [MCP ownership](./mcp-ownership.md) covers the team admin's scoped MCP area, owner and visibility in the registry, the owner filter, publishing policy, and platform ownership transfer.
- [MCP client OAuth](./mcp-client-oauth.md) covers the `401` challenge, discovery metadata, client registration, the consent page, token exchange, `oat_` MCP calls, refresh rotation, and revocation.
- [MCP](./mcp.md) covers registry discovery, Tool Sets, generated client configuration, explicit grants, direct and aggregate tool calls, upstream authentication failure, invocation records, and cleanup.
- [Skills](./skills.md) covers namespace claims, ZIP uploads, previews, immutable versions, browser downloads, shared reads, owner-only updates, service-account access, and scoped RustFS cleanup.
- [Profile](./profile.md) covers the post-sign-in landing page, budget meter API parity, usage heatmap, personal keys, and the prefilled create-key link.
- [Agent plugins](./agent-plugins.md) covers the placeholder Agent Plugins page and its empty catalog.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall