Skip to content

ahstn/oceans-llm / .agents/skills/verify-oceans-admin

verify-oceans-admin

Verified

Verify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.

FileKindSize
SKILL.mdskill md29.0 KB
examples/mcp-candidates.jsonconfig3.0 KB
features/README.mdmarkdown10.0 KB
features/account-onboarding.mdmarkdown4.3 KB
features/agent-harnesses.mdmarkdown4.3 KB
features/agent-plugins.mdmarkdown1005 B
features/agent-sessions.mdmarkdown3.9 KB
features/anthropic-messages.mdmarkdown20.7 KB
features/api-keys.mdmarkdown6.5 KB
features/attribution.mdmarkdown13.9 KB
features/audit-log.mdmarkdown4.9 KB
features/backend-gateway.mdmarkdown5.2 KB
features/batch-requests.mdmarkdown4.2 KB
features/design-system.mdmarkdown4.6 KB
features/guardrails.mdmarkdown7.0 KB
features/harness-integration.mdmarkdown7.8 KB
features/identity.mdmarkdown4.2 KB
features/image-generation.mdmarkdown7.3 KB
features/leaderboard.mdmarkdown3.3 KB
features/live-llm-requests.mdmarkdown12.6 KB
features/mcp-client-oauth.mdmarkdown9.9 KB
features/mcp-ownership.mdmarkdown7.4 KB
features/mcp.mdmarkdown19.6 KB
features/models.mdmarkdown8.2 KB
features/password-sign-in.mdmarkdown8.3 KB
features/profile.mdmarkdown2.5 KB
features/rate-limits.mdmarkdown6.0 KB
features/request-logs.mdmarkdown8.6 KB
features/review-agent.mdmarkdown3.7 KB
features/routing.jsonconfig33.6 KB
features/service-tiers.mdmarkdown6.3 KB
features/skills.mdmarkdown9.8 KB
features/spend-controls.mdmarkdown7.3 KB
features/usage-costs.mdmarkdown3.7 KB
features/workspace-connections.mdmarkdown6.1 KB
scripts/check-messages.mjsexecutablescript2.4 KB
scripts/check-messages.test.mjsscript3.0 KB
scripts/control-oceans-adminexecutablescript25.3 KB
scripts/control-oceans-admin.test.mjsscript7.6 KB
scripts/drive-backend-gateway.mjsexecutablescript12.5 KB
scripts/drive-harness-integration.mjsscript6.6 KB
scripts/drive-mcp.mjsscript13.0 KB
scripts/drive-models.mjsscript12.9 KB
scripts/drive-observability.mjsscript16.8 KB
scripts/drive-profile.mjsscript8.2 KB
scripts/drive-service-tiers.mjsexecutablescript20.1 KB
scripts/drive-skills.mjsscript27.2 KB
scripts/mcp-browser.mjsscript26.5 KB
scripts/mcp-canary.mjsscript18.8 KB
scripts/mcp-candidates.mjsscript3.5 KB
scripts/mcp-verification.test.mjsscript10.2 KB
scripts/messages-regressions.mjsscript15.8 KB
scripts/plan-verification.mjsscript5.2 KB
scripts/plan-verification.test.mjsscript6.1 KB
scripts/proof-cli.mjsscript11.1 KB
scripts/proof.mjsscript9.2 KB
scripts/proof.test.mjsscript10.4 KB

features/request-logs.md

View on GitHub
# Request logs

Request logs lets an authorized user review seeded gateway requests, filter them by service and tags, inspect stored payloads and provider attempts, and follow related MCP invocation data.

## Sub-features

- `logs-list` shows the seeded request list in desktop or mobile form.
- `logs-filter` adds removable filter chips for model, provider, service, component, environment, tag pair, and request ID. Each chip writes its URL search param (`model_key`, `provider_key`, `service`, `component`, `env`, `tag_key` and `tag_value`, `request_id`), so the same filters deep-link (for example `?request_id=demo-req-016`).
- `logs-search` is the debounced `Search request logs by model or user` box (placeholder `Search by model or user…`). It writes `q` to the URL with a history replace, so Back skips intermediate keystrokes.
- `logs-detail` opens the request detail sheet (`request-log-detail-sheet`). Its header shows the status badge, operation, timestamp, the request ID as the sheet title with a `Copy request ID` button, and the model and provider (`request-log-model-route`). The body shows the `Latency`, `Tokens`, `Cost` and `Cache hit` tiles (`request-log-key-metrics`), a two-column fact list (`request-log-facts`: `Caller`, `API key`, `Agent harness`, `User-agent`, `Request ID`, `Request log ID`, `Operation` (Anthropic Messages, Chat Completions, Responses, Embeddings, or Decisions), `Stream`, `Service tier`, `Occurred at`, `Error code`, `Tags`), then the `MCP & tools` (`request-log-tools`), `Provider attempts` (`request-log-attempts`) and `Payloads` sections. The MCP token overhead estimate (`request-log-mcp-overhead`) sits inside `MCP & tools` when present. IDs truncate on one line, with the full value in the `title` attribute. `Service tier` is hidden when the log has neither a requested nor a served tier; [Service tiers](./service-tiers.md) covers its badges.
- `logs-mcp-link` follows a request to related MCP invocations when permitted.
- `logs-budget-rejection` shows a request that a hard budget rejected before any provider call. Its row has the status the caller received (the gateway's `budget_exceeded` status), `Error code` `budget_exceeded`, and `No provider attempts recorded`.

## How to get to it (user POV)

- Sign in and choose `Request Logs` under `Observability`.
- Open `/admin/observability/request-logs` directly.
- Choose `Inspect` on a request row or card to open its detail.

## Driving it with control-oceans-admin

Preconditions:

- `control-oceans-admin doctor` passes.
- `mise run dev-stack` completed its local demo seed.
- The signed-in session includes the `request_logs` page permission.

- **Open list.** Choose `Request Logs`. The `Request logs` heading and `Request list` card are visible. At desktop width, `request-log-desktop-table` is visible.
- **Filter.** Choose `Filters`, then choose `Service` in the `Filter by` menu. A `Service` chip appears with its input focused. Fill `Service filter value` with `developer-productivity` and press Enter. The URL search gains `service=developer-productivity`, the `Filters` button shows a count badge of `1`, and the toolbar reads `Showing 10 of 10`. `Component` and `Environment` work the same way.
- **Tag filter.** Choose `Add filter`, then `Tag`. Fill `Tag key` only. The `Incomplete tag filter` alert shows and the URL does not change. Fill `Tag value` and press Enter to commit the pair as `tag_key` and `tag_value`.
- **Clear.** Choose `Clear`. The chips are removed, the URL returns to its unfiltered query, and `Showing N of M` returns to the unfiltered total.
- **Inspect.** Add a `Request ID` chip for `demo-req-016`, or open `?request_id=demo-req-016`, then choose `Inspect`. The detail sheet opens with the heading `demo-req-016`, a `200` status badge, and `gpt-oss-120b` via `bedrock-us-east-1`. The key metric tiles read `978 ms`, `1,960` (`1,340 in · 620 out`), `$0.4210` and `31%` (`415 of 1,340 prompt tokens`). `MCP & tools` shows the `Offered` group (`MCP servers` 4, `Request tools` 31, `Tools exposed` 27, `Tools filtered` 5) and the `Called` group (`Tool calls` 6, `MCP tool calls` 4, `Distinct tools called` 4), then the MCP token overhead: `3,105` definition tokens, `2.36%` of a `131,072`-token context window, cache hits and misses, confidence and estimator. `Payloads` has the toggle group `Payload view` (`Request` / `Response` / `Split`, default `Split`) and shows the `Request payload` and `Response payload` code blocks (`request-log-payloads`), each with a `full` or `truncated` badge and a copy button, or `No payload stored`.
- **Messages parity.** For the single canary in [Anthropic Messages](./anthropic-messages.md), filter by its response request ID. Require the expected stream flag, model/provider, status and usage, plus the matching successful upstream attempt. Require the `Operation` fact to read `Messages` and `log.metadata.operation` to be `messages`. Mask `request-log-payloads` in screenshots and omit it from ARIA/text artifacts.
- **Budget rejection.** As `admin@local`, create a temporary key owned by `gina@research.local` (seeded at 104% of a hard $200 monthly budget) with access to one model, following the API Keys recipe. Send one non-streaming `/v1/chat/completions` request with a unique `x-request-id`. Expect the `budget_exceeded` status and `"type": "budget_error"` in the body, with no provider call and no spend. Repeat with `/v1/responses`, `/v1/messages`, `/v1/embeddings` (an embeddings model), or `/v1/decisions` (`jev`) when the change touches that route. Add a `Request ID` filter chip for each request ID. Each row shows the same status. Inspect it: the header status badge matches, `Error code` reads `budget_exceeded`, the `Tokens` and `Cost` tiles read `—`, and `Provider attempts` reads `No provider attempts recorded`. Through the API, `items[0].status_code` and `items[0].error_code` match and `attempts` is `[]`. Revoke the key afterwards.
- **Related MCP data.** Choose `View MCP invocations` (in the `MCP & tools` section header) only when the signed-in session has access. The destination query keeps the request ID.
- **Proof.** Capture the list before filtering, the applied filter and result count, and one request detail. Record the request ID and compare its visible fields with `GET /api/v1/admin/observability/request-logs?request_id=…` (`items[0].request_log_id`) and `GET /api/v1/admin/observability/request-logs/{request_log_id}` (`log.*`, `attempts[]`, `mcp_token_overhead`).

## Gotchas

- The mobile list replaces `request-log-desktop-table` below the `lg` breakpoint.
- The detail sheet is one scrolling column (`request-log-detail-body`). Split payloads sit side by side from `xl`; below that they stack. Each payload block shows up to 28 lines and scrolls inside itself, and long lines wrap rather than scroll sideways.
- `gateway.yaml` capture limits apply only to live requests; seeded rows carry fixture payloads. A truncation badge is a valid stored result, not missing UI data.
- Non-platform-admin sessions see only their own logs. Verify parity as `admin@local`.
- Decisions logs store the response in the caller's format. A row from `/v1/decisions` with an OpenAI-format body has an ordered `answers` list, and a row from `/v1/systemone` has an `answers` map. Prompt and completion counts match the upstream usage in both cases.
- The detail does not show guardrail decisions. Use [Guardrails](./guardrails.md) or `/api/v1/admin/guardrails/decisions`.
- There is no apply button. A chip value commits on Enter or when its input loses focus, and only if it changed the filters. Wait for `Showing N of M` to change before you compare it with the API.
- `Clear` resets only the chips. A search term in `q` stays until you empty the search box.
- Tag filtering requires both key and value. The UI keeps a partial pair out of the URL, but the API still returns 400 for `tag_key` without `tag_value` (or the reverse).
- `Add filter` and `Clear` appear only once a chip exists. A field that already has a chip is disabled in the `Filter by` menu. Remove one chip with its `Remove <Field> filter` button.
- Request rows use virtualization on desktop. Locate visible content or scroll the table viewport before selecting an off-screen row.
- A budget rejection writes a row under the same policy as any request: `request_logging.payloads.capture_mode` must not be `disabled`, and a user-owned key's owner must have `request_logging_enabled: true` (Gina does). Service-account keys always log. Batch creation (`POST /v1/batches`) enforces the same budget but writes no request-log row.
- `demo-req-016` has a request-ID deep link to MCP Invocations but no seeded invocation row. Prove the retained query, not a non-empty destination result.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall