Skip to content

ahstn/oceans-llm / .agents/skills/verify-oceans-admin

verify-oceans-admin

Verified

Verify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.

FileKindSize
SKILL.mdskill md29.0 KB
examples/mcp-candidates.jsonconfig3.0 KB
features/README.mdmarkdown10.0 KB
features/account-onboarding.mdmarkdown4.3 KB
features/agent-harnesses.mdmarkdown4.3 KB
features/agent-plugins.mdmarkdown1005 B
features/agent-sessions.mdmarkdown3.9 KB
features/anthropic-messages.mdmarkdown20.7 KB
features/api-keys.mdmarkdown6.5 KB
features/attribution.mdmarkdown13.9 KB
features/audit-log.mdmarkdown4.9 KB
features/backend-gateway.mdmarkdown5.2 KB
features/batch-requests.mdmarkdown4.2 KB
features/design-system.mdmarkdown4.6 KB
features/guardrails.mdmarkdown7.0 KB
features/harness-integration.mdmarkdown7.8 KB
features/identity.mdmarkdown4.2 KB
features/image-generation.mdmarkdown7.3 KB
features/leaderboard.mdmarkdown3.3 KB
features/live-llm-requests.mdmarkdown12.6 KB
features/mcp-client-oauth.mdmarkdown9.9 KB
features/mcp-ownership.mdmarkdown7.4 KB
features/mcp.mdmarkdown19.6 KB
features/models.mdmarkdown8.2 KB
features/password-sign-in.mdmarkdown8.3 KB
features/profile.mdmarkdown2.5 KB
features/rate-limits.mdmarkdown6.0 KB
features/request-logs.mdmarkdown8.6 KB
features/review-agent.mdmarkdown3.7 KB
features/routing.jsonconfig33.6 KB
features/service-tiers.mdmarkdown6.3 KB
features/skills.mdmarkdown9.8 KB
features/spend-controls.mdmarkdown7.3 KB
features/usage-costs.mdmarkdown3.7 KB
features/workspace-connections.mdmarkdown6.1 KB
scripts/check-messages.mjsexecutablescript2.4 KB
scripts/check-messages.test.mjsscript3.0 KB
scripts/control-oceans-adminexecutablescript25.3 KB
scripts/control-oceans-admin.test.mjsscript7.6 KB
scripts/drive-backend-gateway.mjsexecutablescript12.5 KB
scripts/drive-harness-integration.mjsscript6.6 KB
scripts/drive-mcp.mjsscript13.0 KB
scripts/drive-models.mjsscript12.9 KB
scripts/drive-observability.mjsscript16.8 KB
scripts/drive-profile.mjsscript8.2 KB
scripts/drive-service-tiers.mjsexecutablescript20.1 KB
scripts/drive-skills.mjsscript27.2 KB
scripts/mcp-browser.mjsscript26.5 KB
scripts/mcp-canary.mjsscript18.8 KB
scripts/mcp-candidates.mjsscript3.5 KB
scripts/mcp-verification.test.mjsscript10.2 KB
scripts/messages-regressions.mjsscript15.8 KB
scripts/plan-verification.mjsscript5.2 KB
scripts/plan-verification.test.mjsscript6.1 KB
scripts/proof-cli.mjsscript11.1 KB
scripts/proof.mjsscript9.2 KB
scripts/proof.test.mjsscript10.4 KB

features/backend-gateway.md

View on GitHub
# Backend gateway

Backend gateway verification proves that an authenticated OpenAI-compatible request resolves a configured gateway model, reaches its live provider route, applies deterministic guardrails to generated tool calls, and records request, provider-attempt, usage, and guardrail-decision evidence.

## Sub-features

- `gateway-openrouter-route` sends a bounded Chat Completions request through gateway model `deepseek-v4.1-flash` to OpenRouter model `deepseek/deepseek-v4.1-flash`.
- `gateway-deterministic-evaluate` confirms that the active default policy evaluates a destructive shell command with the expected built-in pack and rule.
- `gateway-generated-tool-guard` confirms that a live generated function call receives a request-linked `generated_tool_call` decision.
- `gateway-log-evidence` confirms the request log, provider attempt, tool cardinality, status, and usage for the live request.
- `gateway-key-cleanup` revokes the temporary model-scoped key and confirms that gateway authentication rejects it.

## How to get to it (user POV)

- Sign in as the seeded platform administrator and open `API Keys` to create a temporary key with explicit access to `deepseek-v4.1-flash`.
- Call `/api/v1/guardrails/evaluate` with that key to inspect the active deterministic policy without executing the command.
- Call `/v1/chat/completions` with model `deepseek-v4.1-flash` and one forced synthetic function tool.
- Open `Request Logs` and `Guardrails` under `Observability` to inspect the request and its generated-tool decision.

## Driving it with control-oceans-admin

Preconditions:

- `control-oceans-admin doctor` passes for the stack owned by this verification run.
- `OPENROUTER_API_KEY` resolves through fnox, which hands it to `dev-stack` (see Launch in [SKILL.md](../SKILL.md)). The harness never reads, prints, or saves it.
- The operator accepts one short paid request. Use only the synthetic command in the driver.
- Root `gateway.yaml` keeps the default guardrail policy in `audit` mode. Under another mode, update the expected action before driving.

- **Automated proof.** Run `control-oceans-admin drive backend-gateway`. The driver signs in through the real browser route and creates one user-owned key with explicit access to `deepseek-v4.1-flash`; the raw key remains only in process memory.
- **Deterministic evaluation.** The driver submits `rm -rf /tmp/oceans-verify` to `/api/v1/guardrails/evaluate`. The response must be allowed with action `audit`, pack `core.filesystem`, rule `recursive-force-remove`, and reason `filesystem.recursive_force_remove`. The command is evaluated only and is never executed.
- **Live OpenRouter request.** The driver submits one non-streaming Chat Completions request with a forced `bash` function call, `max_tokens: 1024`, and gateway model `deepseek-v4.1-flash`. It requires HTTP 200 and an `x-request-id`.
- **Decision and log proof.** Through the authenticated browser session, the driver requires a request-linked `generated_tool_call` decision for `core.filesystem`, then loads the matching request-log detail. It confirms provider `openrouter`, configured upstream model `deepseek/deepseek-v4.1-flash`, success, a tool call, and provider usage when supplied.
- **Cleanup.** The driver revokes the temporary key in a `finally` block and confirms that `/v1/models` rejects it. It writes only sanitized metadata to `backend-gateway-canary-proof.json`.
- **Proof.** Run `control-oceans-admin evidence backend-gateway` before and after stack cleanup.

## Gotchas

- DeepSeek Flash models spend reasoning tokens before the forced tool call. With a small `max_tokens` (32 did this on `deepseek-v4-flash-0731`), OpenRouter reports `native_finish_reason: "length"` and returns `{}` arguments, so the driver fails with `generated bash command ... got undefined`. Inspect the request-log payload before assuming a gateway fault.
- A configured route, healthy Models page, or present environment reference is not live-provider proof. Startup requires referenced environment variables to exist but does not validate credential correctness or upstream reachability.
- OpenRouter can choose its serving host. The gateway request log proves the configured OpenRouter provider and upstream model ID, not the physical serving host.
- Root `gateway.yaml` uses audit mode, so a matched generated tool call is returned to the caller and recorded as `audit`. It is not executed by this driver.
- `/api/v1/admin/guardrails/decisions` returns guardrail decision events. It is unrelated to the `/v1/decisions` and `/v1/systemone` inference endpoints (model `jev`), which record no guardrail decisions.
- `core.shell` is also enabled. Evaluate reports the last non-allow decision among enabled packs, so reordering packs in `gateway.yaml` can change the expected pack.
- Direct `/api/v1/guardrails/evaluate` decisions have no inference request ID. Use the generated-tool decision for request-linked proof.
- Payload evidence is redacted and bounded by `gateway.yaml`; do not require raw prompts, responses, credentials, or authorization headers.
- Provider outages, model access changes, expired credentials, quota, and network failures are concrete integration failures. Run `doctor` again before retrying, then clean the failed key and request residue.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall