ahstn/oceans-llm / .agents/skills/verify-oceans-admin
verify-oceans-admin
VerifiedVerify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.
features/harness-integration.md
View on GitHub# Harness integration
Harness integration verification runs the real OpenCode (with the packed `@ahstn/opencode-oceans` plugin), Pi, Oh-My-Pi and Claude Code command-line clients against a gateway built from this checkout. It proves that the inference endpoints, model discovery and generated client configs work in the clients that people use, not only in gateway unit tests. The suite and its fixtures are in [`tests/harness-integration`](../../../../tests/harness-integration/README.md). The published matrix is [Harness Verification](../../../../docs/src/content/docs/harnesses/verification.mdx); keep it in step with `tests/harness-integration/src/matrix.ts`.
This proof packs the checkout, not the npm release. After approval, the independent `mise run //opencode:smoke-published -- 0.1.0` proof installs the exact version anonymously from public npm and uses a local synthetic gateway without paid requests. It does not replace the live harness proof.
## Sub-features
- `matrix` runs harness x model x provider cells from `tests/harness-integration/src/matrix.ts`. Harnesses: OpenCode plugin, Pi (generated `models.json`), OMP (`litellm` discovery) and Claude Code (generated `settings.json`). Models: `gpt-6-luna`, `claude-haiku-5.5`, `gemini-4-hydrogen`. Providers: OpenRouter (always; 10 cells), Vertex AI `global` (Gemini only for OpenCode, Pi and OMP; runs when `OCEANS_HARNESS_VERTEX_PROJECT` and Google credentials resolve, else skipped as optional `blocked`), and Bedrock (every cell blocked today; opt-in flag only). Each cell runs a conversation, a file read/write, a `date` shell call, an `effort` conversation at `low` reasoning effort and a `fast` conversation at the `priority` tier, each set with a setting the harness gives users. `effort` checks that the log has no tier and that the stored request body has `low` in `reasoning.effort` (Responses), `output_config.effort` (Messages) or `reasoning_effort` (Chat Completions). `fast` checks `service_tier: "priority"` in the log; the served tier is printed, not checked. `fast` is `not-applicable` per test for every Pi cell, OMP with Gemini and Claude Code with Haiku (reasons in `matrix.ts`), printed as `Matrix cell:` lines that name the test. Each run checks its request logs for status 200, model, provider, `metadata.operation` (`responses`, `messages` or `chat_completions`) and `agent_harness_key` (`opencode`, `pi`, `oh_my_pi`, and `claude_cli` for Claude Code, whose print mode sends a `claude-cli/` user agent). Cells that don't run are data with a reason: `blocked` (could work; the test account lacks access or quota, or Bedrock `runtime_anthropic_invoke` can't stream for Claude Code) or `not-applicable` (Claude Code with non-Claude models; no OpenAI on Vertex; no Gemini on Bedrock). The suite prints them as `Matrix cell:` lines.
- `mcp` calls Context7 through the aggregate `/mcp` endpoint once per harness.
- `opencode-snippet` runs one conversation with the `opencode.json` that Oceans generates in `Client config`.
- `user-allowlist` proves that a human user whose normalized email is on a model allowlist can call the model from Pi.
- `plugin-discovery` loads the packed plugin in a private OpenCode server and checks the discovered models (including Gemini on `openai-compatible`), hidden-model exclusion, native SDK packages, limits, prices, capability restrictions, and the `low`, `high` and `fast` variants.
- `plugin-live` sends six requests with an image and a PDF: `low`, `high` and `fast` for `gpt-6-luna` (Responses) and `claude-haiku-5.5` (Messages). It checks the native wire fields, the requested and served tier in the request log, and token and cost parity with OpenCode.
- `offline` runs the guardrail hook, event parser, process and setup tests.
## How to get to it (user POV)
- A user installs OpenCode, Pi, OMP or Claude Code. They add `@ahstn/opencode-oceans`, copy the Pi, OpenCode or Claude Code config from `Models` → `Client config`, or add Oceans to OMP's `models.yml` with `discovery.type: litellm`.
- The user picks a model, such as `oceans/gpt-6-luna#fast`, and sends a prompt.
- An admin sees each request in `Request Logs`, tagged with the harness run.
## Driving it with control-oceans-admin
Preconditions:
- No launched stack is needed. The suite builds `target/debug/gateway` from this checkout and starts a temporary gateway with its own LibSQL database. Set `OCEANS_VERIFY_RUN_ID` so the proof has a run.
- `OPENROUTER_API_KEY` is in the environment, or fnox can resolve it without a prompt. The driver passes it only to the suite, which passes it only to the temporary gateway. The driver never prints or saves it.
- The machine can reach OpenRouter and Context7.
- Bun 1.4.4 or later on `PATH` (OMP and the plugin need it), and `bwrap` on Linux for the shell sandbox.
- The operator accepts short paid requests: six plugin requests, 45 matrix runs on OpenRouter (10 cells x 5 tests, less 5 not-applicable fast tests; tool runs take more than one call), four MCP runs, and the snippet and allowlist runs. Gemini 4 Hydrogen costs about 15 times more per token than the other two models. The temporary service account has a hard daily budget of $10, and each plugin request has a 2,048-token output limit.
- **Automated proof.** Run `control-oceans-admin drive harness-integration`. The driver installs the suite's dependencies with `mise run //harness:install`, snapshots the routed source, and runs `npm test` in `tests/harness-integration` with the verbose and JSON reporters.
- **Checks.** The proof has one check per test, named from the test file and title. A skipped test, such as the prompt-cache canary that needs an external gateway, is an optional `blocked` check. `plugin-live-usage` passes only when the plugin printed a live-request line for every live case. `harness-suite-exit` fails when the suite exits non-zero or is killed, even if every reported test passed. The driver deletes the previous JSON report first, so a crashed run cannot reuse it.
- **Details.** `details.live` lists each plugin request: family, variant, endpoint, request-log ID, requested and served tier, tokens, and the OpenCode and Oceans cost.
- **Artifacts.** `harness-integration-results.json` is the vitest JSON report. The full suite log, with the temporary gateway's logs, is `harness-integration.log` beside the evidence directory.
- **Proof.** Run `control-oceans-admin evidence harness-integration --require-current`.
## Gotchas
- The suite rebuilds `target/debug/gateway`. A launched verification stack in this checkout keeps running its old process, but stop it first if you want the two to use the same build.
- On a BYOK OpenRouter account, OpenRouter returns `usage.cost: 0` for OpenAI models and puts the charge in `cost_details.upstream_inference_cost`. OpenCode uses `usage.cost`, so its cost for `gpt-6-luna` is `$0`. Oceans bills the upstream cost. The parity check allows $0.0001, so it still passes at canary size.
- OpenRouter serves `claude-haiku-5.5` at standard speed, so the Anthropic `fast` case is requested at `priority` and served at `default`. The suite accepts any served tier and checks billing against it.
- OpenAI can serve a priority request at `default` when it is short of capacity. The `fast` case then checks cost parity instead of a higher Oceans cost.
- Pi sends a GPT effort only when the generated `models.json` marks the model `reasoning: true`. If the Pi GPT `effort` test finds no `reasoning_effort`, check the generated Pi config before the harness.
- OMP sends Claude fast mode to the custom `oceans` provider only with `compat.fastMode: true` on the model override; `tier.anthropic: priority` alone sends nothing. Claude Code fast mode is Opus-only: `/fast on` switches the model to Opus, so Haiku requests carry no tier.
- Changes to `tests/harness-integration/**` are routed to the "tests only" skip rule. Run this proof by hand when you change the suite.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |