ahstn/oceans-llm / .agents/skills/verify-oceans-admin
verify-oceans-admin
VerifiedVerify the Oceans LLM admin control plane and backend gateway against the real local stack, seeded demo data, and bounded live providers when required. Use for user-path checks of sign-in, models, API keys, observability, usage costs, spend controls, batches, agent sessions, request logs, Anthropic Messages thinking and effort compatibility, MCP registry and tool access, OpenRouter routing, guardrails, and Skills storage.
features/skills.md
View on GitHub# Skills
Skills verification follows the real browser path from namespace selection and ZIP upload to version selection, download, and shared access. It compares displayed content with the production Skills API and uses native RustFS for archive storage.
## Sub-features
- `skills-bundled` imports the reviewed `bundled-skills/` copies into the demo admin namespace, skips an identical second import, verifies attribution and license content, and revokes the temporary user key.
- `skills-entry` redirects anonymous users to sign-in and exposes the `Agent Skills` page to a signed-in regular user.
- `skills-search` searches names, descriptions, and owner namespaces as text is entered, matches rows to the filtered API, and verifies no-match and cleared states.
- `skills-namespace` claims a permanent unique namespace and rejects changes or reuse by another user.
- `skills-upload` uploads a ZIP through the UI and confirms the owner, name, initial version, and digest through the API.
- `skills-preview` compares the displayed reference file with its saved API content.
- `skills-versions` appends a second immutable version, preserves the first default, checks default/latest labels in the version picker without a separate Status field, and changes the default through the UI.
- `skills-download` downloads the selected version through the browser and confirms its SHA-256 digest; the first version stays unchanged.
- `skills-ownership` allows duplicate skill names in separate namespaces, shares reads, hides another user's owner controls, and rejects writes from both another user and a platform administrator.
- `skills-authentication` rejects anonymous reads and confirms service-account reads while denying all service-account write paths.
- `skills-cleanup` deactivates synthetic users and removes the run's database, object prefix, private configuration, and test token during stack teardown.
## How to get to it (user POV)
- Open `/admin/skills` and sign in, or follow the `Agent Skills` sidebar link under `Agent Extensions`. The page heading is `Agent Skills`.
- Type into `Search skills` to search the full catalog by name, description, or owner namespace. Clear the input to restore the catalog; no submit button is required.
- Choose `Upload skill` to claim a namespace on first use and select a ZIP archive.
- Open the skill-name link in the matching Owner row to view instructions, versions, files, and download controls. The detail heading shows `namespace/skill-name`.
- Owners can use `Upload new version`, choose `Version`, and use `Set as default`.
## Driving it with control-oceans-admin
Preconditions:
- Set a fresh `OCEANS_VERIFY_RUN_ID`, free gateway/UI ports, and `OCEANS_VERIFY_SKILLS=true` before launch. Run all commands from the repository root.
- The `rustfs` mise profile must be installable and `lsof` must be available. Launch runs `mise -E rustfs run rustfs:setup`, which creates private local credentials, starts native RustFS, and ensures the Skills bucket exists.
- Use the default `gateway.yaml` or a `GATEWAY_CONFIG` with the `local-ci-runner` service account and its `primary` key. Launch creates a private copy with a synthetic service-account token, local object storage, and a separate `skills.db` beneath this run's state directory. It does not edit the source config or this checkout's `gateway.db`.
- `dev-stack` receives `OPENAI_API_KEY` and `OPENROUTER_API_KEY` from fnox and does not start until both resolve (see Launch in [SKILL.md](../SKILL.md)). For this Skills-only proof, synthetic placeholders are enough in a checkout without the real keys: set each one under a `[secrets]` table in the gitignored `fnox.local.toml`, as `KEY = { default = "synthetic" }`. fnox ignores top-level assignments. The driver sends no model request and does not prove provider authentication.
- Require `doctor` to pass. It checks the recorded gateway, UI, and RustFS listener PIDs and readiness. The existing checkout lock still prevents concurrent verification stacks.
```bash
export OCEANS_VERIFY_RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-skills-$$"
export OCEANS_VERIFY_GATEWAY_PORT=38090
export OCEANS_VERIFY_UI_PORT=33010
export OCEANS_VERIFY_SKILLS=true
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin launch
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin doctor
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin drive skills
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin evidence skills
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin cleanup
.agents/skills/verify-oceans-admin/scripts/control-oceans-admin evidence skills
```
- **Check the curated copy.** After bootstrap admin sign-in, the driver creates a temporary ordinary admin identity and its user-owned key and uses the same importer as `mise run skills:import-bundled`. It claims `demo-admin`, imports `grill-me` twice, and requires one stored version. The browser shows Matt Pocock attribution, the pinned GitHub link, the MIT license, file count, and `D Mon YYYY, HH:MM UTC` upload time. SHA-256 and archive byte counts stay out of the UI; API digest checks remain active. The driver revokes the key, retains the last ordinary admin for private-database cleanup, and captures `01b-skills-bundled.*`.
- **Enter and upload.** The driver opens the protected Skills route, waits for the `Agent Skills` heading, captures the sign-in state, then creates two synthetic regular users through the production invitation API. It signs in through the browser, follows the `Agent Skills` sidebar link, enters the unused owner namespace in `Search skills`, and confirms `No skills found`. `Upload skill` opens `Choose your skill namespace`; the `Namespace` input and `Claim namespace` button persist the namespace. `Upload a skill` accepts the tracked `review-code-v1.zip` fixture through `ZIP archive`.
- **Read and version.** The detail heading must contain the owner's `namespace/review-code`. `skill-instructions` must contain version 1 instructions. The `Skill files` navigation opens `references/checklist.md`; `skill-file-text` must exactly match the production API. The driver uploads version 2, confirms that default version 1 remains selected in saved metadata, and checks `Version 1 (default)` and `Version 2 (latest)` in the picker. No separate `Status` field is shown. It uses `Set as default` to save version 2. `Download ZIP` must return bytes with the API's digest. Version 1 instructions and archive digest must remain unchanged.
- **Check ownership.** A second user fails to claim the first namespace, then uploads the same skill name in a different namespace. That user can view the first skill, while `Upload new version` and `Set as default` are absent. The production API must deny append and default changes from both this user and the platform administrator with HTTP 403.
- **Search the catalog.** The driver searches `review-code`, the description phrase `repeatable checklist`, and the owner's namespace. It verifies both duplicate names by their Owner cells and skill-ID link destinations. Each search updates the `q` URL parameter and matches `GET /api/v1/skills?q=...&limit=50&offset=0`. An unmatched query shows no results; clearing it restores the bundled and uploaded skills. The query and result IDs are saved under `details.catalog.search` in `skills-proof.json`.
- **Check caller types.** Anonymous catalog and archive requests must return HTTP 401. The synthetic service-account key receives HTTP 200 on reads and HTTP 403 for create, append, namespace claim, and default changes. Its value stays in the private run directory and process environment; proof files contain no credentials.
- **Retain proof and clean up.** `skills-proof.json` is the shared envelope with one check per step plus `user-cleanup`; `evidence skills` requires `verdict: pass`. Screenshots and ARIA snapshots record entry, empty catalog, preview, both versions, the default change, another owner's view, and the final catalog. Driver cleanup deactivates regular users and revokes the temporary import key. The last ordinary admin stays in the private database until stack cleanup removes it, as required by the existing admin lifecycle guard. Stack cleanup removes only `skills-verify/<checkout-hash>/<run-id>/` from RustFS, confirms it is empty, removes the private database/config/token, and adds a `storage-cleanup` check to `skills-proof.json`. It stops RustFS only when this run started the recorded process. A failed storage cleanup records that check as `fail`, so `evidence skills` after teardown fails too.
## Gotchas
- The Skills launch uses a run-local database because Skills has no deletion API. Ordinary verification launch still uses this checkout's `gateway.db`.
- The RustFS bucket, local credentials, and data directory persist. Other object prefixes are outside this run's cleanup scope. A RustFS daemon that was already running stays running after cleanup.
- Always run cleanup after a failed launch or driver. A storage cleanup failure retains its recorded scope for a retry and returns a failure status.
- This proof uses real gateway, database, UI, and RustFS processes. It does not prove AWS S3 account permissions, Kubernetes/CSI deployment, Git import, or installed CLI behavior.
- A change to `crates/oceans-cli/src/skills.rs` plans the separate `skills-cli` proof. Run `set -o pipefail; mise -E rustfs run rustfs:e2e 2>&1 | tee "/tmp/oceans-admin-verification/$OCEANS_VERIFY_RUN_ID/evidence/skills-cli-e2e.log"` for the CLI round trip, and record `cli-round-trip=pass` only if that command exits 0: `control-oceans-admin record skills-cli --check cli-round-trip=pass --artifact skills-cli-e2e.log`.
- The service-account check uses the production configuration seed boundary. This proof does not create a provider request or change real user credentials.
- Preserve the run ID and evidence directory in the report. A health check or final screenshot alone is not proof of saved Skills behavior.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |