Skip to content

stark-ai-de/agent-skills / incubator/skills/skill-maintenance/skillopt-setup

skillopt-setup

Flagged

Set up Microsoft SkillOpt for Agent Skills improvement. Use when installing SkillOpt, preparing per-skill train/val/test eval splits, configuring Codex CLI or provider-backed execution, running SkillOpt against SKILL.md bodies, reviewing best_skill.md, or importing optimized skill changes. Do not use for ordinary skill authoring or manual repo validation.

FileKindSize
SKILL.mdskill md12.9 KB
assets/agent-skills-benchmark/README.mdmarkdown1.7 KB
assets/agent-skills-benchmark/adapter.py.templateother14.4 KB
assets/agent-skills-benchmark/codex_cli_reflector.py.templateother27.5 KB
assets/agent-skills-benchmark/config.codex-cli-all.yamlconfig1.5 KB
assets/agent-skills-benchmark/config.hybrid-codex-target.yamlconfig1.7 KB
assets/agent-skills-benchmark/config.native-provider.yamlconfig1.3 KB
assets/agent-skills-benchmark/dataloader.py.templateother6.6 KB
assets/agent-skills-benchmark/evaluator.py.templateother62.5 KB
assets/agent-skills-benchmark/prompts/analyst_error.mdmarkdown193 B
assets/agent-skills-benchmark/prompts/analyst_error_full_rewrite.mdmarkdown215 B
assets/agent-skills-benchmark/prompts/analyst_success.mdmarkdown183 B
assets/agent-skills-benchmark/prompts/analyst_success_full_rewrite.mdmarkdown157 B
assets/agent-skills-benchmark/prompts/codex_cli_reflect_failure.mdmarkdown202 B
assets/agent-skills-benchmark/prompts/codex_cli_reflect_success.mdmarkdown186 B
assets/agent-skills-benchmark/rollout.py.templateother143.7 KB
assets/agent-skills-benchmark/schemas/codex_cli_patch.schema.jsonconfig1.2 KB
assets/agent-skills-benchmark/schemas/codex_cli_rollout.schema.jsonconfig854 B
references/adapter-contract.mdmarkdown5.8 KB
references/adoption-policy.mdmarkdown1.0 KB
references/codex-cli-runner.mdmarkdown3.3 KB
references/credential-modes.mdmarkdown5.8 KB
references/data-schema.mdmarkdown7.7 KB
references/local-artifact-audit.mdmarkdown1.9 KB
references/local-openai-gateway.mdmarkdown5.1 KB
references/official-best-practices.mdmarkdown4.1 KB
references/runbook.mdmarkdown20.0 KB
references/troubleshooting.mdmarkdown5.8 KB
scripts/apply-skillopt-best.mjsscript13.9 KB
scripts/audit-skillopt-local-artifacts.mjsscript22.0 KB
scripts/check-skillopt-readiness.mjsscript99.6 KB
scripts/codex-local-openai-chat-gateway.mjsscript51.6 KB
scripts/prepare-local-skillopt-adapter.mjsscript25.4 KB
scripts/prepare-skillopt-split.mjsscript22.9 KB
scripts/probe-codex-cli.mjsscript4.3 KB
scripts/probe-openai-compatible-endpoint.mjsscript7.9 KB
scripts/setup-skillopt-local.mjsscript40.9 KB
scripts/summarize-skillopt-run.mjsscript17.9 KB
scripts/verify-skillopt-run-artifacts.mjsscript7.4 KB

references/adapter-contract.md

View on GitHub
# Agent Skills Adapter Contract

The local adapter is copied into `.agents/tools/SkillOpt/skillopt/envs/agent_skills/`. It is not committed after expansion into the SkillOpt clone.

`prepare-local-skillopt-adapter.mjs` also creates `.agents/skillopt-work/<skill>/_base_/default.yaml` for generated work configs, writes `.agents/skillopt-work/<skill>/adapter-manifest.json`, and patches known local SkillOpt `scripts/train.py` and `scripts/eval_only.py` registries so `env.name: agent_skills` can instantiate the adapter. Registry patches are isolated to the ignored local SkillOpt clone.

## Files

- `adapter.py`: coordinates dataset loading, rollout, scoring, and reflection.
- `dataloader.py`: loads split JSON and returns deterministic batches.
- `rollout.py`: executes one case in an isolated workspace.
- `evaluator.py`: scores Agent Skill responses. Native-provider mode uses the configured provider optimizer path as a semantic judge; Codex CLI modes use a read-isolated Codex-CLI judge through the user's local login. Deterministic and visual checks remain hard gates before either semantic judge.
- `codex_cli_reflector.py`: implements exploratory Codex CLI patch reflection.

## Required Behavior

- Optimize only the skill body by default.
- Preserve frontmatter outside explicit trigger-description work.
- Route `codex_exec` targets through the read-isolated local Codex runner and supported HTTP provider chat targets through installed SkillOpt `skillopt.model.chat_target`. Fail closed for `claude_chat`, which invokes a local host-readable CLI in upstream SkillOpt rather than a provider HTTP boundary, and for `claude_code_exec`, which this rollout adapter does not implement. Treat `minimax_chat` as target-only because installed SkillOpt has no MiniMax optimizer branch.
- Match the installed SkillOpt `EnvAdapter` lifecycle: constructor kwargs or `config` can come from the flattened config, `setup(cfg)` initializes splits, `get_dataloader()` returns a batch planner, and rollout/reflection methods accept the trainer's batch signatures.
- Return rollout records with `id`, `hard`, `soft`, `prediction`, `assertion_results`, and `task_type`.
- Require semantic judging for official-parity runs: `judge_backend: provider` uses the configured supported optimizer provider, while `judge_backend: codex_cli` uses the verified read-isolated Codex judge. Keep the literal-match heuristic exploratory-only.
- Keep provider targets and non-visual Codex target rollouts text-only. Provider chat targets must fail closed on active `visual_assertions`; use the generated text-only split or an artifact-capable Codex target with `tool_rollout_for_visual_assertions` enabled.
- Inline only a bounded, UTF-8, secret-sanitized snapshot of seeded textual helpers and fixtures for text-only targets. Fail closed when a required fixture is binary or exceeds the per-file/total context limits.
- Enforce a strict Codex permission profile for every local target rollout. Non-visual cases receive minimal runtime reads only, no workspace read/write grant, a protected host-written final-output path, no network, and a minimal shell environment.
- For `visual_assertions` cases, enforce a strict Codex permission profile with minimal platform reads, a narrow read-only grant for the resolved Codex executable/package, write access only to the rollout workspace, a denied control-output directory, and network disabled. Within that boundary, allow shell/file operations only for copied helper scripts, draw.io XML, deterministic validators, and requested PNG/SVG artifacts. Browser, hosted, install, network, and out-of-workspace access remain forbidden.
- Reject symlink artifacts and enforce file-count, byte, dimension, decoded-pixel, and decompression limits during trusted post-processing so generated artifacts cannot reintroduce host reads or exhaust the trainer.
- If `require_drawio_cli_for_visual_rollouts` is enabled and neither `drawio` nor `diagrams.net` is on PATH, return a fast-fail rollout with `visual_rollout_blocker` instead of starting nested Codex.
- Before any Codex-backed target, judge, or reflection handoff, require readiness to run its bounded `--strict-config` parse probe with the real permission profile. The probe deliberately stops on a missing local output schema before a session or network/model call can start. Help text alone is not proof, and a text-only split bypasses draw.io readiness only, never Codex read isolation.
- Generated configs must honor `visual_eval_policy`: `auto` uses the full split only for an artifact-capable `codex_exec` target with render tooling and uses `data-text-only` for provider targets or missing render tooling; `full` keeps visual cases active and therefore blocks provider targets; `text-only` always uses the generated text-only split.
- In `codex-cli-all`, judge expected behavior semantically with `codex exec`; do not require OpenAI API keys or inspect Codex auth material.
- In `native-provider`, set `judge_backend: provider` and judge expected behavior semantically through `skillopt.model.chat_optimizer`; do not classify literal-substring heuristic judging as official parity.
- In `codex-cli-all`, return at most one reflected raw patch capped to the edit budget, so SkillOpt's aggregate/select stages use deterministic fallbacks and do not call provider-backed optimizer ranking or merging.
- Reject reflection patches that include frontmatter, secrets, raw auth paths, whole-skill rewrites in patch mode, or unsupported dependencies.

## Drift Handling

SkillOpt source shape can change. The preparer records a manifest, template checksums, and local registry/config patch status. It must refuse silent registry edits when the local clone does not match known patterns. Readiness must block training when the manifest is missing, legacy/global, for another target/mode/profile, checksum-mismatched, or records anything other than `registry_patch.status: ready`.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall