Skip to content

stark-ai-de/agent-skills / incubator/skills/skill-maintenance/skillopt-setup

skillopt-setup

Flagged

Set up Microsoft SkillOpt for Agent Skills improvement. Use when installing SkillOpt, preparing per-skill train/val/test eval splits, configuring Codex CLI or provider-backed execution, running SkillOpt against SKILL.md bodies, reviewing best_skill.md, or importing optimized skill changes. Do not use for ordinary skill authoring or manual repo validation.

FileKindSize
SKILL.mdskill md12.9 KB
assets/agent-skills-benchmark/README.mdmarkdown1.7 KB
assets/agent-skills-benchmark/adapter.py.templateother14.4 KB
assets/agent-skills-benchmark/codex_cli_reflector.py.templateother27.5 KB
assets/agent-skills-benchmark/config.codex-cli-all.yamlconfig1.5 KB
assets/agent-skills-benchmark/config.hybrid-codex-target.yamlconfig1.7 KB
assets/agent-skills-benchmark/config.native-provider.yamlconfig1.3 KB
assets/agent-skills-benchmark/dataloader.py.templateother6.6 KB
assets/agent-skills-benchmark/evaluator.py.templateother62.5 KB
assets/agent-skills-benchmark/prompts/analyst_error.mdmarkdown193 B
assets/agent-skills-benchmark/prompts/analyst_error_full_rewrite.mdmarkdown215 B
assets/agent-skills-benchmark/prompts/analyst_success.mdmarkdown183 B
assets/agent-skills-benchmark/prompts/analyst_success_full_rewrite.mdmarkdown157 B
assets/agent-skills-benchmark/prompts/codex_cli_reflect_failure.mdmarkdown202 B
assets/agent-skills-benchmark/prompts/codex_cli_reflect_success.mdmarkdown186 B
assets/agent-skills-benchmark/rollout.py.templateother143.7 KB
assets/agent-skills-benchmark/schemas/codex_cli_patch.schema.jsonconfig1.2 KB
assets/agent-skills-benchmark/schemas/codex_cli_rollout.schema.jsonconfig854 B
references/adapter-contract.mdmarkdown5.8 KB
references/adoption-policy.mdmarkdown1.0 KB
references/codex-cli-runner.mdmarkdown3.3 KB
references/credential-modes.mdmarkdown5.8 KB
references/data-schema.mdmarkdown7.7 KB
references/local-artifact-audit.mdmarkdown1.9 KB
references/local-openai-gateway.mdmarkdown5.1 KB
references/official-best-practices.mdmarkdown4.1 KB
references/runbook.mdmarkdown20.0 KB
references/troubleshooting.mdmarkdown5.8 KB
scripts/apply-skillopt-best.mjsscript13.9 KB
scripts/audit-skillopt-local-artifacts.mjsscript22.0 KB
scripts/check-skillopt-readiness.mjsscript99.6 KB
scripts/codex-local-openai-chat-gateway.mjsscript51.6 KB
scripts/prepare-local-skillopt-adapter.mjsscript25.4 KB
scripts/prepare-skillopt-split.mjsscript22.9 KB
scripts/probe-codex-cli.mjsscript4.3 KB
scripts/probe-openai-compatible-endpoint.mjsscript7.9 KB
scripts/setup-skillopt-local.mjsscript40.9 KB
scripts/summarize-skillopt-run.mjsscript17.9 KB
scripts/verify-skillopt-run-artifacts.mjsscript7.4 KB

references/official-best-practices.md

View on GitHub
# Official Best Practices Gap Notes

Use this when deciding whether a local Agent Skills run is `official-parity` or `exploratory`.

## Wizard Principle

Guide users toward the best configuration by asking for one setup decision at a time:

1. target skill,
2. cleanup or reuse of the existing `.agents/` setup,
3. no-provider exploration vs official-parity proof,
4. Python setup with `uv` preferred,
5. data floor readiness,
6. provider credentials and model pins when official parity is requested,
7. dry-run vs production setup,
8. new-terminal training handoff after setup.

Do not present production setup or training commands during the dry-run step. Do not claim official parity until credentials, model pins, data floor, config schema, and run artifacts are verified.

## Official-Parity Profile

Use `official-parity` only when the optimizer path is provider-backed:

- `native-provider`: provider-backed target rollout and optimizer/reflection.
- `hybrid-codex-target`: Codex CLI target rollout and judging, with provider-backed optimizer/reflection.

Expected setup properties:

- Provider credentials are present for native optimizer/reflection.
- Validation gate and test evaluation are enabled.
- Slow update and meta skill are enabled.
- Learning-rate schedule is `cosine`.
- Edit budget is close to the official default: `learning_rate: 4`, `min_learning_rate: 2`.
- Model choices are pinned or the summary explicitly records inherited defaults.
- Target data has enough positive cases and held-out validation/test cases to avoid overfitting.
- `use_skill_aware_reflection` is optional and experimental; leave it off by default unless the user explicitly wants EmbodiSkill-style appendix routing and the proof report records the extra surface.

Recommended data floor for an official-parity claim:

- 20 or more positive optimization cases.
- 5 or more validation cases.
- 5 or more test cases.
- Deterministic assertions or fixture-backed checks where possible.

## Exploratory Profile

Use `exploratory` for:

- `codex-cli-all`.
- tiny datasets.
- local smoke runs.
- missing model pins.
- setup-only validation before provider credentials are available.

`codex-cli-all` is useful because it avoids provider credentials by using the user's Codex login for rollouts, semantic judging, and adapter-managed reflection. It is not upstream-native official optimizer parity because provider-backed reflection, aggregation, ranking, slow update, and meta skill are bypassed. Keep slow update and meta skill disabled in this mode; enabling either uses upstream `chat_optimizer` and is no longer a provider-free run.

## Reporting Requirements

Readiness and summaries should report:

- proof status and proof blockers,
- run profile,
- official-parity status,
- official-parity gaps,
- dataset split counts,
- deterministic assertion, fixture, and expected-artifact coverage,
- model pins or inherited defaults,
- SkillOpt commit and config schema check result,
- generated config defaults,
- registry patch status,
- slow update, meta skill, and skill-aware-reflection status,
- expected local artifacts such as `config.json`, `history.json`, `runtime_state.json`, and `best_skill.md`.

## Proof Checklist

Before calling a run official-parity proof:

- Re-check the local SkillOpt clone, `scripts/train.py --help`, and `scripts/eval_only.py --help` against generated config keys.
- Use a fresh output directory unless the goal is explicitly to test resume behavior.
- Require provider credentials and explicit optimizer/target/judge model pins for provider-backed proof modes.
- Verify `config.json`, `history.json`, `runtime_state.json`, `best_skill.md`, `steps/`, and `skills/`.
- If slow update or meta skill is enabled, verify the corresponding output directory or record why it was skipped.
- If skill-aware reflection is enabled, verify appendix artifacts/markers and record that the run used the experimental appendix surface.
- Run or separately schedule eval-only on `best_skill.md`.
- Check whether optional WebUI support is importable before recommending it as available.
- Publish only curated summaries under `skill-evals/<target>/runs/`; keep raw `.agents/` logs and trajectories ignored.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall