tzachbon/srulik-toolkit / plugins/srulik-toolkit/skills/skill-creator
skill-creator
VerifiedCreate or edit a skill, improve its instructions or triggers, evaluate it on real tasks, or benchmark a revision. Every skill creation or edit uses create-plan and implement-plan, including small changes; evaluation compares actual outputs and includes human review.
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 11.1 KB |
| SKILL.md | skill md | 4.2 KB |
| agents/analyzer.md | markdown | 10.3 KB |
| agents/grader.md | markdown | 9.1 KB |
| agents/openai.yaml | config | 118 B |
| assets/eval_review.html | other | 6.8 KB |
| eval-viewer/generate_review.py | script | 7.2 KB |
| eval-viewer/viewer.html | other | 40.7 KB |
| references/authoring.md | markdown | 2.8 KB |
| references/claude-code.md | markdown | 3.9 KB |
| references/codex.md | markdown | 2.1 KB |
| references/evaluation.md | markdown | 3.9 KB |
| references/hosts.md | markdown | 861 B |
| references/oh-my-pi.md | markdown | 1.2 KB |
| references/schemas.md | markdown | 3.9 KB |
| references/t3-code.md | markdown | 2.2 KB |
| scripts/__init__.py | script | 0 B |
| scripts/aggregate_benchmark.py | script | 9.3 KB |
| scripts/generate_report.py | script | 12.5 KB |
| scripts/improve_description.py | script | 11.3 KB |
| scripts/run_eval.py | script | 12.6 KB |
| scripts/run_loop.py | script | 13.9 KB |
| scripts/utils.py | script | 3.0 KB |
references/evaluation.md
View on GitHub# Evaluate and improve
Adapted from Anthropic's skill-creator at
`ca45fc496a3436251f191354091d557da31c15b5`; modified for comparable runs,
truthful measurements, and static review. See [license](../LICENSE.txt).
## Prepare
Use the approved authoring plan and its iteration limit. Resolve the actual
skill path/hash, fixture inputs, expected outputs, provider/model/settings,
permissions, and output workspace. Keep evaluation output outside installed
skill caches and outside the source tree unless durable evidence is requested.
Use [schemas](schemas.md) as the single artifact contract.
For creation compare `with_skill` with `without_skill`: the baseline is a fresh
session with no target skill. For improvement preserve an immutable snapshot of the target before any
edit and compare `with_skill` with `old_skill`. Do not reuse the authoring
conversation as an independent baseline. Use identical inputs, provider,
model, settings and permissions within each pair. A provider/model change is a
new comparison, not a way to turn a failed run into a pass.
## Run and grade
Give every run the prompt, input paths, resolved skill (or explicitly none),
output directory, and expected artifacts. Start paired runs together when
capacity permits; otherwise run fresh isolated sessions serially. Keep child
settings within the parent's authority and record the actual native mapping.
If independent sessions are unavailable, disclose the limitation and do not
claim a comparative benchmark.
Use `iteration-N/eval-ID-name/CONFIG/run-N/outputs/`; keep one shared
`eval_metadata.json` in the eval directory. Save grades beside each `outputs/`
with evidence from actual files. Capture timing/tokens when the host supplies
them; omit unavailable fields. A measured zero is valid, an invented zero is
not. Use scripts for deterministic assertions and [grader](../agents/grader.md)
for judgment requiring evidence. Never grade only the final assistant claim.
From this skill's directory:
```sh
python3 -m scripts.aggregate_benchmark /absolute/path/iteration-1 --skill-name target-name
python3 eval-viewer/generate_review.py /absolute/path/iteration-1 \
--skill-name target-name --benchmark /absolute/path/iteration-1/benchmark.json \
--static /absolute/path/review.html
```
The helper uses equal fixture weights after averaging repetitions within each
fixture/configuration. Counts remain visible. Time/token deltas require the
same measured fixture coverage. Variation across fixture means is not
repetition-level uncertainty. Incomplete pairs or invalid grades fail visibly;
repair missing evidence rather than accepting a partial complete-looking run.
## Review before revising
Use the bundled generator, not a custom replacement viewer. It writes a static,
offline file, starts no server, and never stops another process. Present the
actual artifact through the host's supported file/HTML surface. The reviewer
can inspect Outputs and Benchmark and export completed feedback as JSON.
Validate completed feedback against the current run IDs before consuming it:
```sh
python3 eval-viewer/generate_review.py /absolute/path/iteration-1 \
--feedback /absolute/path/feedback.json
```
Synthetic feedback proves the round trip only. Record actual human acceptance
or requested changes separately. Do not treat blank unsubmitted feedback as
acceptance or claim to have opened an artifact that was merely written.
For covered revisions, execute within the approved plan through
`implement-plan`, produce a new `iteration-N`, rerun affected actual tasks and
baselines, and show the new viewer with `--previous-workspace`. New scope or an
exhausted iteration limit returns to `create-plan` before editing. Read
[analyzer](../agents/analyzer.md) for non-discriminating assertions, variance,
and cost/quality tradeoffs. A tie is a valid observation; never invent a gain.
Before delivery, `implement-plan` invokes `review-pro-max` on authored changes.
Return artifacts, actual evidence, feedback provenance, and limitations.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |