Adonis0123/adonis-skills / skills/agent-skill-evaluation
agent-skill-evaluation
FlaggedUse when comparing or trial-testing agent skills before installing — including GitHub/skills.sh URLs, 测试一下/值不值得装, package screening, and A/B artifact runs.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 30.3 KB |
| references/conceptual-multi-agent-skill-trial.md | markdown | 4.0 KB |
| references/image-producing-skill-trial.md | markdown | 2.7 KB |
| references/mattpocock-catalog-renames.md | markdown | 2.7 KB |
| references/mattpocock-shortlist-20260807.md | markdown | 3.2 KB |
| references/mattpocock-skills-screening.md | markdown | 15.3 KB |
| references/mattpocock-tdd-trial-20260807.md | markdown | 2.8 KB |
| references/mattpocock-tdd-trial-20260908.md | markdown | 2.5 KB |
| references/mattpocock-to-prd-trial-20260807.md | markdown | 1.8 KB |
| references/mattpocock-wayfinder-trial-20260909.md | markdown | 2.3 KB |
| references/output-style-skill-screening.md | markdown | 6.0 KB |
| references/peer-review-cli.md | markdown | 5.4 KB |
| references/single-skill-temp-trial.md | markdown | 7.7 KB |
| references/skill-eval-harness-trial.md | markdown | 3.6 KB |
| references/tdd-process-skill-trial.md | markdown | 2.9 KB |
| references/ui-design-ab-testing.md | markdown | 5.8 KB |
references/image-producing-skill-trial.md
View on GitHub# Image / video-producing skill trial
Use on the single-skill URL / 「测试一下」route when the candidate's core job is **generate bitmaps, then assemble them** (image-generation jobs feeding a renderer, story → illustrated video, page-flip from uploaded pages, Remotion-style compositions after master images).
Local specifics (accounts, proxies, the owner's own trial records): `references/local-image-producing-skill-trial.md` (if present).
## Hard gate: do not score from plan-only output
A storyboard JSON, caption split, job list, or `--mode plan` run is **not** evidence the skill works. The exercise is incomplete until all of these exist:
1. **Masters on disk** — the scene master images the skill expects (e.g. `*_master.png`) or the uploaded pages, at the exact paths the skill reads.
2. **Import / derive succeeds** — the skill's own import step turns the masters into its working plates or assets (e.g. black-and-white / color variants) without manual patching.
3. **A real exported video** — a preview or final MP4 rendered by the skill's pipeline, verified with `ffprobe` (duration and width × height match what the skill promised).
Plan-only = incomplete trial. Say so in the verdict rather than scoring the plan.
## Acceptance steps (minimum)
1. Temp clone under `/tmp/<skill>-eval-YYYYMMDD/`; install deps there (e.g. `npm ci`). No global skill install.
2. Run the plan step once to lock scene count and captions.
3. Generate the masters with the generator the skill documents (or the one the user names), and place them at the skill's exact output paths; create the directories first.
4. Run the skill's import step, then its preview/render step.
5. `ffprobe` the MP4 and record duration + resolution; keep a few stills.
6. Deliver the MP4 (with a cover/still) and the stills where the user will actually look, not just a path in text.
## Generator vs. skill
- Many such skills default to an agent's **built-in** image tool and keep a script as an explicit API fallback that needs a real provider key. An API proxy or wrong key failing is an **environment** problem, not proof the skill is broken, and not a reason to skip the visuals.
- If the user names a generator mid-eval ("use Codex / Grok to make the images"), switch to it immediately instead of continuing to probe the failing API.
- Do not retry a model the endpoint has already rejected once; skip master files that already exist and look complete.
## Intervention cost
Record generator steering separately from renderer quality. Heavy "wrong API, then another generator" cost belongs to the environment and should not inflate or deflate the skill's daily-fit score. Default verdict stays **C** (temporary-only) unless the job recurs often enough to justify a single install.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |