jianshuo/claude-skills / wjs-evaling-voicedrop-prompts
wjs-evaling-voicedrop-prompts
VerifiedUse when 王建硕 wants to evaluate whether a change to VoiceDrop's 挖矿 system prompt is actually better than the live version — runs the local eval harness (golden fixtures × champion-vs-candidate, same input), dispatches blind pairwise judge subagents, aggregates a win-rate verdict, and on approval promotes the candidate into agent/src/prompts/mine.js. Triggers — "评估 prompt"、"挖矿 prompt 改好了吗"、"eval prompt"、"比一比两版 prompt"、"/wjs-evaling-voicedrop-prompts".
VoiceDrop 挖矿 prompt 评估
harness = 本 skill(协议)+ jianshuo.dev agent/eval/(脚本/数据)。运行时 = 本地 Claude Code。
被测对象 = agent/src/prompts/mine.js 的 MINE_SYSTEM(git 即版本库)。
何时用
用户改了挖矿 prompt(MINE_SYSTEM),想用数据判断改好了还是改坏了,而不是凭感觉看一两次。
流程(按序)
- 拿候选 prompt:把候选版
MINE_SYSTEM文本写到一个临时文件(如/tmp/cand-prompt.txt);冠军 = 当前mine.js的MINE_SYSTEM(脚本自动读)。 - 跑产出:
cd ~/code/jianshuo.dev/agent && CLAUDE_API_KEY=$CLAUDE_API_KEY node eval/run-eval.mjs /tmp/cand-prompt.txt <runId>。产出落eval/runs/<runId>/。先看终端有没有「确定性回退」警告——有就先停,多半是候选 prompt 破坏了 JSON 输出。 - 成对盲评:对每条 fixture,dispatch 一个 subagent,喂
references/judge-rubric.md+ 该 fixture 的 transcript + 两份产出。A/B 顺序随机(一半 fixture 把 candidate 放 A、一半放 B,记录映射,收到结果后还原成 champion/candidate)。裁判模型用与生成(opus)不同家族的模型。收每条的{winner, dims, reason}。 - 聚合:把还原后的
verdicts(winner ∈ candidate/champion/tie)+ 候选 proxyFails 喂aggregate(),渲染renderReport()→ 写eval/runs/<runId>/report.md。 - 人工终审:把胜负最接近、分歧最大的 1–2 条产出并排摆给用户。机器只筛掉明显更差的,文风最后一票是用户。
- 晋级:仅当
decision==="promote"(胜率 ≥70% 且无回退)且用户点「认可」——把候选写回agent/src/prompts/mine.js的MINE_SYSTEM,commit(message 附 runId 与胜率),并跑npm test确认没破坏。否则保留报告、不动生产版。
边界
- 只评挖矿 prompt(
MINE_SYSTEM/MINE_SYSTEM_FORCE)。审核/语音编辑 prompt 是不同 eval 模式,不在本 skill。 - 不测成本/缓存/延迟(本地缓存行为≠生产);不做无人值守。
- 真实金标集要 ≥10 条才可信(见
agent/eval/fixtures/README.md的补充流程);种子 2 条只够自测流程。
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |