kklimuk/docx-cli / .claude/skills/weak-agent-test
weak-agent-test
VerifiedRun the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process.
scripts/collect-exercises.ts
View on GitHub#!/usr/bin/env bun
/**
* Deterministically collect the local corpus runner's per-scenario exercise.json
* results into the `args.exercises` array the weak-agent-test workflow accepts INLINE.
*
* Why this exists: the workflow's JS is sandboxed with no filesystem access, so the
* only way it can obtain the local results is (a) an in-workflow LOAD agent — an LLM
* that reads the files and hands them back, or (b) `args.exercises` passed in from the
* caller. This script is path (b): it reads the files with plain code, so the
* code-computed `status` (completed | failed) and the rest of the account reach the
* workflow verbatim instead of being laundered through a model. Pass its output as
* `args.exercises` and the workflow skips the LOAD agent entirely.
*
* Only the fields the workflow consumes are emitted (status/completed, summary,
* deadEnds, frictions, outputPath) — small and low-risk to pass through. The big
* `docxCommands` / `_local` ledger blocks stay on disk in exercise.json, which is the
* deterministic source of truth for anything that needs them.
*
* Usage: collect-exercises.ts <runDir> [key...]
* Prints a JSON array to stdout — pass it verbatim as the workflow's args.exercises.
*/
const [runDir, ...keys] = Bun.argv.slice(2);
if (!runDir) {
await Bun.write(Bun.stderr, "Usage: collect-exercises.ts <runDir> [key...]\n");
process.exit(2);
}
const wanted = new Set(keys);
const exercises: Record<string, unknown>[] = [];
for await (const rel of new Bun.Glob("*/exercise.json").scan({ cwd: runDir })) {
const key = rel.slice(0, rel.indexOf("/"));
if (wanted.size && !wanted.has(key)) continue;
try {
const entry = await Bun.file(`${runDir}/${rel}`).json();
exercises.push({
key: entry.key ?? key,
// `status` is the code-computed process lifecycle (new files); `completed`
// is only present on files produced before the switch. JSON.stringify drops
// whichever is undefined.
status: entry.status,
completed: entry.completed,
summary: entry.summary,
deadEnds: entry.deadEnds,
frictions: entry.frictions,
outputPath: entry.outputPath,
});
} catch (error) {
await Bun.write(
Bun.stderr,
`[${key}] skipped (unreadable exercise.json): ${error}\n`,
);
}
}
// Stable order for reproducibility.
exercises.sort((a, b) =>
String(a.key) < String(b.key) ? -1 : String(a.key) > String(b.key) ? 1 : 0,
);
await Bun.write(Bun.stdout, JSON.stringify(exercises));
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |