Skip to content

kklimuk/docx-cli / .claude/skills/weak-agent-test

weak-agent-test

Verified

Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process.

scripts/collect-exercises.ts

View on GitHub
#!/usr/bin/env bun
/**
 * Deterministically collect the local corpus runner's per-scenario exercise.json
 * results into the `args.exercises` array the weak-agent-test workflow accepts INLINE.
 *
 * Why this exists: the workflow's JS is sandboxed with no filesystem access, so the
 * only way it can obtain the local results is (a) an in-workflow LOAD agent — an LLM
 * that reads the files and hands them back, or (b) `args.exercises` passed in from the
 * caller. This script is path (b): it reads the files with plain code, so the
 * code-computed `status` (completed | failed) and the rest of the account reach the
 * workflow verbatim instead of being laundered through a model. Pass its output as
 * `args.exercises` and the workflow skips the LOAD agent entirely.
 *
 * Only the fields the workflow consumes are emitted (status/completed, summary,
 * deadEnds, frictions, outputPath) — small and low-risk to pass through. The big
 * `docxCommands` / `_local` ledger blocks stay on disk in exercise.json, which is the
 * deterministic source of truth for anything that needs them.
 *
 * Usage: collect-exercises.ts <runDir> [key...]
 *   Prints a JSON array to stdout — pass it verbatim as the workflow's args.exercises.
 */

const [runDir, ...keys] = Bun.argv.slice(2);
if (!runDir) {
	await Bun.write(Bun.stderr, "Usage: collect-exercises.ts <runDir> [key...]\n");
	process.exit(2);
}

const wanted = new Set(keys);
const exercises: Record<string, unknown>[] = [];
for await (const rel of new Bun.Glob("*/exercise.json").scan({ cwd: runDir })) {
	const key = rel.slice(0, rel.indexOf("/"));
	if (wanted.size && !wanted.has(key)) continue;
	try {
		const entry = await Bun.file(`${runDir}/${rel}`).json();
		exercises.push({
			key: entry.key ?? key,
			// `status` is the code-computed process lifecycle (new files); `completed`
			// is only present on files produced before the switch. JSON.stringify drops
			// whichever is undefined.
			status: entry.status,
			completed: entry.completed,
			summary: entry.summary,
			deadEnds: entry.deadEnds,
			frictions: entry.frictions,
			outputPath: entry.outputPath,
		});
	} catch (error) {
		await Bun.write(
			Bun.stderr,
			`[${key}] skipped (unreadable exercise.json): ${error}\n`,
		);
	}
}
// Stable order for reproducibility.
exercises.sort((a, b) =>
	String(a.key) < String(b.key) ? -1 : String(a.key) > String(b.key) ? 1 : 0,
);
await Bun.write(Bun.stdout, JSON.stringify(exercises));
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall