Skip to content

kklimuk/docx-cli / .claude/skills/weak-agent-test

weak-agent-test

Verified

Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process.

scenarios/contract-markup/criteria.md

View on GitHub
# Grading rubric — contract-markup (JUDGE ONLY)

## Pass conditions

1. **Changes are tracked** — the clause edits exist as real tracked revisions
   (`docx track-changes list` is non-empty). Either route counts: the document-level
   toggle (`track-changes on`) OR per-call `--track` on each edit. The document's
   toggle being OFF at the end is NOT a failure by itself — what matters is that the
   wording changes are visible redlines.

2. **≥ 3 tracked text edits or deletions** — at least three of the following egregious
   terms are revised as tracked changes (visible redlines, not silent edits):
   - §6 Non-Competition: five-year / worldwide scope reduced or struck
   - §9 Limitation of Liability: the asymmetric $100 cap made mutual or raised
   - §3 Compensation: Net-90 changed to Net-30 (or similar)
   - §8 Indemnification: Company's-own-negligence carve-out added

   Verified by: `docx track-changes list` shows ≥ 3 entries of type `insertion` or
   `deletion` touching the relevant clause paragraphs.

3. **§9 line spacing increased** — the §9 Limitation of Liability paragraph's line
   spacing is opened up (to roughly 1.5, and/or extra space added after). Tracking
   this formatting fix is OPTIONAL: a human reviewer applies house-style and
   legibility fixes directly and redlines only the wording, so an untracked spacing
   change (`docx read --ast` shows `spacing.line` ≈ 360 with `lineRule` auto, or a
   `docx:p … line-spacing="1.5"` hint) passes exactly like a tracked one (a
   `track-changes list` entry whose `current` spacing exceeds its `prior`).
   Restructuring §9's wording or leaving it cramped does not satisfy this — the
   spacing on that paragraph must actually increase.

4. **≥ 4 anchored comments** — comments attached to text in at least four of:
   - §4 IP over-assignment
   - §11 Personal guarantee
   - §12 Company-selected arbitration / one-way fee-shifting
   - §7 Waiver of independent counsel
   - §13 Unilateral amendment by URL posting

   Verified by: `docx comments list` returns ≥ 4 entries, each with a non-empty
   `anchor` field pointing to text in the relevant clause.

5. **IP sub-points reformatted to lowercase Roman numerals** — the lettered
   sub-points under the Intellectual Property clause (the level-1 list items: "all
   work product…", "all intellectual property…", "all of the Contractor's
   pre-existing…", "all feedback…") are switched from lower-alpha (a/b/c/d) to
   lower-roman (i/ii/iii/iv). The top-level clause numbering is left untouched.

   Verified by: `docx read contract.docx --ast` shows those level-1 list
   paragraphs with `list.format` = `"lower-roman"` (the natural path is
   `docx lists set --at <sub-point pN> --format lower-roman`). Note the markdown
   body renders ordered items as decimal regardless, so the `--ast` `list.format`
   (or a `<!-- docx:list … format="lower-roman" -->` hint) is the signal, not the
   rendered glyph.

## How to verify

```
# 1. Check tracked changes (the clause redlines; the §9 spacing may or may not be here)
docx track-changes list contract.docx

# 2. Check anchored comments
docx comments list contract.docx

# 3. Spot-check §9 for the line-spacing increase — tracked (a spacing entry whose
#    values go up, e.g. "spacing.line ·→360") OR applied directly (the docx:p hint)
docx track-changes list contract.docx | grep -iE 'format|spacing'
docx read contract.docx | grep -E 'line-spacing='

# 4. Confirm the IP sub-points are lower-roman (level-1 list items)
docx read contract.docx --ast | grep -o '"format":"lower-roman"'
```

All five conditions must pass for the scenario to be scored green.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall