Skip to content

moonlight-lupin/agent-skills / plugins/skill-governance/skills/contrastive-refinement

contrastive-refinement

Verified

Use when a skill or harness edit needs evidence before it lands — probe candidates on the same task, compose from contrasts, keep lessons with evidence.

FileKindSize
SKILL.mdskill md15.5 KB
references/harness2-adaptation.mdmarkdown3.3 KB
scripts/contrast.pyscript6.2 KB

references/harness2-adaptation.md

View on GitHub
# Harness² Adaptation Notes

Source: https://github.com/google-research/harness2
Paper: arXiv:2610.14224 (Xu et al., 2026). License: Apache-2.0.
Trial commit: `1cfe254` (main, 2026-10-14). Bounded trial run 2026-10-19:
`harness2-demo` passed (2 tasks, 0 failed) on managed Python 3.12.

## What was ported (concepts, not code)

| # | Concept | Upstream form | Ported form |
|---|---------|---------------|-------------|
| 1 | Propose–Probe–Compose | improver model edits a harness tree; probe rollouts on the same task; composition from behavioral contrasts | candidate patches to a skill dir; probe runs via parallel subagents on one fixed task; compose from blind-deliverable contrasts |
| 2 | No-verifier contrast | selection works without ground-truth scores by comparing deliverables from identical inputs | deliverable comparison plus a `<why>` line naming the contrast; baseline preferred when nothing differs |
| 3 | Blinded selection | labels sorted, seeded shuffle from `sha1(task|k|mode)[:12]`, anon ids A..Z, AA.. (`core/selection.py`) | `scripts/contrast.py ids` — same algorithm, same id alphabet |
| 4 | Evidence-required lessons | `EXPERIENCE.md` refuses entries without a cue/lesson/evidence/confidence block (`core/library.py`) | `scripts/contrast.py lesson-check` — exit 1 without a real (non-placeholder, non-template) value for every field; Step 7 requires it |
| 5 | Anti-contamination prompt rule | `BenchPrompts` rejects grader/evaluation/rubric vocabulary ('must describe the WORK') | Step 1 anti-contamination note: probe prompts describe the work only |
| 6 | Fallback on invalid stage | STAGE_ATTEMPTS=2, then inherit the current harness (`core/recursion.py`) | Step 5 fallback rule: one retry, then keep the baseline |

## What was NOT adopted (deliberate)

- **The OpenCode / Codex substrate layer** — targets OpenCode agent trees
  (c1..c8 surface); our surface is Hermes skills/memory/subagents/hooks.
- **The judge and report CLIs** — Vertex-centric grading; we judge by
  deliverable contrast instead.
- **The benchmark runners** (JobBench, LAB, WorkBuddy-Bench) — require
  Google Cloud / Vertex credentials, Docker, Pandoc/Poppler; our probe tasks
  are our own.
- **The per-domain experience library directory** — Mnemosyne and skill
  lesson sections already carry lessons; we add the evidence gate instead of
  a parallel store.
- **Parallel/sequential recursion scaling sweeps** — one standard cycle
  (baseline + INV/DOM candidates + composition probe + selection) is the
  practical budget in this environment; larger budgets wait for demand.

## Syncing with upstream

Concept port — no verbatim files. Re-review the core loop (Propose–Probe–Compose)
when these upstream issues close: #5 (`--bench local` task folders), #4
(`--max-usd` spend limit), plus the first tagged release and bug-fix batch
(#44 judge keeps old grade, #65 resume cache reuse).

## Trial evidence

- Install: `uv venv --python 3.12 .venv && uv pip install -e '.[render]'` —
  core package has zero hard dependencies.
- `harness2-demo` (credential-free, deterministic editors): 2 tasks, 0
  failed, 28 executions, 32 editor calls, both recursion modes ran.
- Source read: `demo.py`, `core/{spec,recursion,library,selection,tree}.py`,
  `substrate/opencode.py`. Leakage guards and blinded-seed selection verified
  in source, not just in the README.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall