Skip to content

ashaychangwani/imprint / .agents/skills/imprint-reteach-audit

imprint-reteach-audit

Verified

Re-teach Imprint sites from existing recordings and independently audit generated tools, including fresh-run repeatability checks and failed-run diagnosis.

Re-teach + audit imprint sites

Use this to rebuild a site's tools from its existing recording and confirm they work.

Fresh master-flow validation

Check the branch and timeline.md first. For continuing the September 2026 experiment, read the handoff. Its local paths and process state are historical facts to verify, not instructions to resume old processes or assumptions that the data exists on another machine.

Use a new isolated IMPRINT_HOME and an explicit recording. Do not use a cleanup script that erases earlier output during a repeatability experiment. After code or prompt changes, start fresh, never resume a failed teach. Do not feed shipped examples or prior generated artifacts to the teaching agent.

Commands

# Replace placeholders with verified paths; do not reuse a prior run home.
IMPRINT_HOME=<new-isolated-home> bun run src/cli.ts teach <site> \
  --agent codex --from-session <recording.json> --timeout 90m --no-interactive

# Audit (spawns a headless Codex that calls every compiled tool live and grades
# correct/broken; exit 0 = PASS, default threshold 95%):
IMPRINT_HOME=<same-isolated-home> bun run src/cli.ts audit <site> \
  --provider codex-cli --timeout 45m --json

Hard rules

  • Codex fresh teaches use a native family capped at ten concurrent children. Preserve native assignment, lifecycle and descendant usage history when comparing runs. Other providers retain four focused workers. Run validation teaches and audits sequentially.
  • Respect the user's deadline. In this experiment the target is about 30 minutes, a reasoned progress decision at 60 minutes, and a 90-minute hard deadline. These are not universal defaults or permission to extend a run silently.
  • Use the product's monitoring mechanism for long runs; stop task-owned monitors at handoff. A quiet agent is not by itself stuck. Check current facts before killing a run. No fixed repair-attempt limit is implied.
  • To re-teach several sites + audit each as one hands-off job, loop them sequentially in a single background driver script (teach → audit → next).

Monitoring a background teach (the log is spinner-heavy)

# strip ANSI + collapse \r redraws + drop spinner frames
sed -E 's/\x1b\[[0-9;?]*[a-zA-Z]//g' LOG | tr '\r' '\n' \
  | grep -aviE '◐|◓|◑|◒|◇|◆|●|Triaging|Replaying|Compiling •|cycle [0-9]/5|^\s*$|^│|^├'

Watch for: shared modules built + verified vs pruning; per-tool index.ts appearing under ~/.imprint/<site>/<tool>/; final Done! N tools ready. Avoid grepping cycle/Compiling • lines raw — they are giant \r-concatenated blobs.

Known failure modes

  • Teach passed but audit failed: baseline verification is narrower than parameter coverage. Read the generated parser, checked request, actual research observations, MVP review, and audit results. Equal unlabeled numbers can have different meanings; do not turn one site's inference into a runtime classifier.
  • Partial/proven handoff rejected: check basedOnObservationId against the exact tested candidate, including input values and backend selection. A malformed agent report is not proof that an API failed. Preserve observations and let the master direct repair; do not accept untested changes as proof.
  • Missing upstream values: inspect call_producer observations and the consumer's actual test. Recorded opaque values are diagnostic evidence, not replacements for fresh producer output. Do not mix values from unrelated rows.
  • Slow browser setup: compare cold startup with warm same-tool/same-rung invocations. Do not share browser state across tools/rungs to improve a number. Cookie names and HTTP 200 alone establish neither usability nor semantics.

The following older shipped-path notes are diagnostic background, not a reason to invoke removed legacy teaching entrypoints:

  • Valid shared module pruned for "must export a transform function" — was a bun stale-.ts-import-cache bug; fixed in src/imprint/prereq-builder.ts (importModuleFresh copies the module to a unique sibling path before re-importing, defeating the cache). If it recurs, the fix is there.
  • Audit grader non-determinism on a no-op param. A param that legitimately has no observable effect (e.g. a brand filter where two brands share the same data) can be graded correct one run and tool_broken the next, flipping a site between PASS and ~85%. The tools still return correct data — re-audit once to confirm it's variance before treating it as a defect (and report honest numbers, don't re-roll just to pass).
  • A network outage during the replay/capture stage corrupts the session — events time out with 0 requests captured (the captured-count plateaus). Replay normally takes ~3 min; if it's dragging for many minutes with a flat capture count, the network dropped. Re-teach when the connection is stable. Diagnose via the per-event capture trajectory in the teach log (Replaying event N/M (K requests captured)).

Verify + expose for live testing

  • Read deterministic totals and actual invocation/parameter arrays; agent prose may miscount. Report failures, excluded bad inputs, unsupported parameters, and the selected scope alongside a percentage.

  • A narrower public contract may be a valid MVP, but dropping an unproven parameter does not prove that feature was repaired. Keep all failed attempts in the comparison. Do not reroll audits merely to hide failures.

  • Record run duration, input/output tokens, cache-read/cache-write tokens, and pricing assumptions. Do not sum aggregate and child trace usage twice; use provider-reported usage semantics. Interrupted usage may be incomplete.

  • Check available disk before another large run. Never remove recordings, evidence, or caches without authorization. Keep private raw evidence local.

  • PASS = audit exit 0 and score ≥ 95% (graded N of N invocations).

  • The MCP registration imprint mcp-server <site> is a command pointer — it re-reads ~/.imprint/<site>/ on every spawn, so it always serves the latest teach. After a re-teach, a running Codex session must reconnect (/mcp) or start fresh to pick up new tools; a new session needs nothing.

  • Confirm what's served: Codex mcp list (look for imprint-<site> ✓ Connected) and Codex mcp get imprint-<site>.

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall