Skip to content

code-yeongyu/oh-my-openagent / packages/shared-skills/skills/ulw-research

ulw-research

Unverified

Runs maximum-saturation research with a cooperating team, claim-graph gating, and a cited, QA'd deliverable. Use when the user explicitly asks for research or a deep investigation, including any 'ulw' research wording.

FileKindSize
AGENTS.mdmarkdown9.2 KB
ATTRIBUTION.mdmarkdown2.4 KB
SKILL.mdskill md42.2 KB
references/deliverable-phase.mdmarkdown49.1 KB
references/fact-ledger-grammar.mdmarkdown6.8 KB
references/fact-ledger.mdmarkdown24.4 KB
references/fidelity-claim-kinds.mdmarkdown5.2 KB
references/markdown-blocks.mdmarkdown721 B
references/report-gates.mdmarkdown32.0 KB
references/report-visualization.mdmarkdown15.4 KB
references/research-collection.mdmarkdown26.8 KB
references/research-delivery.mdmarkdown14.3 KB
references/unit-rules.mdmarkdown16.3 KB
scripts/claim-kinds.mjsscript11.2 KB
scripts/claim-kinds.test.tsscript13.5 KB
scripts/claim-prefix.test.tsscript4.9 KB
scripts/claims-index.mjsscript5.3 KB
scripts/claims-index.test.tsscript5.7 KB
scripts/cli-support.mjsscript4.1 KB
scripts/cli-support.test.tsscript4.3 KB
scripts/contracts-validation.test.tsscript3.3 KB
scripts/contracts.mjsscript26.1 KB
scripts/contracts.test.tsscript7.9 KB
scripts/conversion-altered.test.tsscript11.0 KB
scripts/conversion-originals.test.tsscript22.7 KB
scripts/conversion-probe-classes.test.tsscript45.2 KB
scripts/conversion-probe.mjsscript13.5 KB
scripts/conversion-probe.test.tsscript11.6 KB
scripts/copy-returns-fidelity-policy.test.tsscript10.4 KB
scripts/copy-returns-polish.test.tsscript12.1 KB
scripts/copy-returns.test.tsscript10.6 KB
scripts/css-hiding.mjsscript9.1 KB
scripts/css-lite.mjsscript7.6 KB
scripts/css-lite.test.tsscript6.4 KB
scripts/delivery-barrier-meaning.test.tsscript12.3 KB
scripts/delivery-barrier.test.tsscript11.4 KB
scripts/design-spec.mjsscript7.2 KB
scripts/design-spec.test.tsscript6.7 KB
scripts/entities.mjsscript2.5 KB
scripts/entry-guard.mjsscript794 B
scripts/fact-commands-fill.mjsscript6.6 KB
scripts/fact-commands-meaning.mjsscript12.6 KB
scripts/fact-commands-polish.mjsscript10.3 KB
scripts/fact-commands-verify.mjsscript12.6 KB
scripts/fact-commands.mjsscript13.8 KB
scripts/fact-derive-2.test.tsscript7.2 KB
scripts/fact-derive.mjsscript4.9 KB
scripts/fact-derive.test.tsscript10.8 KB
scripts/fact-expr.mjsscript4.5 KB
scripts/fact-fill.mjsscript11.7 KB
scripts/fact-fill.test.tsscript10.7 KB
scripts/fact-ledger-grammar.test.tsscript11.8 KB
scripts/fact-ledger-kinds.mjsscript16.5 KB
scripts/fact-ledger-kinds.test.tsscript9.3 KB
scripts/fact-ledger-legacy.test.tsscript8.9 KB
scripts/fact-ledger-schema.mjsscript11.9 KB
scripts/fact-ledger-schema.test.tsscript1.5 KB
scripts/fact-ledger.mjsscript5.5 KB
scripts/fact-ledger.test.tsscript9.7 KB
scripts/fact-locate.mjsscript12.1 KB
scripts/fact-locate.test.tsscript7.2 KB
scripts/fact-pairs.mjsscript8.3 KB
scripts/fact-pairs.test.tsscript6.6 KB
scripts/fact-recompute.mjsscript4.8 KB
scripts/fact-unit-dimensions-2.test.tsscript8.4 KB
scripts/fact-unit-dimensions.test.tsscript11.9 KB
scripts/fact-unit-rules.test.tsscript8.9 KB
scripts/fidelity-binding.mjsscript6.7 KB
scripts/fidelity-binding.test.tsscript4.0 KB
scripts/fidelity-brief.mjsscript6.4 KB
scripts/fidelity-brief.test.tsscript7.8 KB
scripts/fidelity-claims.mjsscript5.1 KB
scripts/fidelity-csv.mjsscript1.1 KB
scripts/fidelity-lexicon.mjsscript8.1 KB
scripts/fidelity-lexicon.test.tsscript6.0 KB
scripts/fidelity-originals.mjsscript9.9 KB
scripts/fidelity-policy.mjsscript10.1 KB
scripts/fidelity-records.mjsscript13.4 KB
scripts/fidelity-snapshot-readability.mjsscript2.2 KB
scripts/fidelity-sources.mjsscript4.3 KB
scripts/fidelity-state.mjsscript2.1 KB
scripts/fidelity-state.test.tsscript1.7 KB
scripts/fidelity-trace.mjsscript6.1 KB
scripts/fidelity-trace.test.tsscript3.3 KB
scripts/fidelity-value-unit.mjsscript13.3 KB
scripts/fidelity-values.mjsscript16.1 KB
scripts/fidelity-values.test.tsscript15.8 KB
scripts/format-extract-css.mjsscript5.1 KB
scripts/format-extract.mjsscript10.3 KB
scripts/format-extract.test.tsscript11.2 KB
scripts/gates-context.test.tsscript2.6 KB
scripts/gates-fidelity.mjsscript6.3 KB
scripts/gates-fidelity.test.tsscript3.3 KB
scripts/gates-figures.mjsscript9.9 KB
scripts/gates-layout.mjsscript6.6 KB
scripts/gates-layout.test.tsscript12.4 KB
scripts/gates-static-headings.test.tsscript961 B
scripts/gates-static.mjsscript8.1 KB
scripts/gates-static.test.tsscript14.9 KB
scripts/gates-structure.mjsscript4.7 KB
scripts/gates-text.mjsscript7.2 KB
scripts/glossary-vocabulary.test.tsscript1.7 KB
scripts/html-foreign-object.mjsscript2.8 KB
scripts/html-lite.mjsscript9.7 KB
scripts/html-lite.test.tsscript7.9 KB
scripts/html-markup.mjsscript7.0 KB
scripts/html-text-map.mjsscript4.8 KB
scripts/html-text-map.test.tsscript15.2 KB
scripts/layout-probe.mjsscript4.6 KB
scripts/layout-probe.test.tsscript4.2 KB
scripts/lexicon.mjsscript2.6 KB
scripts/lexicon.test.tsscript3.2 KB
scripts/markdown-closers.mjsscript4.9 KB
scripts/md-lite-boundaries.test.tsscript2.2 KB
scripts/md-lite.mjsscript7.8 KB
scripts/md-lite.test.tsscript10.3 KB
scripts/meaning-check.mjsscript9.7 KB
scripts/meaning-check.test.tsscript12.0 KB
scripts/number-grammar.mjsscript2.1 KB
scripts/number-grammar.test.tsscript1.8 KB
scripts/numerals.mjsscript12.0 KB
scripts/numerals.test.tsscript6.1 KB
scripts/outcome-start.mjsscript3.8 KB
scripts/outcome-start.test.tsscript5.0 KB
scripts/outcome.mjsscript10.5 KB
scripts/outcome.test.tsscript12.8 KB
scripts/page-source-locator.test.tsscript11.7 KB
scripts/particles-ko.mjsscript4.8 KB
scripts/particles-ko.test.tsscript4.3 KB
scripts/placeholder.mjsscript9.3 KB
scripts/placeholder.test.tsscript3.8 KB
scripts/polish-apply.mjsscript6.8 KB
scripts/polish-apply.test.tsscript19.7 KB
scripts/polish-diff.mjsscript1.7 KB
scripts/polish-diff.test.tsscript10.1 KB
scripts/polish-markup.test.tsscript14.0 KB
scripts/polish-partial.test.tsscript2.5 KB
scripts/polish-project.mjsscript5.8 KB
scripts/polish-rules.mjsscript9.8 KB
scripts/rational.mjsscript5.2 KB
scripts/rational.test.tsscript1.9 KB
scripts/raw-decode.mjsscript4.5 KB
scripts/raw-locator.mjsscript2.4 KB
scripts/raw-snapshot.test.tsscript7.3 KB
scripts/references-fences.test.tsscript1.2 KB
scripts/repair-tracker.mjsscript7.5 KB
scripts/repair-tracker.test.tsscript10.8 KB
scripts/report-tools-2.test.tsscript16.1 KB
scripts/report-tools-3.test.tsscript9.3 KB
scripts/report-tools-4.test.tsscript10.1 KB
scripts/report-tools-5.test.tsscript14.6 KB
scripts/report-tools-commands.mjsscript12.9 KB
scripts/report-tools.mjsscript2.6 KB
scripts/report-tools.test.tsscript12.1 KB
scripts/sentence-split.mjsscript3.0 KB
scripts/sentence-split.test.tsscript4.5 KB
scripts/snapshot-readability.test.tsscript7.0 KB
scripts/source-commands.mjsscript12.4 KB
scripts/source-integrity.test.tsscript18.1 KB
scripts/source-refresh.mjsscript5.2 KB
scripts/source-snapshot.mjsscript9.9 KB
scripts/source-snapshot.test.tsscript6.3 KB
scripts/source-view.mjsscript5.0 KB
scripts/sources-citation.mjsscript12.1 KB
scripts/sources-citation.test.tsscript14.8 KB
scripts/tests/contracts-support.tsscript3.5 KB
scripts/tests/delivery-barrier-session.tsscript3.3 KB
scripts/tests/derive-clock.tsscript341 B
scripts/tests/fact-derive-support.tsscript1.9 KB
scripts/tests/fact-unit-dimensions-support.tsscript1.5 KB
scripts/tests/fact-unit-rules-support.tsscript16.8 KB
scripts/tests/fidelity-values-support.tsscript1.6 KB
scripts/tests/fixtures/assets/figure-1.svgother141 B
scripts/tests/fixtures/bad-report.htmlother2.1 KB
scripts/tests/fixtures/boxes.jsonconfig3.2 KB
scripts/tests/fixtures/conversion-probe.htmlother485 B
scripts/tests/fixtures/design-spec.mdmarkdown458 B
scripts/tests/fixtures/fidelity-brief-locked.mdmarkdown571 B
scripts/tests/fixtures/fidelity-brief-pending.mdmarkdown574 B
scripts/tests/fixtures/fidelity/assets/share.csvother40 B
scripts/tests/fixtures/fidelity/claim-graph.mdmarkdown449 B
scripts/tests/fixtures/fidelity/gen-legacy-dev.mjsscript5.0 KB
scripts/tests/fixtures/fidelity/ledger-basic.jsonconfig5.1 KB
scripts/tests/fixtures/fidelity/ledger-legacy-dev.jsonconfig18.3 KB
scripts/tests/fixtures/fidelity/ledger-verify.jsonconfig9.1 KB
scripts/tests/fixtures/fidelity/polish-edits.jsonconfig303 B
scripts/tests/fixtures/fidelity/report-bad.htmlother821 B
scripts/tests/fixtures/fidelity/report-good.htmlother2.4 KB
scripts/tests/fixtures/fidelity/sources/S4.htmlother680 B
scripts/tests/fixtures/fidelity/sources/S4/acronyms.jsonconfig36 B
scripts/tests/fixtures/fidelity/sources/S4/meta.jsonconfig256 B
scripts/tests/fixtures/fidelity/sources/S4/sentences.jsonlother623 B
scripts/tests/fixtures/fidelity/sources/S4/text.txtmarkdown300 B
scripts/tests/fixtures/fidelity/sources/S5/acronyms.jsonconfig3 B
scripts/tests/fixtures/fidelity/sources/S5/meta.jsonconfig235 B
scripts/tests/fixtures/fidelity/sources/S5/sentences.jsonlother53 B
scripts/tests/fixtures/fidelity/sources/S5/text.txtmarkdown10 B
scripts/tests/fixtures/fidelity/sources/S9.txtmarkdown128 B
scripts/tests/fixtures/good-report.htmlother2.7 KB
scripts/tests/fixtures/reference-report.htmlother2.5 KB

scripts/report-tools-2.test.ts

View on GitHub
import { describe, expect, onTestFinished, test } from "bun:test"
import { createHash } from "node:crypto"
import { copyFileSync, existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"
import { tmpdir } from "node:os"
import { delimiter, join } from "node:path"
import { PROMOTION } from "./fact-commands-meaning.mjs"
import { writeFidelityRecord } from "./fidelity-records.mjs"
import { PAIR_KINDS } from "./fact-pairs.mjs"
import { VERDICT_SCHEMA } from "./meaning-check.mjs"
import { POLISH_MARKUP_REASON } from "./polish-apply.mjs"
import { COMMANDS, run } from "./report-tools.mjs"
import {
  SCRIPT,
  FIXTURES,
  sessionDir,
  nodeWithoutBun,
  defectsFile,
  FIDELITY,
  sha256,
  CliRow,
  CliJson,
  nodeJson,
  fidelitySession,
  verify,
  codesOf,
  Pair,
  pairedSession,
  verdicts,
  calibrationRows,
  runLines,
  PolishJson,
  polishSession,
  polishApply,
  setField,
  fillTemplate,
  textSource,
  ledgerRow,
} from "./tests/report-tools-support"

describe("fact, source, and claims under node", () => {
  test("#given a fresh session #when snapshot, add, quote, derive, fill, and verify run #then the filled report verifies clean", () => {
    // given
    const { dir, s, snap, add, quote, derive, mm } = fidelitySession()
    // when
    const filled = nodeJson(["fact", "fill", join(dir, "report.src.html"), "--out", join(dir, "report.html"), ...s])
    const checked = verify(dir, "report.html")
    // then
    expect(snap.code).toBe(0)
    expect(add.code).toBe(0)
    expect(add.json.data.row.value).toBe("320")
    expect(add.json.data.context.match).toBe("320 m")
    expect(quote.code).toBe(0)
    expect(quote.json.data.row.fact_id).toBe("Q1")
    expect(derive.code).toBe(0)
    expect(mm).toBe("F2")
    expect(filled.code).toBe(0)
    expect(readFileSync(join(dir, "report.html"), "utf8")).toContain("320,000 mm (320 m)")
    expect(checked.code).toBe(0)
    expect(checked.json.defects).toEqual([])
    const record = JSON.parse(readFileSync(join(dir, "fidelity-defects.json"), "utf8"))
    expect(record.artifact_sha256).toBe(sha256(join(dir, "report.html")))
  })

  test("#given a filled report #when a value is hand-edited or a number typed #then verify exits 1 with the code", () => {
    const { dir, s } = fidelitySession()
    nodeJson(["fact", "fill", join(dir, "report.src.html"), "--out", join(dir, "report.html"), ...s])
    const html = readFileSync(join(dir, "report.html"), "utf8")
    writeFileSync(join(dir, "unit.html"), html.replace("320 m", "320 mm"))
    writeFileSync(join(dir, "typed.html"), html.replace("<h2>How", "<p>Revenue grew 12% [S4].</p>\n<h2>How"))
    const unit = verify(dir, "unit.html")
    const typed = verify(dir, "typed.html")
    expect(unit.code).toBe(1)
    expect(codesOf(unit.json)).toContain("unit_mismatch")
    expect(typed.code).toBe(1)
    expect(codesOf(typed.json)).toContain("untraceable_number")
  })

  test("#given an anchor that is not in the source #when fact add runs #then it exits 1 with fact_not_located and writes no row", () => {
    const { dir, s } = fidelitySession()
    const proc = nodeJson(["fact", "add", "--source", "S4", "--find", "nope", "--label", "x", ...s])
    expect(proc.code).toBe(1)
    expect(proc.stdout).toContain("fact_not_located")
    expect(
      JSON.parse(readFileSync(join(dir, "fact-ledger.json"), "utf8")).rows.map((r: { fact_id: string }) => r.fact_id),
    ).toEqual(["F1", "Q1", "F2"])
  })

  test("#given a snapshotted source #when snapshotted again with the same or different content #then same is a no-op and different exits 1", () => {
    const { dir, s } = fidelitySession()
    writeFileSync(join(dir, "other.txt"), "Span length: 999 m\n")
    expect(
      nodeJson(["source", "snapshot", "--id", "S4", "--html", join(FIDELITY, "sources", "S4.html"), ...s]).code,
    ).toBe(0)
    expect(nodeJson(["source", "snapshot", "--id", "S4", "--text", join(dir, "other.txt"), ...s]).code).toBe(1)
  })

  test("#given an assumed value with and without a rationale #when added #then the row is kind assumed, and no rationale exits 2", () => {
    const { s } = fidelitySession()
    const added = nodeJson([
      "fact",
      "add",
      "--assumed",
      "12",
      "--unit",
      "%",
      "--rationale",
      "planning share chosen by the requester",
      "--label",
      "share_assumed",
      "--json",
      ...s,
    ])
    expect(added.code).toBe(0)
    expect(added.json.data.row.kind).toBe("assumed")
    expect(nodeJson(["fact", "add", "--assumed", "12", "--label", "x", ...s]).code).toBe(2)
  })

  test("#given the fixture claim graph #when claims index runs #then claims.json lists its claims", () => {
    const dir = sessionDir()
    const proc = nodeJson(["claims", "index", join(FIDELITY, "claim-graph.md"), "--out", join(dir, "claims.json")])
    expect(proc.code).toBe(0)
    expect(JSON.parse(readFileSync(join(dir, "claims.json"), "utf8")).claims.length).toBeGreaterThan(0)
  })

  test("#given a ledger #when exported as csv #then rows carry fact_id and plot_value and stderr has the coverage line", () => {
    const { s } = fidelitySession()
    const proc = nodeJson(["fact", "export", "--csv", ...s])
    expect(proc.code).toBe(0)
    expect(proc.stdout.split("\n")[0]).toContain("fact_id")
    expect(proc.stdout).toContain("F2")
    expect(proc.stderr).toContain("1 of 1 sources snapshotted")
  })
})

describe("meaning verifier under node", () => {
  test("#given the filled fixture #when fact pairs runs #then pairs.json holds one quote pair per cited ledger quote", () => {
    const { dir, pairs } = pairedSession()
    const written = JSON.parse(readFileSync(join(dir, "pairs.json"), "utf8"))
    expect(pairs.code).toBe(0)
    expect(written.skipped).toBe(false)
    expect(written.pairs.map((p: Pair) => [p.pair_id, p.kind, p.fact_ids])).toEqual([
      ["P1", "quote", ["F1"]],
      ["P2", "quote", ["Q1"]],
    ])
    expect(written.pairs_sha256).toMatch(/^[0-9a-f]{64}$/)
    expect(Object.keys(written.counts).filter((key) => key !== "total" && key !== "force_strong")).toEqual([
      ...PAIR_KINDS,
    ])
    expect(written.verdict_schema).toEqual(JSON.parse(JSON.stringify(VERDICT_SCHEMA)))
  })

  test("#given cheap and strong verdicts agreeing on hedge_dropped #when meaning-check runs #then it exits 1 with one confirmed defect and one calibration line", () => {
    const { dir, s } = pairedSession()
    const drift = {
      pair_id: "P2",
      verdict: "drift",
      drift_kind: "hedge_dropped",
      fragment: "no earlier than 2027",
      source_fragment: "we estimate",
      explanation: "The estimate reads as a fact.",
    }
    const cheap = verdicts(dir, "cheap.json", "cheap", [drift, { pair_id: "P1", verdict: "ok" }])
    const strong = verdicts(dir, "strong.json", "strong", [drift])
    const checked = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      "--verdicts",
      cheap,
      "--verdicts",
      strong,
      ...s,
    ])
    const record = JSON.parse(readFileSync(join(dir, "fidelity-meaning-defects.json"), "utf8"))
    expect(checked.code).toBe(1)
    expect(codesOf(checked.json)).toEqual(["meaning_drift_confirmed"])
    expect(codesOf(record)).toEqual(["meaning_drift_confirmed"])
    expect(calibrationRows(dir)).toHaveLength(1)
    expect(calibrationRows(dir)[0]).toMatchObject({
      pair_id: "P2",
      drift_kind: "hedge_dropped",
      confirmed: true,
      disputed: false,
    })
    const header = JSON.parse(readFileSync(join(dir, "fidelity-calibration.jsonl"), "utf8").split("\n")[0])
    expect(header.promotion).toEqual({ ...PROMOTION })
    const again = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      "--verdicts",
      cheap,
      "--verdicts",
      strong,
      ...s,
    ])
    expect(again.code).toBe(1)
    expect(calibrationRows(dir)).toHaveLength(1)
  })

  test("#given a dispute on the confirmed pair #when meaning-check runs #then the defect stays and the calibration line is disputed", () => {
    const { dir, s } = pairedSession()
    const drift = {
      pair_id: "P2",
      verdict: "drift",
      drift_kind: "hedge_dropped",
      fragment: "no earlier than 2027",
      source_fragment: "we estimate",
    }
    const files = [
      "--verdicts",
      verdicts(dir, "cheap.json", "cheap", [drift, { pair_id: "P1", verdict: "ok" }]),
      "--verdicts",
      verdicts(dir, "strong.json", "strong", [drift]),
    ]
    const checked = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      ...files,
      "--dispute",
      "P2",
      "--reason",
      "the operator states it",
      ...s,
    ])
    expect(checked.code).toBe(1)
    expect(codesOf(checked.json)).toEqual(["meaning_drift_confirmed"])
    expect(calibrationRows(dir)[0]).toMatchObject({ pair_id: "P2", disputed: true, reason: "the operator states it" })
  })

  test("#given --edition md #when meaning-check runs #then it writes that edition's record, and an unknown edition refuses", () => {
    const { dir, s } = pairedSession()
    const clean = [
      { pair_id: "P1", verdict: "ok" },
      { pair_id: "P2", verdict: "ok" },
    ]
    const files = [
      "--verdicts",
      verdicts(dir, "cheap.json", "cheap", clean),
      "--verdicts",
      verdicts(dir, "strong.json", "strong", clean),
    ]
    const checked = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      ...files,
      "--edition",
      "md",
      ...s,
    ])
    const unknown = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      ...files,
      "--edition",
      "epub",
      ...s,
    ])
    expect(checked.code).toBe(0)
    expect(JSON.parse(readFileSync(join(dir, "fidelity-meaning-defects.md.json"), "utf8"))).toMatchObject({
      record_kind: "meaning",
      artifact_sha256: sha256(join(dir, "report.html")),
    })
    expect(existsSync(join(dir, "fidelity-meaning-defects.json"))).toBe(false)
    expect(unknown.code).toBe(2)
  })

  test("#given a dropped verdict and an unread pair #when meaning-check runs #then both pairs are meaning_unverified and it exits 1", () => {
    const { dir, s } = pairedSession()
    const cheap = verdicts(dir, "cheap.json", "cheap", [
      {
        pair_id: "P2",
        verdict: "drift",
        drift_kind: "hedge_dropped",
        fragment: "certainly by 2027",
        source_fragment: "we estimate",
      },
    ])
    const checked = nodeJson(["fact", "meaning-check", "--pairs", join(dir, "pairs.json"), "--verdicts", cheap, ...s])
    const summary = JSON.parse(checked.stderr.trim().split("\n").at(-1) ?? "{}")
    expect(checked.code).toBe(1)
    expect(checked.json.status).toBe("fail")
    expect(codesOf(checked.json)).toEqual(["meaning_unverified", "meaning_unverified"])
    expect(checked.json.unverified.map((gap: { pair_id: string }) => gap.pair_id)).toEqual(["P1", "P2"])
    expect(summary).toMatchObject({ command: "fact meaning-check", defects: 2, dropped: 1, unverified: 2 })
  })

  test("#given an unreadable verdict file or a malformed pairs file #when meaning-check runs #then it exits 2", () => {
    const { dir, s } = pairedSession()
    const broken = join(dir, "broken.json")
    writeFileSync(broken, "{ not json")
    expect(
      nodeJson(["fact", "meaning-check", "--pairs", join(dir, "pairs.json"), "--verdicts", broken, ...s]).code,
    ).toBe(2)
    const ok = verdicts(dir, "cheap.json", "cheap", [])
    for (const body of ["{}", "null", '{ "pairs": "P1" }']) {
      writeFileSync(join(dir, "bad-pairs.json"), body)
      expect(
        nodeJson(["fact", "meaning-check", "--pairs", join(dir, "bad-pairs.json"), "--verdicts", ok, ...s]).code,
        body,
      ).toBe(2)
    }
  })

  test("#given a confirmed run then a dispute on the same inputs #when meaning-check reruns #then the log still holds one run with the dispute", () => {
    const { dir, s } = pairedSession()
    const drift = {
      pair_id: "P2",
      verdict: "drift",
      drift_kind: "hedge_dropped",
      fragment: "no earlier than 2027",
      source_fragment: "we estimate",
    }
    const files = [
      "--verdicts",
      verdicts(dir, "cheap.json", "cheap", [drift, { pair_id: "P1", verdict: "ok" }]),
      "--verdicts",
      verdicts(dir, "strong.json", "strong", [drift]),
    ]
    const first = nodeJson(["fact", "meaning-check", "--pairs", join(dir, "pairs.json"), ...files, ...s])
    const disputed = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      ...files,
      "--dispute",
      "P2",
      "--reason",
      "the operator states it",
      ...s,
    ])
    expect(disputed.code).toBe(1)
    expect(disputed.json.run_id).toBe(first.json.run_id)
    expect(disputed.json.promotion_eligibility).toMatchObject({
      runs: 1,
      confirmed: 1,
      disputed_confirmed: 1,
      disputed_to_confirmed: 1,
    })
    expect(calibrationRows(dir)).toEqual([
      expect.objectContaining({ pair_id: "P2", disputed: true, reason: "the operator states it" }),
    ])
    const plain = nodeJson(["fact", "meaning-check", "--pairs", join(dir, "pairs.json"), ...files, ...s])
    expect(plain.json.promotion_eligibility).toMatchObject({ runs: 1, disputed_confirmed: 1 })
  })

  test("#given a run with no flagged pair #when meaning-check runs #then the run is still logged and counted", () => {
    const { dir, s } = pairedSession()
    const ok = (pair_id: string) => ({ pair_id, verdict: "ok" })
    const files = [
      "--verdicts",
      verdicts(dir, "cheap.json", "cheap", [ok("P1"), ok("P2")]),
      "--verdicts",
      verdicts(dir, "strong.json", "strong", [ok("P1"), ok("P2")]),
    ]
    const checked = nodeJson(["fact", "meaning-check", "--pairs", join(dir, "pairs.json"), ...files, ...s])
    expect(checked.code).toBe(0)
    expect(calibrationRows(dir)).toEqual([])
    expect(runLines(dir)).toEqual([expect.objectContaining({ run_id: checked.json.run_id, flagged: 0 })])
    expect(checked.json.promotion_eligibility.runs).toBe(1)
  })

  test("#given a corrupt calibration log #when meaning-check runs #then it exits 2 naming the file and line", () => {
    const { dir, s } = pairedSession()
    writeFileSync(join(dir, "fidelity-calibration.jsonl"), "{ not json\n")
    const checked = nodeJson([
      "fact",
      "meaning-check",
      "--pairs",
      join(dir, "pairs.json"),
      "--verdicts",
      verdicts(dir, "cheap.json", "cheap", []),
      ...s,
    ])
    expect(checked.code).toBe(2)
    expect(checked.stderr).toContain(`${join(dir, "fidelity-calibration.jsonl")}:1`)
  })

  test("#given a skipped pairs file #when meaning-check runs #then it is accepted only when the locked brief says off", () => {
    const { dir, s } = fidelitySession()
    const skip = join(dir, "skip.json")
    writeFileSync(skip, JSON.stringify({ schemaVersion: 1, skipped: true, pairs: [] }))
    const check = () =>
      nodeJson(["fact", "meaning-check", "--pairs", skip, "--verdicts", verdicts(dir, "cheap.json", "cheap", []), ...s])
    const on = check()
    expect(on.code).toBe(1)
    expect(on.json.error).toBe("meaning_check_required")
    writeFileSync(join(dir, "brief.md"), "# Brief\n")
    expect(check().json.error).toBe("fidelity_interview_incomplete")
    expect(check().code).toBe(1)
    writeFileSync(
      join(dir, "brief.md"),
      readFileSync(join(FIXTURES, "fidelity-brief-locked.md"), "utf8").replace(
        "meaning check: on",
        "meaning check: off",
      ),
    )
    const off = check()
    expect(off.code).toBe(0)
    expect(off.json.status).toBe("skipped")
  })

  test("#given a brief with meaning check off #when fact pairs runs #then it exits 0 and writes skipped", () => {
    const { dir, s } = fidelitySession()
    const brief = readFileSync(join(dir, "brief.md"), "utf8").replace("meaning check: on", "meaning check: off")
    writeFileSync(join(dir, "brief.md"), brief)
    nodeJson(["fact", "fill", join(dir, "report.src.html"), "--out", join(dir, "report.html"), ...s])
    const pairs = nodeJson(["fact", "pairs", join(dir, "report.html"), "--out", join(dir, "pairs.json"), ...s])
    expect(pairs.code).toBe(0)
    expect(JSON.parse(readFileSync(join(dir, "pairs.json"), "utf8"))).toMatchObject({ skipped: true })
  })
})
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall