Skip to content

chrisgagne/grounded-forge / .claude/skills/ingesting-resources

ingesting-resources

Verified

Runs the 9-pass ingestion protocol against a single source dropped in {corpus-root}/sources/ingest/, producing a .source.md sidecar, the converted markdown, a deep reference, a light reference, one distillation per applicable task axis, and index and image updates. On success, the input source is removed from sources/ingest/.

FileKindSize
9-pass-protocol.mdmarkdown46.1 KB
SKILL.mdskill md27.1 KB

9-pass-protocol.md

View on GitHub
# 9-Pass Ingestion Protocol: Detail

Per-pass procedures, templates, contracts, and failure modes for [`ingesting-resources`](SKILL.md).

Runbook (when to invoke, setup gates, post-pass operations) is in [`SKILL.md`](SKILL.md).

**Audit records describe the source and the passes.** Every file a pass writes to `{corpus-root}/references/_audit/` records what was read, checked, changed and left open, in terms of the source. Write the operator's reason for ingesting it (the engagement, client, person or question that prompted it) under `corpus.local/`, or leave it out. In a `corpus.commons/` corpus the audit records ship publicly; the pre-push hook's private-term check is the backstop, not the rule.

---

## Pass A: Context

**Produces:** the source / structure / citation-style block that opens the deep reference, plus the deep ref's HTML-comment frontmatter.

**Procedure:**

1. Capture the metadata gathered at input: author, title, edition, year, ISBN/DOI.
2. Capture the source's licence as the exact string (e.g. `CC BY-NC-SA 4.0`, `CC BY 4.0`, `All rights reserved`). If unverified, record `unverified`.
3. Capture the distribution scope, one of `open`, `open-nc`, `copyrighted`, `confidential`, `personal`. Scope is the operator's distribution intent, not the upstream licence; usually correlates with licence but can diverge (a CC BY paper used in a confidential engagement is Licence `CC BY 4.0`, Scope `confidential`). The build excludes references whose scope exceeds the profile's `max_scope`; `personal` ships in no profile.
4. Map the source's high-level structure: preface, parts, chapter list, appendices. **Enumerate the full chapter list explicitly in the deep ref's source/structure block.** This list becomes the anchor set Pass I checks the deep ref's coverage against; skipping a chapter from this list without an operator-approval marker is a Pass I fail condition.
5. Note any front-matter framing (preface arguments, dedications that signal an audience, footnotes that signal lineage).

**Frontmatter contract (deep reference):**

One line, HTML-comment-wrapped, with five fields: model identity, effort level, source-grounded marker, citation-style decision, ingestion date. Schema:

```html
<!-- Generated by: {model-id} | Effort: xhigh (9-pass) | Source-grounded | Citation style: {style} | Date: YYYY-MM-DD -->
```

Example:

```html
<!-- Generated by: claude-opus-4-7[1m] | Effort: xhigh (9-pass) | Source-grounded | Citation style: (Ch N, "Section name") | Date: 2026-05-02 -->
```

Record the actual model that ran the protocol. Downstream operators audit provenance against this field.

**Source/structure block (opens the deep ref):**

```markdown
**Source:** {full citation including ISBN/DOI and licence string}
**Scope:** {open|open-nc|copyrighted|confidential|personal}
**Structure:** {chapters, with one-line per chapter or part}
**Citation style:** `(Ch N, "Section name")` {or} `(Ch N, p. M)` {with rationale}
```

The `**Scope:**` line is mechanical: the build reads it directly to decide whether the reference ships under the profile's `max_scope` ceiling. Misspelled or omitted, the reference is excluded from every profile.

**Failure modes:** missing metadata, missing pages, ambiguous edition / year, missing or invalid scope. Stop and resolve before Pass B.

## Pass B: Structural read

**Produces:** an internal map of the source: which chapters carry which arguments, which named cases recur, which statistics anchor which claims. Not yet written to the deep ref.

**Procedure:**

1. Read once at speed (not the deep read; that's Pass C).
2. Note named cases (the author's worked examples), recurring motifs, and any statistics the author flags as central.
3. Identify the author's signature contrarian positions (where the author explicitly frames against a position).
4. Identify which other authors the source cites and how (approving, critical, neutral attribution).

**Failure modes:** mistaking the author's signposts (sometimes chapters titled "Conclusion" carry the central argument; sometimes a preface buries the thesis). Cross-check the structural read against the source's own pointers.

## Pass C: Deep read with citations

**Produces:** the candidate set of claims for the deep reference, each with an inline citation pointing at a specific passage in the source.

**Procedure:**

1. Read the source in full. Not chunked. Not summarised in passing. **Every chapter enumerated at Pass A must be read.** If you decide partway through that some chapter is "less essential" or "covered through cross-references in other chapters," that is a source-integrity violation — stop, escalate to the operator, and let them decide whether to (a) continue the read, (b) explicitly approve a `coverage: partial` with rationale, or (c) abandon. Never set `coverage: partial` on your own initiative.
2. If context budget runs out before all chapters are read, stop and escalate. Do not write the deep ref against incomplete coverage. Resume in a new session, or have the operator approve deferral.
3. For every non-trivial claim, attach a citation in the chosen style (`(Ch N, "Section name")` or `(Ch N, p. M)`).
4. Direct quotation appears in standard quotation marks at this stage, not blockquoted (Pass D handles blockquotes).
5. Preserve the author's framing language. If the author uses a specific term, the deep reference uses the same term, not a paraphrase.

**Failure modes:**

- *Reader-side coverage skip.* Deciding partway through Pass C that some chapter is less essential and labeling the deep ref `coverage: partial` after the fact. The canonical source-integrity violation: if you can read the chapter, you must read it. Pass I's TOC-vs-anchors check is the audit backstop.
- *Training-data leakage.* Stating something about the author or topic that the source itself does not state. If you cannot quote the source line(s) supporting a claim, the claim does not belong here.
- *Post-source vocabulary.* Importing a concept from the author's later work that the source under ingest did not use.
- *Cross-corpus drift.* Connecting the source to another author the source does not cite.

## Pass D: Blockquote extraction

**Produces:** the blockquoted passages in the deep reference. A small number of carefully chosen verbatim passages, each verified exact against the source.

**Procedure:**

1. From the candidate quotes captured in Pass C, select passages worth quoting verbatim (the author's own framing of the central argument; a paragraph that compresses a thesis).
2. Format as `> "..."` blockquotes with the chosen-style citation on the line below or appended.
3. **Verify exactness.** Re-check each blockquote character-by-character against the source. Verbatim accuracy is mandatory. `python3 scripts/check-verbatim.py {deep-ref}` checks every blockquote and `[V]` quotation in one run.
4. Keep blockquotes short. Long blockquotes are a sign that paraphrase would carry the meaning more cleanly.

**Failure modes:** introducing apparent corrections (capitalisation tidied up, a word, dash or punctuation mark changed). Verbatim means verbatim. If the source has an unusual choice, preserve it. Apostrophe and quote-mark style, straight or curly, is typography rather than the author's words, and is not a slip.

## Pass E: Synthesis (assemble the deep reference)

**Produces:** the finished deep reference file.

**Procedure:** assemble the artefacts of Passes A-D into the canonical deep-reference structure.

**An empty section is a normal outcome, and `None.` is the correct way to write one.** Two headings in the template invite invention, because they are shaped as slots and a model handed a slot fills it: *Connections the author makes in the text* and *Positions the author explicitly frames against*. Most research papers — a materials characterisation, an assay, a field trial — cite work without arguing against anyone, and report results without a contrarian stance. Writing `None.` there is a finding. Manufacturing an opponent is a Pass I defect, and an audited corpus in 2026 found invented oppositions its single most recurrent form of source-external framing: a bounded positive claim ("this method suits fragile grains") restated as a position the author opposed ("against harsh processing"). Reach for these sections only when the source itself does the framing — an explicit disagreement, a named alternative it rejects, a literature it positions against. Your knowing the debate the paper sits inside is not evidence that the paper joined it.

**Deep reference template:**

```markdown
<!-- {Pass A frontmatter line} -->

# {Author's surname}, {short topic}

**Source:** {Pass A source line, including licence string}
**Scope:** {Pass A scope decision: open|open-nc|copyrighted|confidential|personal}
**Structure:** {Pass A structure line}
**Citation style:** {Pass A citation-style line}

## Author's thesis (paraphrased from source)

{Two to four paragraphs paraphrasing the source's central
argument, with citations. The thesis section sets the frame
the rest of the deep reference operates within.}

## Part I: {Major theme from Pass B}

### {Concept/principle 1}

{Body paragraphs with citations and inline evidence-class markers.
Use the author's own naming if they number/name things.}

### {Concept/principle 2}

...

## Part II: {Next major theme}

...

## Key statistics (with provenance)

| Metric | Value | Classification |
|---|---|---|
| ... | ... | [V] (Ch N, "Section") or [AP] author's calc |

## Connections the author makes in the text

- {Source A}, {how the author cites them: approving, critical, neutral} [BT] (Ch N).
- ...

{or, where the source cites no one in a way worth recording:}

None.

## Positions the author explicitly frames against

- *{Position name}.* {How the author frames against it.}
- ...

{or, where the source argues against nothing:}

None.

## Citation and source-integrity notes

{Note the conversion path (PDF / EPUB / pandoc settings), the
citation-style decision and why, and any partial-coverage notes
if applicable.}
```

**Evidence-classification taxonomy** (inline markers in body paragraphs):

| Marker | Meaning | Example use |
|---|---|---|
| `[V]` | Verbatim. The author's own words appear in the source, reproduced character-exact. | Quoted sentences. |
| `[AP]` | Author paraphrase. Restates the author's point in different words; the author makes that point. | Most prose paragraphs. |
| `[AR]` | Author argument. Reports the author's argument structure (claim, warrant, conclusion). | Thesis-section paragraphs. |
| `[AE]` | Author example. The author's own case, summarised. | Worked-example references. |
| `[BT]` | Borrowed-through. The author cites someone else for the claim. | Cross-author connections. |

**Two ways `[V]` goes wrong that are easy to miss.**

**A translation can never carry `[V]`.** An English rendering of a non-English source is the ingester's wording, not the author's, however faithful. Mark the gloss `[AP]` and label it as a gloss; reserve `[V]` for the source's own characters, quoted in the source's own language. Auditing a Chinese-language pharmacopoeia reference in 2026 took its `[V]` count from 51 to zero on this rule alone — every marker sat on OCR'd-then-translated text.

**On an OCR'd source, check the tape's reading order before trusting a character-match.** Two-column conversions interleave columns and scramble line order, so a passage can be present word-for-word and still not exist as a contiguous sequence anywhere in the tape. Matching the words is not verifying the quotation. Where reading order is unreliable, either verify against the page image or reclassify to `[AP]` — and say which you did.

**Check a scanned source's quotations with a second OCR before looking at any page.** Render the pages that carry `[V]` quotations and OCR them with tesseract, then run the verbatim check against that text:

```
pdftoppm -f {p} -l {p} -r 300 -gray -png {source}.pdf {dir}/p{p}   # per quoted page
tesseract {dir}/p{p}-*.png - -l eng >> {dir}/fresh-ocr.txt
python3 scripts/check-verbatim.py {deep-ref} --source {dir}/fresh-ocr.txt
```

A quotation found in both the tape and the fresh OCR has two independent readings that agree, and is verified. Only the ones that disagree need a look at the page image. Do that look in a small separate agent that holds the deep reference and those page renders, never in the producing agent. Page images read in a long context are expensive and fragile: in a 2026 run a producer at 600k context re-read the same 40 renders four times over and rewrote its prompt cache each time, about 7M tokens of cache writes for one source. On that source, tesseract confirmed 27 of 33 quotations in about two minutes with no model tokens. Two of the six it missed were tesseract's own character slips (`tts` for `its`), and those are what the visual check is for.

**Hard rule for the deep reference:**

- No information not in the source. No biographical detail the source does not state. No historical context the author does not provide.
- No task-application guidance. Diagnostic questions, anti-patterns, worked-example projections live in distillations (Pass G).
- No post-source vocabulary. Concepts from the author's later work do not appear in the deep reference of an earlier work.

## Pass F: Light-reference derivation

**Produces:** the light reference file, derived from the verified deep. Three sub-passes; do not re-read the source.

**Sub-passes:**

1. *Condense.* Reduce the deep ref's parts to flat depth-2 headings. Drop sub-section structure.
2. *Tier-separate.* Strip any task-application guidance that bled in. The light ref is faithful summarisation, not distillation.
3. *Trace verify.* Confirm every claim in the light ref traces to a paragraph in the deep ref. Record the trace count in the closing HTML comment.

**Light reference template:**

```markdown
<!-- Generated by: claude-opus-4-7[1m] | Effort: high | Derived from verified deep ref | Date: 2026-MM-DD -->

# {Author's surname}, {short topic}

**Source:** {short citation}
**Structure:** {one-line shape}

## Author's Thesis (condensed)

{One to two paragraphs.}

## {Domain 1}

{Two paragraphs.}

## {Domain 2}

...

## Key Statistics

{Compact table or list.}

## Key Connections

{Bullet list.}

## Signature Contrarian Positions

{Numbered list.}

<!-- Verified: N claims traced to deep ref. No task-application guidance (tier-separated). -->
```

## Pass G: Distillation projection

**Produces:** one distillation per applicable task axis. Owned by [`creating-distillations`](../creating-distillations/SKILL.md); the full procedure (applicability gate, sub-passes, template, lens-as-optional-framing, failure modes) is in its companion [`projection-protocol.md`](../creating-distillations/projection-protocol.md).

**Summary:** for each task axis in the corpus, run the applicability gate (clear yes / clear no / ambiguous-ask-operator). For each applicable axis, project the verified deep onto the task's working vocabulary, phase-organise the diagnostic questions, and cross-check that every claim traces back to the deep.

**Trigger-grain projection (when the task spec carries field 2a).** `creating-distillations` does two extra things: (i) authors a `## Runtime triggers this source addresses` section in the distillation distillation, mapping the source's content to specific triggers from the seed table with teach-in-the-moment scripts; (ii) adds rows to the distillation index's per-phase runtime listener tables (Pass H). Task axes without field 2a produce the standard distillation shape; trigger-grain is a no-op.

**Trigger extensions.** When a source surfaces triggers the seed table didn't anticipate, the distillation distillation logs them in a `## Trigger extensions surfaced by this source` section. The operator folds extensions back into the task spec manually; the ingest surfaces them in the completion summary.

For licence considerations when distributing derived artefacts, see [`docs/architecture/copyright.md`](../../../docs/architecture/copyright.md).

## Pass H: Cross-reference

**Produces:** updated JSON runtime indexes that route to the new files, plus the operator-inspection `.md` views regenerated alongside.

Pass H used to ask the agent to author concept-A-Z entries by inference and hand-edit per-task distillation indexes. The mechanical-index migration moved that work into a deterministic preprocessor plus constrained LLM passes. The agent now drives the pipeline rather than performing the work directly. The split is the spec's anti-pattern guard: mechanical work (structural extraction, slug-table updates, .md table parsing) goes to Python; semantic work (alias merging, novel-concept naming, applicability) goes to named agents operating against slim staged payloads. Those agents pin Opus: the vocabulary they build is read on every query, so it runs on a model proven on judgment work, not the cheapest one. Their Codex twins in `.codex/agents/` run on the session's model instead; see `AGENTS.md`.

**Procedure (single-source run):**

1. **Allocate the slug-ID.** Run `python3 scripts/slug_table.py generate --corpus {corpus-root}` once the deep reference is on disk. It takes the corpus *path* (`corpus.local/aarbuddy`), not the bare name the build-index scripts take. It scans `references/*-deep.md` and appends an ID to `{corpus-root}/references/slug-table.json` for every slug not yet in it; existing IDs never change, so a rerun is safe.

2. **Run the deterministic preprocessor** against the converted source:
   ```
   python -m scripts.mechanical_index.preprocess \
       --source corpus.commons/{corpus}/sources/converted/{slug}.md \
       --discovery _planning/discovery/{slug}.json \
       --output _planning/extracted/{corpus}/{slug}.json
   ```
   Reads the converted markdown and the discovery JSON (run the discovery scan first if missing), emits a tier-aware extraction artefact: book-index entries, enumerated methods, heading tree, page-marker map, derived-tier log. No LLM calls; no source rereads.

   **Discovery scan, when `_planning/discovery/{slug}.json` is missing.** Run the mechanical baseline from the repo root, `python -m scripts.mechanical_index.discovery_scan_mechanical --corpus {corpus-root}`. It takes the corpus path and writes a baseline for every source that lacks one. Then dispatch the [`ingest-discovery-scanner`](../../agents/ingest-discovery-scanner.md) agent with the corpus root and slug. The agent reads the converted source in full and adds the author's enumerated named methods, which the regex baseline leaves empty. Those methods pass the concept-candidate filter from a single source, where a book-index entry needs two: the scan is how a source's own named methods enter the concept index.

3. **Extract reference-index fields via the refs pass.** Dispatch the [`ingest-refs-extractor`](../../agents/ingest-refs-extractor.md) agent with the corpus name, corpus root and slug. It reads the new deep reference's header and thesis and writes `{author, year, title, primary_topic, concept_tags}` to `_planning/staging/{corpus}/refs/{slug}.json`. Frontmatter prose varies (`Beck et al. (2001)`, `OpenStax (2019)`, organisational authors, working papers without years); regex would mis-attribute silently, so this is delegated to a constrained LLM step. Dispatch the named agent rather than a general-purpose one: it carries the extraction rules.

4. **Rebuild `reference-index.json`.** Run `python -m scripts.build_indexes.build_reference_index --corpus {corpus}`. Python merges the refs-pass output with mechanical fields (slug-table ID, `**Scope:**` from the deep, light/deep line counts) and writes `corpus.commons/{corpus}/reference-index.json`.

5. **Emit concept candidates.** Run `python -m scripts.build_indexes.build_concept_index --corpus {corpus} --emit-candidates`. Aggregates extracted artefacts across every source in the corpus, filters to cross-source-or-enumerated-method entries (single-source back-matter index entries are noise at the corpus-level concept axis), seeds curated tags from `reference-index.json`, writes the slim candidates payload to `_planning/staging/{corpus}/concepts/candidates.json`.

6. **Run the cross-link pass.** Dispatch the [`ingest-concept-linker`](../../agents/ingest-concept-linker.md) agent with the corpus name. It emits `(canonical, name, aliases, sources[])` records in one of two ways:
   - **Corpus already has `decisions.json`:** also pass the new sources' slug + ID pairs and a short label. The agent judges only those sources' candidates and writes `_planning/staging/{corpus}/concepts/pending-{label}-{date}.json`, carrying `new_records` and `add_sources` (new sources for existing canonicals). Back up `decisions.json`, then merge: append the new records, add each `add_sources` entry to its canonical's `sources` unless that source is already there (match on ID *or* slug: older records carry slug only), sort by canonical, write with `indent=2` and `ensure_ascii=False`.
   - **New corpus:** the agent judges the whole payload and writes `_planning/staging/{corpus}/concepts/decisions.json` directly.

   The spec calls this step out explicitly (§"Anti-patterns to avoid"): concept aliasing, novel-vs-existing decisions, and noise filtering need LLM judgment, not regex. A prior orchestrator attempt that substituted regex here had to be unwound.

7. **Assemble `concept-index.json`.** Run `python -m scripts.build_indexes.build_concept_index --corpus {corpus} --assemble`. Reconciles slug/ID drift mechanically (ID is authoritative; LLM slug typos correct silently with a drift log at `_planning/staging/{corpus}/concepts/drift.json`), attaches section + md_line pointers per source via the extracted artefacts' heading trees and enumerated-method body lines, writes `corpus.commons/{corpus}/concept-index.json`. For a source whose extracted artefact isn't on this machine, it keeps the pointer from the previous deep index.

8. **File concepts under topics.** Run `python -m scripts.build_indexes.build_concept_index --corpus {corpus} --emit-topic-payload`. It writes the assembled vocabulary to `_planning/staging/{corpus}/concepts/topic-payload.jsonl`, one concept per line, marking which concepts are already filed. Back up `topics.json` if it exists, then dispatch the [`ingest-topic-linker`](../../agents/ingest-topic-linker.md) agent with the corpus name and, for a per-source run, the new sources. Without a `topics.json` it files every concept; with one, it files only the unfiled concepts and adds to the topics without renumbering them. Confirm `python -m scripts.build_indexes.check_topics --corpus {corpus}` prints PASS, then run `--assemble` again. With `topics.json` present it writes `concept-index.json` as schema 3: a topics block a model reads whole, then concept rows found by topic ID. It also assigns IDs to new topics.

   Filing needs a judgment on every concept. Keyword rules here file concepts under the wrong topics, and a question routed through a wrong topic reaches the wrong material without any sign that it did.

   **Staged, for a large vocabulary.** One agent can file about 1,000 concepts; past that, run the staged pass. Add `--chunk-size 600` to `--emit-topic-payload`: it also writes `topic-names.jsonl` (every concept's name and source count) and `topic-payload.chunk-NN.jsonl` (consecutive runs of the payload). Dispatch the topic linker three times over: once to propose the topic list (`topics.skeleton.json`), then once per chunk, in parallel, to file that chunk against the list (`topics.chunk-NN.json`), then once to consolidate. The consolidating agent merges the chunks with `scripts.build_indexes.merge_topic_chunks`, settles the topics the filing agents proposed, splits oversized topics, and leaves `topics.json` passing `check_topics`. Then run `--assemble` as above.

   **New sources in a large vocabulary.** Once a corpus is filed this way, a per-source run files only the new concepts. Add `--unfiled-chunk` to `--emit-topic-payload`: it writes the unfiled concepts as `topic-payload.chunk-new.jsonl`, and the existing topic list, IDs kept, as `topics.skeleton.json`. Dispatch the topic linker once, for a per-source run in a large corpus. It files the chunk, merges it into `topics.json` with `merge_topic_chunks merge --into-existing` (every existing placement and ID stays), settles its own proposed topics, and leaves `check_topics` passing. Then run `--assemble`.

   **Settling duplicate flags.** The topic linker flags concepts that look like one idea under several keys (`merge_candidates` in `topics.json`); the flags wait until someone settles them. Dispatch the [`ingest-concept-linker`](../../agents/ingest-concept-linker.md) agent for a merge review: it decides each flag (merge, keep apart, or split a concept whose sources mean different things by it) and writes `concept-merges.json`. `python -m scripts.build_indexes.merge_concepts apply --corpus {corpus}` changes `decisions.json` and `topics.json` together and keeps both previous states beside them. Then run `--assemble`, `--emit-topic-payload` and `check_topics`.

   **Re-filing sources from their deep refs.** The cross-link pass credits a source only where its extracted candidates matched a row, and a source that treats an idea in one section, in its own words, ends up credited on rows about something else; a question about that idea never reaches it. Run this after a large ingest, or when routing misses sources that cover a question. `python -m scripts.build_indexes.refile_sources batches --corpus {corpus}` groups every source with a deep ref into batches of about 340 KB of deep refs, with each source's converted text path (`--texts` maps slugs whose text isn't at `sources/converted/{slug}.md`). Dispatch the [`ingest-refiler`](../../agents/ingest-refiler.md) agent once per batch, in parallel: it reads each source's deep ref in full against the topics block and proposes credits on existing rows, topic filings, new rows and removals of credits the source doesn't support, writing `refile/refile-{batch}.json`. `refile_sources apply --corpus {corpus} --label {label}` changes `decisions.json` and `topics.json` together, keeps both previous states, and logs every change's outcome. Then run `--assemble`, `--emit-topic-payload` and `check_topics`. For a corpus too large for one round of agents, apply and re-assemble between rounds so later batches see the rows earlier ones added. `apply` flags new rows whose names contain one another as merge candidates, for the merge review above.

9. **Update the per-task `task-index.json` files.** Pass G has already authored the operator-inspection `.md` views (`corpus.commons/{corpus}/distillations/{task}/{TASK}-DISTILLATION-INDEX.md`) with the new source's rows. Run `python -m scripts.build_indexes.build_task_index --corpus {corpus}` to regenerate every `corpus.commons/{corpus}/distillations/{task}/task-index.json` from the updated `.md` files.

10. **Merge into `IMAGE-INDEX.yaml`.** Run `python3 scripts/build_indexes/build_image_index.py --corpus {corpus}`. Image classification runs per source and writes to `_planning/staging/{corpus}/images/_ingest_image_index_{slug}.yaml`; this step merges that corpus's staging files into the corpus index at `{corpus-root}/sources/converted/IMAGE-INDEX.yaml`. **Skipping it makes every classified image invisible**, because the skill's rule is that an image absent from `IMAGE-INDEX.yaml` is not part of the library — and the staging files give no sign that anything is wrong. Each source with a staging file has its entries replaced by the staging file's; every other entry is carried forward as it stands, with its count printed per source. The index, not the staging, is the durable record: staging is gitignored and missing on a fresh clone, and some corpora hold entries that never had a staging file. So a source's entries leave only when you say so: `--drop {slug}` (repeatable). Read the output before committing: a `WARN` that entries will go means a source's staging holds fewer than the index had. `--check` reports without writing. Staging is filed by corpus because one slug can be ingested into several corpora, each describing the figures in its own words. It sits in gitignored `_planning/` with the other per-corpus staging, outside the corpus and outside the published `docs/` folder, so a private corpus's figure descriptions never reach a public remote.

11. **Cross-check sibling distillations.** If the new source belongs in another existing distillation's "Integration with Other References" section, update the existing file. This is the one remaining hand-edit in Pass H: it operates on per-distillation prose, not on the JSON runtime indexes.

   **Known gap, stated so it is not mistaken for verified.** Rows written here assert what *another* source says, and no pass checks them: Pass I verifies a reference against its own source, and nothing verifies a distillation against a different reference's source. The same applies to corpus-scope superlatives ("the only source in the corpus that reports X"). Keep such rows narrow and attributable, and treat them as the least-verified content in the corpus.

**Index hygiene.** The operator-inspection `.md` views describe the corpus *as it is right now*. They are read by humans browsing the corpus; the runtime reads only the JSON. Do not add `Last updated` dates, recent-changes summaries in the header, parenthetical "now includes…" lists, or changelog sections: git history covers the journey to here. The JSON indexes are derived artefacts; never hand-edit them. All updates flow through frontmatter + slug-table + the build scripts.

**Parallel-batch operation.** When ingesting many sources in one work-session, steps 2–4 (preprocess + refs pass) run in parallel per source (each writes to its own staging path). Steps 5–8 (concept index and topics) run once after every parallel source agent completes: the cross-link and topic passes need the whole corpus's vocabulary. Steps 9 and 10 also run once.

**The convergence steps are the ones that get skipped, and skipping them is invisible.** Per-source agents write their staging outputs faithfully and report success; the corpus-level assembly at the end of the batch is a single run that nobody's task list owns. A 55-source corpus audited in 2026 had missed four of them at once — no `IMAGE-INDEX.yaml` despite 46 staging files, 13 router rows still sitting in `_ingest_*.md` stubs so five distillations were unreachable, `concept-index --assemble` never run, and 20 source sidecars unwritten. Nothing looked broken from any per-source view. Before declaring a batch complete, run this checklist and check the counts, not the exit codes:

```
python -m scripts.build_indexes.build_reference_index --corpus {corpus}
python -m scripts.build_indexes.build_concept_index   --corpus {corpus} --emit-candidates
#   … cross-link pass → decisions.json …
python -m scripts.build_indexes.build_concept_index   --corpus {corpus} --assemble
python -m scripts.build_indexes.build_concept_index   --corpus {corpus} --emit-topic-payload
#   … topic pass → topics.json …
python -m scripts.build_indexes.check_topics          --corpus {corpus}
python -m scripts.build_indexes.build_concept_index   --corpus {corpus} --assemble
python -m scripts.build_indexes.build_task_index      --corpus {corpus}
python3 scripts/build_indexes/build_image_index.py    --corpus {corpus}
python3 scripts/check_derived_provenance.py           --corpus {corpus}
```

Two failure shapes to check for by hand, because no script reports them: `_ingest_*.md` staging stubs left inside `distillations/{task}/` mean Pass G rows never reached the operator view, and a stray `---` inside a phase table splits it in two — rows after a horizontal rule have no header, so the builder silently stops seeing them as a table.

## Pass I: Source-only audit

**Produces:** the gate decision: deep reference ships or it does not. Plus a Pass I audit log at `{corpus-root}/references/_audit/_ingest_pass_I_{slug}_source_audit.md` recording N claims audited / N source-anchored / N stripped or marker-corrected.

**Procedure:** read the deep reference cold. For every claim (prose, table cell, evidence-class marker, cross-reference, thesis paragraph), ask: can I trace this to a passage in the source? Apply fixes directly to the deep reference as you go—claim strips, marker corrections, blockquote completeness fixes—*before* finalising the audit log. The log records the audit's outcome, not a separate to-do list.

**Write the audit log to disk before reporting, and stamp only after.** An agent that reads heavily and reports without writing produces nothing recoverable; an agent that edits the deep reference and dies before logging leaves damage no one can attribute. Set the frontmatter's `Pass I applied {date}` marker *after* the log exists, so a stamp always implies a log. A stamp without a log is an unearned stamp: treat it as an un-audited reference whose contents may have been half-changed by a dead attempt.

**A fresh subagent is necessary and not sufficient: run a cross-family leg.** The cold read removes *context* leakage — a session cannot see the training-priors it just wrote in as source-grounded. It does not remove *prior* leakage. An auditor from the same model family as the generator holds the same background knowledge, finds the generator's leaked domain reasoning unremarkable, and passes it through as though the source had supplied it.

Measured in 2026 on 58 references across two corpora, each audited by a fresh same-family subagent and, blind, by a different model family: **a cold same-family audit misses 27–31% of the confirmed defects**, a hole in recall shaped like the generator's own priors. **A cross-family audit on its own misses more** (43% on fourteen references, thirteen of them books), because its priors have blind spots of their own. Each leg over-flags up to about 6% of its findings, and the sort below clears them. Neither leg is the audit alone; the two together are.

So: audit in a fresh subagent and, separately, in a *different model family*. Give the cross-family auditor the deep reference and the source and let it trace blind; only after it has written its own findings does it read the first audit and sort every finding into confirmed / missed / over-flagged / disputed. Reading the prior log first anchors it, and an anchored second opinion is not a second opinion. Where a second family is unavailable, say so in the audit log and treat the reference as audited-but-unverified rather than passed.

**Calibration exemplars (mandatory before the cold read).** Before reading the deep reference under audit, read three exemplars from `tests/audit-fixtures/` to anchor what counts as a violation and what counts as clean:

- `tests/audit-fixtures/01-training-leakage.md`: canonical strip case (a biographical claim the source does not make).
- `tests/audit-fixtures/07-marker-mismatch-V-without-verbatim.md`: canonical correction case (a `[V]` marker on a paraphrased sentence).
- `tests/audit-fixtures/12-clean-negative-control.md`: a clean snippet that must not be flagged; over-flagging is as costly as missing real violations.

**Run the verbatim check before the cold read.** `python3 scripts/check-verbatim.py {deep-ref}` checks every `[V]` quotation and blockquote against the converted source and prints each mismatch with the source's own text beside it. Work quotation fixes from that report rather than tracing each quote with its own search; the cold read still traces every other claim. Its `apostrophe`, `initial`, `ligature` and `cross-source` lines need no fix.

The full twelve-fixture set and regression runner are in [`tests/audit-fixtures/README.md`](../../../tests/audit-fixtures/README.md); re-run the regression set when adopting a new model.

**Fail conditions:**

- **TOC-vs-anchors mismatch.** Compare the deep ref's body anchors (chapter / section names referenced in citations and headings) against the full TOC enumerated at Pass A. **If any chapter from the TOC is missing from the deep ref's anchor coverage and the frontmatter does not declare `coverage: partial — operator-approved ...`, fail the audit.** A reader-side decision to skip readable chapters is the canonical violation this gate catches. The fix is not a documentation update; the fix is reading the missing chapters and re-synthesizing.
- A claim that fails the trace test. Strip or flag.
- An evidence-class marker that does not match the source evidence ([V] without verbatim text, [AP] without author content, etc.).
- A cross-reference to another author the source does not cite.
- Task-application guidance smuggled into the deep tier.
- Post-source vocabulary that the source did not have.
- **Source-external analytical framing.** The most common defect by a wide margin, and the one a same-family auditor misses: domain-prior explanation wearing the source-grounded voice. It survives because it is usually *reasonable* and often *true*. Recurring forms, all observed:
  - **Evidence-quality rankings** the source never makes — "the strongest evidence", "highest-grade", "the best-designed study here".
  - **Plausibility judgements** — "physically plausible", "unusually clean", "unremarkable for its date", "a sensible rationale".
  - **Methodological verdicts** — "conclusions outrun the data", "generalises beyond its sample", "unvalidated", "an empirical threshold". These cluster in the source-integrity notes, where the ingester slips from recording provenance into grading the source's rigour.
  - **Evaluative labels** — "classic", "modest", "a surprise", "the most quotable".
  - **Causal or mechanistic explanation** the source reports without explaining. Absence of a citation establishes only that the paper does not cite; it does not establish that a step is *established practice*.
  - **Invented oppositions** — a bounded positive claim restated as a position the author "frames against". See the template note under Pass E.
  - **Market, legal or task framing** in a deep reference — cost advantages, competitive incumbents, registration burden, field maturity. That belongs in a distillation if anywhere.

  The test is not "is this true?" but "which passage says it?" If you find yourself agreeing because you already know it, that is the moment to go and find it in the tape.

If any fail condition surfaces, fix it in place in the deep reference, then re-audit. The deep reference does not ship until Pass I passes.

**Derived-tier re-derivation and stamping.** Passes F and G run *before* this audit, so the light ref and distillations were derived from the *pre-audit* deep — a strip here can leave them carrying content the deep no longer supports (a private-corpus incident: Pass I stripped a fabricated cross-reference table from the deep, but the light ref and distillations kept it). "Derived from the verified deep" is therefore not automatic. After Pass I passes, **re-derive the light ref (Pass F) and distillations (Pass G) from the audited deep, then stamp them with its content hash:**

```
python3 scripts/check_derived_provenance.py --corpus {corpus} --stamp --slug {slug}
```

The stamp is an HTML-comment line (`<!-- derived-from-deep: sha256:… -->`) recording the sha256 of the audited deep. Check mode — `python3 scripts/check_derived_provenance.py --corpus {corpus}` — fails on any derived artefact whose stamp no longer matches its deep (and warns on unstamped ones; adoption is gradual). This turns "derived from the verified deep" from a prose promise into a mechanical guarantee.

A distillation also stamps its task spec (`<!-- derived-from-task: … -->`) and the deep of every other source it names by slug (`<!-- derived-from-neighbours: {slug}@{hash} … -->`), because its cross-reference rows say what those sources say. So this Pass I's edit to the deep can leave *other* sources' distillations STALE, the ones whose rows cite this source. Check mode names the neighbour that moved: re-check those rows against this source's audited deep, fix what it no longer supports, and re-stamp each citing source with `--stamp --slug {that-slug}`. Editing a task spec does the same to every distillation on that axis.

**Calibration:** a Pass I that finds zero issues across a 500+ line deep reference is suspicious. Re-read with fresh eyes; the audit almost always surfaces at least one marker correction or one over-extension to fix.

---

## Failure-mode reference

Failure modes the protocol catches. Most are caught at the pass that introduced them; Pass I is the backstop.

- **Training-data leakage.** A claim about the author's career, later works, or position in a "field" the source does not name. Strip in Pass C; Pass I is the backstop.
- **Post-source vocabulary.** A concept from the author's later work introduced into a deep ref of an earlier work. The author did not have that vocabulary at the time.
- **Cross-corpus drift.** A connection to another author the source does not cite. The deep ref records what the source cites, not what the corpus ingester knows.
- **Task-application guidance smuggled into the deep.** Diagnostic questions, anti-patterns, and worked examples belong in distillations (Pass G), not in the deep reference.
- **Verbatim accuracy slips.** Apparent corrections (capitalisation tidying, a changed word, dash or punctuation mark) introduced into blockquotes in Pass D. Re-verify. Apostrophe and quote-mark style is not a slip.
- **Unverified verbatim in distillations.** A distillation quotation marked `[V]` that is not word for word in the source: reworded, silently shortened, or credited to the wrong source. A `[V]` quotation in a distillation matches the converted source exactly; correct it to the source's words, or paraphrase it without the marker.
- **Silent partial coverage (source-side).** Proceeding with the deep ingest when the available text is incomplete and not flagging the gap. The post-conversion check and the source-integrity pre-flight catch this before Pass A by hard-stopping on incomplete sources, including a conversion cut short by a size cap; the operator decides whether to approve partial-coverage ingestion with an explicit `coverage: partial — operator-approved ...` rationale.
- **Silent partial coverage (reader-side).** Skipping readable chapters during Pass C and labeling the deep ref `coverage: partial` after the fact. This is the canonical violation of the source-integrity rule: if you can read the chapter, you must read it. If context budget runs out mid-read, stop and escalate to the operator so they decide whether to continue, defer, or abandon. The `coverage:` frontmatter field is operator-consent only — the model never sets `coverage: partial` unilaterally. Pass I's TOC-vs-anchors check is the audit backstop.
- **Derived-tier lag (light / distillation stale against the audited deep).** Passes F and G derive *before* Pass I edits the deep, so a Pass I strip can leave the light ref or a distillation asserting content the deep no longer supports. Mitigation: after Pass I, re-derive the derived tier and stamp it with the deep's content hash (`scripts/check_derived_provenance.py --stamp`); check mode fails on a stale stamp. Origin: a private-corpus incident (2026-07). The same lag reaches across sources: a distillation's cross-reference rows go stale when a *neighbour's* deep moves, which the neighbours stamp catches.
- **Image-scope mismatch.** Starting a run with image classification enabled and then truncating mid-classification because of context budget produces silent partial coverage of the image axis. Mitigation: declare image scope at pre-flight (full or text-only); if full classification doesn't fit one session, accept text-only scope explicitly and run the [`ingesting-images`](../ingesting-images/SKILL.md) skill later; never auto-cap inside a single ingestion.
- **Image-classification heuristic drift.** Bulk-classifying images by filename, file extension (PNG vs JPEG), file size, or any other metadata-only signal. Empirically unreliable across publishers: the same format carries diagrams in one source and photographs in another. Visual inspection per image is non-negotiable.
- **Parallel-agent index race.** Pre-migration ingestion subagents edited canonical `REFERENCE-INDEX.md` / per-task distillation indexes / `IMAGE-INDEX.yaml` simultaneously and corrupted merges via read-modify-write loss. The mechanical-index pipeline removes the race surface: per-source agents write only to their own staging paths (`_planning/extracted/{corpus}/{slug}.json`, `_planning/staging/{corpus}/refs/{slug}.json`); the canonical JSON indexes are derived once after every per-source step completes by the build scripts in `scripts/build_indexes/`. See [`SKILL.md`](SKILL.md) "Parallel-batch operations" and "Pass H: Cross-reference" above.

### What the protocol does not catch (and is calibrated against)

The failure modes above are protocol-internal: things the nine passes are designed to catch, and the audit fixtures in `tests/audit-fixtures/` calibrate Pass I against. Three upstream failure modes sit outside the protocol's scope and are named here so the protocol's discipline is not mistaken for the architecture's full warrant.

- **Form-vs-meaning gap** (Bender, Gebru, McMillan-Major & Shmitchell, *On the Dangers of Stochastic Parrots*, FAccT 2021). The protocol verifies that the deep ref's claims trace to the source text. It does not verify that the source's prose corresponds to anything the source's prose is about. A fluent-but-ungrounded source ingested under full 9-pass fidelity produces a fluent-but-ungrounded deep ref. The architecture's response is corpus selection: see [`docs/architecture/known-architectural-limits.md`](../../../docs/architecture/known-architectural-limits.md) "Corpus-to-world fidelity is not a property of this architecture".
- **Classification-axis drift** (Bowker & Star, *Sorting Things Out*, MIT Press, 1999; Hacking's looping effect). The protocol verifies the deep ref against the source. It does not verify that the task axis the deep ref will be projected onto is the right cut. Task-axis choices are world-making: see [`docs/architecture/known-architectural-limits.md`](../../../docs/architecture/known-architectural-limits.md) "Classification axes as world-making".
- **Consultant-frequency drift in source selection** (Chris × Larman, *LLM-Epistemology Corollary*, 2026). The protocol assumes the operator selected a primary source. The protocol does not catch a consultant-derivative repackaging of a primary source: it audits whatever was ingested. The architecture's response is the corpus the operator chooses to ingest against: see [`docs/architecture/llm-epistemology.md`](../../../docs/architecture/llm-epistemology.md).

These three sit *upstream* of the protocol. Naming them here keeps the protocol's claim precise: it verifies fidelity to source, not the source's relation to the world, not the task-axis choice, not the source-selection prior.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall