agentsope/SkillAlchemy / skills/agentsop-idempotent-ingestion
agentsop-idempotent-ingestion
VerifiedRe-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — **ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op** — plus the docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` + `DocstoreStrategy`), the delete-propagation problem, and cross-framework equivalents (LangChain `index()` + `RecordManager`, manual hash table). ENHANCE overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but the re-ingest-correctness contract is not surfaced.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.8 KB |
| SKILL.md | skill md | 27.2 KB |
| intermediate/operation_candidates.json | config | 8.3 KB |
| references/R1-source-evidence.md | markdown | 6.1 KB |
| references/R2-cross-framework.md | markdown | 6.2 KB |
README.md
View on GitHub# idempotent-ingestion skill
A coder-agent skill for **re-ingest correctness**: making an ingestion pipeline
safe to run more than once over a changing corpus, so the second (and
thousandth) run never duplicates or corrupts the index.
**Version**: 0.1.0
**Status**: ENHANCE overlay over `[[llamaindex]]`. Covers LlamaIndex
`IngestionPipeline` + docstore, LangChain indexing API + `RecordManager`, and a
framework-free manual hash table.
## The core rule
> **Ingestion must be idempotent: a document's content hash decides
> insert / update / skip — so re-running the pipeline over unchanged docs is a
> no-op.**
The interesting run is the *second* one. A correct pipeline keeps a persisted
hash ledger (a "docstore") separate from the vector store, consults it before
embedding, and inserts / updates / skips per document. Deletes need an explicit
switch and a complete-snapshot guard.
## Why this overlay exists
Phase B found that `[[llamaindex]]` names `IngestionPipeline` + docstore +
`UPSERTS_AND_DELETE` as a production must-have (its OP-08 / A10) but stops at
"use it" — the re-ingest-correctness *contract* is never surfaced. This is
foundational for any production RAG that re-indexes, so this overlay codifies:
- what idempotency actually means (no-op-on-re-run),
- how the document hash + docstore decide insert/update/skip,
- why deletes don't propagate for free,
- how to *prove* it with a twice-run test,
- and the cross-framework equivalents.
## What this skill answers
- How to make a scheduled / CI / webhook-triggered ingestion safe to re-run.
- How to wire a docstore so unchanged docs cost zero embedding tokens on re-run.
- When to use `UPSERTS` vs `UPSERTS_AND_DELETE` vs `DUPLICATES_ONLY`.
- How to handle a slightly-edited document (re-embed all, or split the source?).
- How to handle deleted source docs without accidentally purging live content.
- How to debug "retrieval returns duplicates" / "deleted file still shows up".
## Files
- `SKILL.md` — main skill (7 sections: activation → mental model → SOP →
operation models → dilemma cases → anti-patterns → cross-framework table).
- `references/R1-source-evidence.md` — every claim traced to the base
`[[llamaindex]]` skill + primary LlamaIndex / LangChain docs.
- `references/R2-cross-framework.md` — full code for LlamaIndex / LangChain /
manual hash table, with delete-propagation variants and the decisive
"second run" table.
- `intermediate/operation_candidates.json` — OP-01..09 in machine-readable
Trigger / Action / Output / Evidence form.
## Quick start
If you just need to make one pipeline re-run-safe today:
1. Read `SKILL.md` §3 Stage 1-2 — attach a **persisted** docstore and pin
**stable** doc ids.
2. Pick a strategy (§3 Stage 3): `UPSERTS_AND_DELETE` if the corpus mirrors a
source that can shrink; `UPSERTS` if append-mostly.
3. Add the **twice-run test** (§3 Stage 4 / OP-06) — the second `run()` must
process 0 nodes and not grow the vector count. This is the gate.
4. Persist the docstore after each run and schedule it (§3 Stage 5).
## Anti-patterns it catches
`from_documents()` on a cron tick · `IngestionPipeline` with no `docstore=` ·
`SimpleDocumentStore` never persisted · random per-load doc ids · timestamps in
the hashed payload · `DUPLICATES_ONLY` mistaken for cross-run dedup ·
upsert-only on a deletable corpus · blind `UPSERTS_AND_DELETE` on partial input ·
no twice-run test in CI.
## Source basis
- Base skill: `[[llamaindex]]` (`llamaindex-sop-skill/SKILL.md`) OP-08, A10.
- LlamaIndex — *Ingestion Pipeline* + *Document Management*
(`developers.llamaindex.ai/.../loading/ingestion_pipeline/`).
- LangChain — *Indexing* (`RecordManager`, `cleanup`)
(`python.langchain.com/docs/how_to/indexing/`).
Date stamp: May 2026. `DocstoreStrategy` / `RecordManager` APIs move; re-check
the current docs before relying on exact enum / kwarg names.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |