dimagi-internal/ace / skills/ocs-knowledge-refresh
ocs-knowledge-refresh
UnverifiedAdd documents to a chatbot's RAG collection and republish it. Needs the bot's experiment and collection ids and the docs.
OCS Knowledge Refresh
Uploads documents into a chatbot's existing RAG collection, waits for
indexing, and publishes a new chatbot version. Inside /ace:run the bot is
the run's Phase 5 chatbot and the documents are the four Phase 6 training
documents. On a live program the caller names the bot and the documents
directly (e.g. a revised FAQ for a bot ACE did not build), and the skill does
the same thing with no run.
Uses: ocs_upload_collection_files, ocs_wait_for_collection_indexing,
ocs_publish_chatbot_version, ocs_get_chatbot_embed_info.
Why this is a separate skill from ocs-agent-setup
ocs-agent-setup used to do both halves. That created a cycle in the
dataflow graph, because the two halves have different inputs:
| Half | Needs | Available in |
|---|---|---|
Create + publish the bot → emits widget_url | PDD, app summaries, opp details | Phase 5 |
| Populate the knowledge base | the four training documents | Phase 6 |
Phase 6's guides embed the widget_url from the first half; the first half's
collection wants the documents from Phase 6. Bundled into one step that runs in
Phase 5, that is unsatisfiable — and the way it was papered over is instructive:
Phase 5 read 6-qa-and-training/* tolerantly, so the files being absent was
not an error. Nothing re-uploaded them afterwards, so every opportunity's
chatbot shipped without the four documents its users ask about, and every
check stayed green (the tolerant read succeeds, the collection indexes cleanly,
and Phase 5's smoke gate asks three universal Connect questions that need none
of them).
Splitting the node at its real dependency seam removes the cycle rather than working around it. Phase 5 now consumes nothing from Phase 6, this skill consumes nothing from Phase 5 except the state file it wrote, and the pipeline is a plain DAG:
5. ocs-agent-setup (create + publish) → widget_url
6. training-* (consume widget_url) → the four documents
6. ocs-knowledge-refresh → documents into the collection
A first-class step, not a corrective second pass. ocs-agent-setup --reindex
(0.13.1022–0.13.1026) was the fix-up version of this and is retired.
Inputs
| Parameter | Used for | Resolver |
|---|---|---|
experiment_id | the bot to republish | explicit arg; default: runs/<run-id>/5-ocs/ocs-agent-setup.md |
collection_id | the RAG collection to upload into | explicit arg; default: same file; else ocs_get_chatbot / the program context doc |
pipeline_id | recorded alongside the bot | optional; default: same file; ocs_get_chatbot_pipeline_id when omitted |
files[] | the documents to index (Drive file ids or local paths) | explicit arg; default: the four Phase 6 documents below |
run_id | whether to write back to a run | optional; absent = live-program mode (no run write-back) |
Default files[] inside a run:
| Phase 6 artifact | Used for |
|---|---|
runs/<run-id>/6-qa-and-training/training-llo-guide.md | RAG content |
runs/<run-id>/6-qa-and-training/training-flw-guide.md | RAG content |
runs/<run-id>/6-qa-and-training/training-faq.md | RAG content |
runs/<run-id>/6-qa-and-training/training-quick-reference.md | RAG content |
Two modes, one procedure. Run mode (run_id bound, ids resolved from
Phase 5) is the pipeline step described below. Live-program mode (explicit
experiment_id + collection_id + files[], no run) runs the same Steps 1-3;
Step 0's re-run check and Step 4's write-back switch to the live-program
variants named in each step. Never halt for a missing 5-ocs/ocs-agent-setup.md
when the caller passed the ids.
Existing objects
- A publish changes the live bot for everyone.
ocs_publish_chatbot_versionmakes the new collection contents the published default, and the public widget link serves ONLY the published version, so a refresh without Step 3 changes nothing a user sees, and a refresh with it changes the bot for every opportunity and worker that embeds it. On a live program, get owner approval for the document set before Step 1 (partner-on-connect-program § Changing a live program). - Uploads append; they do not replace. Re-uploading a document already in
the collection indexes a duplicate. Step 0's per-document check is what
prevents it, and in live-program mode it is the ONLY guard (there is no
last_reindexed_atto read). - A shared collection is shared. If
collection_idis also attached to other bots (checkocs_get_chatboton each bot that embeds it, or the context doc), the upload reaches them too, though they serve it only after their own republish. Name every affected bot in the product record.
Products
-
runs/<run-id>/6-qa-and-training/ocs-knowledge-refresh.md— which documents were uploaded, the resulting collection file count, and the publishedversion_number. -
Updates
5-ocs/ocs-agent-setup.mdwithlast_reindexed_atand the refreshedversion_number. -
Writes
phases.ocs-setup.products.ocs_chatbot.knowledge_sources(pluslast_reindexed_atand the refreshedversion_number) torun_state.yaml. This skill is the only producer ofknowledge_sources— ace-web's public run-summary page is the consumer. See Step 4.This is a cross-phase write, and it is deliberate.
ocs-agent-setupis the declared producer of that file; a Phase 6 skill appending two fields to it is the one place this skill reaches outside its own phase. It is a write, not a read, so it creates noqa-and-training -> ocsdependency and no cycle —lib/artifact-manifest.tsmodels producers and consumers, and has no way to express it, which is why it is stated here instead. The fields live with the chatbot's other identifiers because that is where an operator looks for chatbot state, andocs-agent-setup § Productsdeclares both as optional fields written from Phase 6. If this ever becomes a read as well, it is a cycle again — split it before it does.
Process
Step 0: Preconditions — say what is missing, do not improvise
Live-program mode (explicit ids, no run): confirm the bot and collection
exist (ocs_get_chatbot({experiment_id}), ocs_list_collection_files({collection_id})).
Either one missing is a hard stop: never create a bot here. Then run the
per-document check below ("was THIS file ever uploaded?") against
ocs_list_collection_files by name. That is the whole re-run guard in this
mode, then go to Step 1.
Run mode: read 5-ocs/ocs-agent-setup.md.
- Absent. Phase 5 did not run or did not complete. No-op with a loud
note — record
status: skippedwith the reason. Do NOT create a chatbot; that is Phase 5's job and doing it here would produce a bot with no widget-handoff and no smoke gate. - Present. Take
experiment_id,collection_id,pipeline_id.
Then check last_reindexed_at before uploading anything — this skill is
re-run. Phase 6 is retried, /ace:step ocs-knowledge-refresh is a supported
manual entry point, /ace:iterate targets phases 3+4+6 by default, and a fork
replays the tail of a run. ocs_upload_collection_files appends; it does
not replace. Uploading the same four documents into the same collection a
second time leaves duplicates in the index — worse retrieval, a second 5–10
minute indexing wait, and a file count that no longer means what the product
artifact says it means.
last_reindexed_atabsent → first pass. Continue.last_reindexed_atpresent, and every training document is older than it (compare DrivemodifiedTime) → no-op. Recordstatus: skippedwithreason: already-currentand the existingversion_number. Do not re-upload and do not re-index.last_reindexed_atpresent but a training document is NEWER → the documents changed after the last refresh, so the collection is stale. See the sub-cases below, because the obvious remedy does not exist.
Use ocs_list_collection_files + ocs_remove_collection_file. Both exist
as of 0.13.1145 and this branch is now executable as written. It previously said
"list the collection and remove the four by filename" while neither atom
existed, and the only removal atom was ocs_delete_collection, which destroys
the WHOLE collection and its pipeline wiring — never a substitute.
ocs_list_collection_files({ collection_id })
-> { files: [{ collection_file_id, file_id, name }] }
ocs_remove_collection_file({ collection_id, file_id }) // the stale copy
ocs_upload_collection_files(...) // the new one
ocs_wait_for_collection_indexing(...) // then publish a version
file_id is NOT the id an upload gave you. There are two id spaces and they
do not overlap: ocs_upload_collection_files returns — and
ocs_wait_for_collection_indexing polls — the collection_file_id, while
removal takes the file_id. Measured live on collection 568: 42083…42466 versus
62981…63412, zero overlap. Always read the file_id from
ocs_list_collection_files; never reuse an upload's return value for a removal.
So, splitting by what actually changed:
- A document never uploaded before is NEW → just upload it. No prior copy exists, so appending duplicates nothing. This is the common case — a document that was MISSING last pass and exists now, and it needs no removal at all.
- A document that WAS uploaded and has since changed → remove the stale copy
by
file_id, upload the new one, re-index, publish a version. Removal alone does not re-index, and an un-republished chatbot keeps serving the old embeddings. - A stale document you cannot regenerate (someone else authored it, or the
source is gone) → keep the older option: upload an authored correction
sheet, a short dated document declaring itself authoritative over
conflicting statements and naming which identifiers are retired. Phase 5 uses
this pattern in these collections (
current-design-and-changes-<date>.md). Record in the product artifact that the stale copy is still indexed, and why.
Check last_reindexed_at against reality before trusting it. It is written
by hand and has been wrong: on the run above it read 11:58Z while the pass it
described actually completed around 12:12Z. Two documents modified in between
therefore looked NEWER than the refresh that had already indexed them, and a
naive modifiedTime comparison would have demanded exactly the
delete-then-reupload this branch cannot perform. When the timestamp is close to
the documents' modifiedTime, prefer the per-document question — "was THIS
file ever uploaded?" — over the aggregate one.
ocs-agent-setup § Step 0 short-circuits the same way on a re-run; this skill
did not until 0.13.1037.
Then read the four training documents. Do not tolerate a missing one silently — that is the failure this skill exists to end. Record each missing document by name in the product artifact and in the step record, upload the ones that exist, and carry on. A partial refresh that says which documents are missing is recoverable; a silent one is not.
Step 1: Upload
ocs_upload_collection_files({ collection_id, files: [...] }) with the training
documents that exist.
Download each document with drive_read_file({exportAs: 'text/markdown', writeToPath: <abs tmp path>}) into ONE directory, then — before any upload
call — run the strip pass over that directory:
ACE_ROOT="${CLAUDE_PLUGIN_ROOT:-$(python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json'))); print(d['plugins']['ace@ace'][0]['installPath'])")}"
node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/strip-inline-data-uris.ts" <download-dir>
node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/redact-kb-answer-keys.ts" <download-dir>
node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/screen-kb-deny-list.ts" <confidentiality.yaml|none> <download-dir>
The third screens every file against the opp's inputs/confidentiality.yaml
(llo_facing_deny_list) — this collection is LLO- and FLW-facing, so a
deny-listed party (a confidential funder) must not reach it
(dimagi-internal/ace#3223; repro connect-fi/20261018-1440, chatbot 14329).
Download the yaml OUTSIDE the download dir, or pass none when the opp has
none. Exit 1 means a file names a deny-listed party, or is not text and
cannot be screened — do NOT upload; stderr names the file and term. Rewrite
or drop that document and re-run; never edit the deny list to pass. Exit 2 is
a harness error (missing or malformed input) — fix it, do not skip the screen.
Full rule: ocs-agent-setup § Step 5.
The first rewrites in place and prints one JSON line of per-file
{bytes_before, bytes_after, payloads_stripped}. Record those numbers in the
product artifact. A non-zero exit means a payload survived a shape the
stripper does not know about — do NOT upload; report it on
dimagi-internal/ace#1827 with the file.
The second removes any assessment answer key a training document quotes (an FLW guide walking through the Learn quiz, a FAQ answering "what is the answer to question 3") — this collection serves the workers those assessments gate (dimagi-internal/ace#2869). A non-zero exit means a key survived — do NOT upload; report it on ace#2869 with the file.
Four constraints carry over from ocs-agent-setup § Step 5 and are not
restated in full here — read them there:
-
Never index an assessment answer key (ace#2869) — the redactor above is the enforcement.
-
Spreadsheets cannot be indexed (
is_index: truecollections use thefile_searchallowlist). Upload a text extract instead. -
Never index
2-scenarios/pdd-to-test-prompts.md— it carries the deep-QA answer key, and indexing it makes the eval grade the bot on documents containing its own expected answers. -
Never index a base64 payload.
exportAs: 'text/markdown'inlines every embedded image as adata:URI, so an illustrated guide arrives at 8–28x its prose size and that bulk becomes indexed retrieval content. Onbednet-check-2-visit/20260828-0629the four-document pack was 91% base64 by volume and the FLW guide alone was 264,538 bytes carrying ~9 KB of prose (16 screenshots). Nothing failed — the upload succeeded and indexing reportedready: true, pending: 0— the chunks just competed with real prose for the bot'smax_resultsretrieval slots.scripts/strip-inline-data-uris.tsabove is the fix; it is mandatory, not advisory. The images are not lost: they live in the published Google Doc the reader opens, which is also whattraining-*-evalcounts (docs_get→inlineObjects), so stripping the uploaded copy costs nothing downstream.
Step 2: Wait for indexing
ocs_wait_for_collection_indexing({ collection_id, timeout_sec: 300 }).
This is the 5–10 minutes. It is the cost of the bot knowing its own training material; do not skip it to save wall-clock, and do not report success before it returns.
Step 3: Publish
ocs_publish_chatbot_version({ experiment_id }). The prompt is unchanged —
this republish exists so the new collection contents are live. Capture the new
version_number.
That return is the POST-publish published default. Before ace#1828 it was a
page scrape that could lag the publish by one, and on
bednet-check-2-visit/20260828-0629 it did: the atom returned 2 for a publish
that created 3, and this step's write-back would have put a stale version into
run_state.yaml. The atom now reads it back from the API. Two things still
worth knowing:
task_id: "none"is not a signal. It is present on a publish that did real work. A return whose number looks unchanged is not evidence of a no-op — do not republish to chase it.- If
sourcecomes backhome-page-badge, the API read failed and the number is the old scrape. Corroborate withocs_inspect_chatbot({ public_id, version: 'default' })→version_numberbefore writing it in Step 4. That call is also the independent authority in general — but read it underversion: 'default', never the top-levelversion_number, which is the working/next counter and runs ahead.
Step 4: Write back
Live-program mode: there is no run file to update. Record the result
(documents uploaded, documents skipped and why, collection file count, new
version_number, every bot affected) in the program context doc's Decisions,
or on the board task that asked for the refresh, and stop. The run write-back
below does not apply.
Run mode: write the product artifact, and update 5-ocs/ocs-agent-setup.md with
last_reindexed_at (ISO timestamp) and the new version_number.
last_reindexed_at is the operator's check. Its absence on a run whose
Phase 6 completed means the chatbot never received the training documents.
So read both write-backs back, and halt loud if either field is missing. A
returned write is not a written field, and this half of Step 4 has no schema
behind it: the run_state.yaml patch is checked by
validateAs: {kind: 'phase-products'}, but the ocs-agent-setup.md edit is
free text that fails silently when it is skipped. After writing, re-read
5-ocs/ocs-agent-setup.md and assert last_reindexed_at is present carrying
the timestamp you just wrote, and re-read run_state.yaml and assert the same
of phases.ocs-setup.products.ocs_chatbot.last_reindexed_at. This is Step 3's
source: api-over-page-scrape discipline applied to the write path.
Skipping it is not hypothetical, and it poisons Step 0. On
bednet-check-2-visit/20260902-1555 the 2026-09-05 pass uploaded all four
documents, published v3, and wrote Status: done — while
5-ocs/ocs-agent-setup.md sat untouched at the revision Phase 5 left it,
still reading "last_reindexed_at is absent" three days later. The work was
done and the only field that records it said otherwise, so an operator reads
"the bot never got its training documents" about a bot that has them. Worse,
Step 0 branches on this field: a later pass reads absent, takes the
first-pass branch, and appends duplicates into a collection that already
holds the documents. That run escaped only because its documents had genuinely
been rewritten, making the replace branch correct by luck. ace#2107.
Then write knowledge_sources into
run_state.yaml → phases.ocs-setup.products.ocs_chatbot — a list of short
human-readable phrases naming what the bot was actually given:
update_yaml_file(run_state_file_id, merge: 'deep',
validateAs: {kind: 'phase-products', phase: 'ocs-setup'},
patch: {phases: {'ocs-setup': {products: {ocs_chatbot: {
version_number: <new>,
last_reindexed_at: <iso>,
knowledge_sources: [
'the design doc',
'the training pack (LLO manager guide, FLW training guide, FAQ, and quick reference card)',
'the Learn and Deliver app guides',
...
]}}}}})
This is the only producer of that field, and it is load-bearing. ace-web's
public run-summary page reads it (apps/opps/summary.py :: _knowledge_sources,
accepting knowledge_sources / knowledge / indexed_sources) and renders
"Ask questions about this opportunity. It was given <list>." Until
ace-web#740 that sentence was a hard-coded claim — "Trained on the design
doc, training pack, and app guides" — derived from nothing, and on
spark-facilitator/20260820-0817 it was false: the collection held 16 files
and none of the training pack. ace-web made it data precisely because this
skill makes the claim true for some runs and leaves it false for others, and a
constant string cannot tell them apart.
So the field is not decoration — it is the evidence for a claim an external
partner reads. Omit it and the page correctly falls back to saying nothing
about what the bot knows, which is honest but throws away the whole point of
having run the refresh. Write it in the same call that records
last_reindexed_at.
Phrases, not filenames. They are rendered into an English sentence for a
reader with no ACE context, so 'the training pack (LLO manager guide, FLW training guide, FAQ, and quick reference card)' reads correctly where
'19-training-quick-reference.md' does not. Group related files; keep the list
to ~5 entries. Describe only what is actually indexed — a document you
skipped (see Step 0) must not appear here.
Enforced: ocs_chatbot is .passthrough() in lib/phase-products-schema.ts,
so validateAs accepts the field rather than rejecting it as drift. The
read-back requirement above is enforced by
test/skills/ocs-knowledge-refresh.test.ts.
Verdict
No paired -eval. The check is structural and lives in the product artifact:
documents uploaded vs. documents expected. A count mismatch is a [WARN] with
the missing filenames; zero uploaded when Phase 6 produced documents is a
[BLOCKER].
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-10-19 | Step 1 runs scripts/screen-kb-deny-list.ts over the download dir before upload; exit 1 (a file names a party in inputs/confidentiality.yaml's llo_facing_deny_list) blocks the upload. Repro connect-fi/20261018-1440, chatbot 14329. ace#3223. | ACE team |
| 2026-10-18 | Inputs are parameters (experiment_id, collection_id, files[]) with the Phase 5/6 run paths as defaults; live-program mode (no run) for Step 0 and Step 4; ## Existing objects (publish is live for every embedder, uploads append, shared collections). ace#3148. | ACE (build-vs-maintain audit) |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 196k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |