Skip to content

ObneyAI/orc / orc-skills/orc-prompt-caching

orc-prompt-caching

Verified

Configure and verify provider prompt caching in ORC nodes and composed trees, including shared delegates, model changes, concurrent users, warming and cache misses. Use for reusable prompt prefixes and inference cost; application result caches are a different mechanism.

references/cache-boundaries.md

View on GitHub
# Boundaries that determine reuse

## Three mechanisms to keep distinct

Provider prefix caching reuses inference work. ORC's workflow build/module caches avoid local preparation; application memoization reuses results. A provider-cache hit still invokes the model and must produce a new contract-valid answer from the current typed inputs. None of these mechanisms gives once-only external effects, authorization or original-tick recovery.

On this ORC path, the field guide and actual runtime values follow the marked instruction. A large reference supplied as a blackboard value is not automatically pulled into that marked prefix. Deliberately authored necessary public static reference material may belong with the static instruction; do not move private/volatile evidence there or remove descriptions to reduce bytes. Provider implicit caching may separately cover more of a later message.

A prefix belongs to the provider/model cache scope and actual rendered request, not to an ORC node ID. The qualification record includes changed authored node names with identical serialized bodies and successful reuse; changed source values were also handled correctly. Different models do not establish shared KV state: inventory and measure their roles independently. The mixed-model proof uses ordinary per-node model settings, not a transfer of another model's cache.

Stable text is insufficient if something before it changes. Inspect tools, tool ordering, output schemas and descriptions, message boundaries, images and reasoning settings as well as instructions. A provider can render schemas before message text even when HTTP JSON properties appear elsewhere. Compare the actual sent contract; JSON key placement is not provider token-order proof.

On the qualified released Haiku route, changing output descriptions, field names/types, output mode or early instructions wrote new prefixes; runtime source and nested input-description changes retained the marked instruction's read. This reflects where that stack put those fields, not permission to remove descriptions or a guarantee for all renderers. Preserve required meanings; optimize duplication only after measuring it.

## Released and candidate layout

| Stack | Enabled function/strict-JSON layout | Qualification |
|---|---|---|
| ORC 4966c8e / SIO e61e11a9 / client f5fa99d8 | One user message: marked static first content part, then variable guide/values/images | Hosted authored-tree proof and request controls below |
| Private cache-boundary candidate, not committed/published at this draft | Marked instruction in first USER message; unchanged guide/values/images in a second USER message | Same-authority request control supports Qwen/Haiku; candidate integration/role proof must be checked separately before adoption |

The existing `:prompt-cache?` flag selects the candidate's improved enabled layout; there is no new layout parameter in this draft. Cache-disabled, blank-instruction and marker-mode behavior must be checked against the consuming revision. Do not implement a model-name switch, application serializer or parser to imitate an unreleased change.

In the released adapter path, message markers survive, while the tool-level marker ORC constructs is dropped during generic tool normalization. A message breakpoint can still cache an earlier rendered tool prefix. Neither marker presence nor absence alone determines implicit caching.

The separate-system/separate-user diagnostic improved Qwen reuse but changed instruction authority. The later two-USER control reproduced reuse without that promotion. Same authority avoids that explicit change; it does not prove arbitrary semantic equivalence for all applications under different message boundaries.

## Roles and modes

The 296-call hosted cohort uses ordinary authored LLM leaves for routing, methods, analysis, parallel work and synthesis. It is not a live cache proof of every LLM-capable node type.

At the released source, conversational conditions retain node options; finite decisions also carry options, but offered choices become instruction content and can change the reusable prefix. Native decision protocols are a different request path, with no cache-marker or native cache-usage qualification here. Researcher prime/default marker mode does not acquire the function/strict-JSON cache layout just because a flag is set. Inline and emitted calls have their own option/default paths. Verify the actual prime/subcall/generated requests through the consuming source and [model/SIO](../../orc-model-sio/SKILL.md), rather than assuming flags propagate to all descendants.

## A discriminating miss investigation

Hold model, serving route, typed task, schema and completion allowance fixed. Compare a first request, an exact full repeat and a changed-runtime-input request. Then inspect the first differing prefix component and, if needed, run a bounded same-authority message-boundary counterfactual. Preserve original bytes and failures; do not retry until one favorable response erases them.

GLM's qualified route cached an exact full repeat (5184 tokens) but not changed-input prefixes in the tested layouts. An upstream debug echo removed the explicit marker; the provider also supports implicit caching, so that removal does not complete the causal explanation. Keep this unresolved rather than declaring no caching capability.

Qwen's echo PRESERVED the marker yet rewrote the prefix on both changed-input calls. It reused changed-input prefixes with separate messages, including the two-USER control. Gateway marker loss is therefore not the observed Qwen explanation; message-boundary sensitivity is demonstrated, provider internals remain unobserved.

Debug echoes report gateway-transformed bodies, not independent provider packet capture. Streaming debug behavior does not prove blocking transformations are identical. Keep raw SSE/echo artifacts private and access restricted; publish only allowlisted structural paths, hashes and counters. Never copy headers, keys, whole exception dumps or unreviewed provider metadata into skill examples.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall