Skip to content

ObneyAI/orc / orc-skills/orc-prompt-caching

orc-prompt-caching

Verified

Configure and verify provider prompt caching in ORC nodes and composed trees, including shared delegates, model changes, concurrent users, warming and cache misses. Use for reusable prompt prefixes and inference cost; application result caches are a different mechanism.

references/warming-and-serving.md

View on GitHub
# Warming useful prefixes in many-user trees

Use [operational checks](operational-checks.md) for the no-call loaded-source check, guarded public `llm/predict` example and counter/owned-attempt mapping. Loading the example never warms anything; invocation requires an explicit paid-call choice and a host budget.

## Choose warm targets by actual request families

List each reached generation role with its model, serving policy, output contract/version, necessary stable material, likely visit rate and time between visits. Group only requests whose eligible rendered prefixes really match. A generic dummy warmup with a different output contract, tool catalogue or instruction does not establish warmth for the target node. Keep the real schema/descriptions, mode, tools and static task prefix in a model-only warmer. A reusable analysis child reached through different parents can share its internal request contract; a different specialist instruction or output schema is a separate warm target unless measured otherwise.

First normal use can establish a prefix. Explicit warming is an additional model call with write/input/output cost. Use bounded model-only requests with a valid declared output contract and harmless synthetic suffix data; do not execute a business tree's tools, payments or other effects just to prime its models. Do not assume an intentionally truncated/invalid answer establishes the same useful boundary. A prompt-cache miss must not become a workflow failure, duplicate action or unauthorized alternate route.

Favor needed common prefixes and frequently revisited specialists. Warm rare branches only when their latency objective and expected reuse justify it. Avoid an unconditional daemon that continually rewrites every node/model cache: sparse branches may expire, writes can carry premiums, and warmups compete with user traffic. Treat TTL and hit probability as observed/model-specific conditions, not application correctness guarantees.

For estimating value, compare total measured warmup+write+read+output cost with the corresponding workload, including misses and fallback models. A break-even calculation based on current quoted rates is a forecast, separate from observed actual cost. Keep time to first output/provider timing apart from application startup, serialization, tracing and datastore overhead. Small samples support ranges and scoped observations, not universal tail-latency claims.

## Multiple prefixes, users and models

The qualified Haiku control reused both A and B prefixes after A→B→A→B interleaving. The hosted tree independently cached eight authored roles, including a rarely selected specialist and synthesis; one owner's three model-only roots preceded twelve other users whose 96 calls all read cached prefixes with no writes. Each new user/work retained its own typed source, root/child identities and exact canonical results. Actual HTTP overlap reached 12. This demonstrates reuse within the tested account/routes, not an infinite capacity claim.

Keep user/work facts out of the public shared instruction prefix and keep domain authorization independently enforced. Placing facts in a suffix preserves reuse; it does not promise a provider will never cache that suffix implicitly. Cached computation is not a new source of evidence or permission. Preserve disjoint parallel writes, child blackboard isolation, source identity/revision checks and canonical completion recording. Use [runtime concurrency guidance](../../orc-runtime-bootstrap/references/concurrent-serving.md) when building the many-user application; within-tree parallelism, many independent root users and multiple server instances are separate contracts.

Use ordinary per-node `:model` and options for mixed-model trees. Each model's first role request may be cold even when another model is warm. A model fallback still performs the same typed task under its normal policy/budget; do not replay effects or weaken parsing to improve cache outcomes. Include fallback model eligibility/warming in cost planning only where it is actually useful.

## Affinity and isolation

OpenRouter `session_id` is a routing/grouping hint, not an ORC identity or authorization boundary. The inspected public ORC execute API does not forward arbitrary per-run extra-body. In this experiment, an explicit transport proxy attached only session metadata AFTER native adapter serialization and retained before/after bodies. That is experimental plumbing, not a newly available per-execution DSL option. Static node/provider settings are different; do not race global provider configuration across users.

The shared/separate/absent session comparison ran sequentially and selected Google for Haiku. First shared traffic included new writes; the later separate and absent groups were already warm. It does NOT show that separate sessions prevent sharing, that a shared session is required, or that a particular key maximizes production throughput. Explicit provider.order can disable OpenRouter sticky routing. Use controlled fresh cohorts and actual resolved endpoints when isolating affinity effects. [OpenRouter documentation](https://openrouter.ai/docs/guides/best-practices/prompt-caching)

Provider credential/organization/workspace/model/processing-region boundaries differ from application user IDs and tenants. Direct Claude documents workspace isolation on some platforms and organization isolation on others; OpenAI documents organization/processing-region routing scope and optional cache accounting keys. Those direct-provider rules do not establish how an intermediary partitions its account. This campaign proves neither cross-account nor cross-region sharing, nor cache probing isolation between app users. Check the selected provider's current policy and enforce app authorization independently. [Claude](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching)

## Idle branches and measurement

Warm common and rare prefixes independently; touch only the common prefix during the idle interval, then compare both with actual cached/read/write counters. Record actual per-prefix dispatch/completion times and schedule jitter. Declare what clock/retention rule applies, whether reads refresh lifetime and whether a requested TTL actually reaches the endpoint. A role visited late is not proof of a specific expiry threshold.

The hosted native common/rare phases MISSED their nominal 3/6-minute qualifications. Their false timing-qualified fields remain authoritative despite the observed rare write/common reads. At the late rare revisit, seven calls read their prefixes and only the rare role wrote its prefix again. Do not relabel the native trajectory as a controlled TTL pass.

A separate five-call Haiku/Anthropic default-TTL control met its measured schedule. Both prefixes initially wrote 8,059 tokens. A read the same prefix near three minutes and again near six minutes after a refresh; untouched B missed and rewrote its prefix near six minutes. Maximum schedule lateness was about 4.4 seconds. This supports independent prefix retention and refresh within the tested five-minute profile: touching common A did not retain idle B. It does not establish an exact expiry instant, another model/provider's TTL, or one-hour retention. All five raw responses validated against their actual sent schemas and matched native decoded values/source facts; this was a request-level control, not durable SQLite ticks. Its actual cost was about $0.0037. See [qualified evidence](qualified-evidence.md#dedicated-default-ttl-control) for exact timings and cost. Continue recording read AND write counters together; a response can do both.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall