Skip to content

JPeetz/agent-skills / skills/ai-agent-engineering/agent-memory-management

agent-memory-management

Verified

Use this skill when designing, implementing, or debugging AI agent memory systems — long-term memory, working memory, episodic memory, vector stores, summarization strategies, state persistence, context window optimization. Activates on "agent memory", "long-term memory", "working memory", "memory architecture", "state management", "context window", "vector store", "memory retrieval", "RAG", "summarization", "memory persistence", "conversation memory", "episodic memory", or "memory optimization".

Agent Memory Management

Overview

An agent without memory is an expensive stateless function: every session it relearns the user, re-reads the same documents, and forgets the decision it made four turns ago. An agent with bad memory is worse — it confidently recalls stale facts, injects chunks that contradict the conversation, and burns 60% of its window on a transcript nobody reads. Memory is the discipline of deciding what to keep, where to put it, when to fetch it back, and what to throw away.

Three constraints shape every design decision:

  • The context window is finite and priced per token. More context is not better context: attention degrades with length (lost-in-the-middle), cost scales linearly, and latency grows with prefill. Every retrieved token should earn its place.
  • Memory has a lifecycle. Write → store → index → retrieve → rank → inject → decay → prune. Most broken systems skip the last three steps and become a landfill with a search box.
  • Persistence is a correctness problem, not a convenience. If an agent crashes mid-workflow, what it remembers decides whether it resumes or repeats a side effect (a sent email, a charged card).

The governing rule: memory is a retrieval problem, not a storage problem. You will rarely fail because you stored too little; you fail because the right thing was not in context when the model needed it. Design for recall at the moment of use.

When to Use

  • Designing a new agent and deciding whether it needs memory at all, and which kind
  • An agent forgets preferences between sessions, or contradicts a decision from earlier in the same session
  • The context window overflows on long conversations, documents, or tool traces
  • Retrieval returns plausible-but-irrelevant chunks and the agent answers from stale context
  • Building RAG over a corpus and choosing chunking, embeddings, index type, and reranking
  • A workflow must survive a crash, restart, or human-in-the-loop pause and resume idempotently
  • Memory costs are climbing and you need to cut them without losing quality
  • Multiple agents or sessions must share state across a store, bus, or blackboard

When not to use: a single-turn, stateless transformation (classify this text, extract these fields) needs no memory layer — adding one is pure overhead. Prompt improvements without state belong to prompt engineering. Measuring whether memory helped belongs to agentic-eval; this skill builds the system, that one measures it.

Memory Types

Four kinds of memory, distinguished by lifetime and what they store. Most agents need two or three, not all four.

TypeStoresLifetimeTypical mediumUse when
WorkingCurrent task state, scratchpad, recent turns, active planSeconds–minutes, one taskContext window, in-process dictAlways. This is the only non-optional one
EpisodicSpecific past events: "on Tuesday the user asked X and we did Y"Days–monthsEvent log, append-only table, vector store with timestampsPersonalization, debugging, learning from past runs
SemanticFacts and knowledge independent of when learned: "the refund policy is 30 days"Months–years, versionedVector DB, knowledge graph, document storeRAG over a corpus, entity facts, domain knowledge
ProceduralHow to do things: playbooks, tool-use patterns, learned routinesLong, revised slowlyPrompt library, skill files, fine-tune, codeRepeated multi-step tasks; the agent should get better at a task it has done before

Working memory

The agent's desk: current goal, plan, last N turns, intermediate tool results. Scoped to one task and discarded when it ends, except the small piece you deliberately promote. It is the only memory you cannot avoid, and it is where context budgeting happens. Design it explicitly — a fixed structure (goal, plan, facts, open questions, recent turns) beats an ever-growing message list, because a structured scratchpad can be rewritten in place while a message list can only grow.

Episodic memory

A timestamped log of what happened, used for two things people conflate: personalization ("last time you asked for metric units", "we already tried that and it failed") and debugging/audit — replay what the agent knew when it acted. It must be timestamped and never silently rewritten; an event log you mutate is not an event log. When a fact changes, append the correction with a new timestamp and let recency ranking resolve the conflict.

Semantic memory

Facts, not events. "The contract renews in March" is episodic (dated); "our SLA is 99.9%" is semantic (true regardless of when learned). Normally a vector index or knowledge graph over documents — this is where RAG lives. The critical property is versioning: a store that overwrites old policy text makes it impossible to explain why the agent cited last quarter's rules. Keep effective dates and supersede rather than delete.

Procedural memory

How-to knowledge: the steps for a recurring task, the tool that actually works, the query that returns the right rows. Declarative (playbooks and skill files loaded when the task appears) is cheap, inspectable, and editable — prefer it. Learned (patterns distilled from past trajectories into a prefix, examples, or weights) is powerful but can encode a bad habit as policy; gate it behind review.

Memory Architectures

Pick the smallest architecture that satisfies the requirement; memory-layer complexity is paid for forever in debugging.

Single-store

One store, usually a vector DB or a message table, serving everything.

agent ──write/read──▶ [ single store ]

Good for prototypes, single-user assistants, short-lived agents, RAG-only systems. Fails when you need "recent turn beats old document" or "user preference beats corpus fact" — one similarity score cannot express priority across memory classes.

Dual-store (working + long-term)

The default professional choice. Working memory lives in the context window as a structured scratchpad; long-term memory is external and retrieved on demand.

        ┌────────────────────┐
 turns ─▶│ working memory     │◀── retrieve ──┐
        │ (context window)   │               │
        └─────────┬──────────┘               │
                  │ compress / promote       │
                  ▼                          │
        ┌────────────────────┐      ┌────────┴─────────┐
        │ summarizer         │─────▶│ long-term store  │
        └────────────────────┘ write│ (vector + KV)    │
                                     └──────────────────┘

Good for most production assistants, coding agents, and support agents. The design point is the boundary: what gets promoted on task completion (durable facts, preferences, outcomes) versus discarded (tool chatter, retries).

Hierarchical

Tiers with different latencies and costs, promoted and demoted by importance.

L0  context window      ~10k tokens   nanoseconds   free-ish
L1  session cache       ~1M tokens    ms            cheap
L2  vector store        ~100M chunks  ~50ms         per-query cost
L3  cold archive / S3   unbounded     seconds       cheapest

Good for long-running assistants with large corpora and multi-session products. The design point is the promotion/demotion policy — without one, everything ends up in L0 and you have reinvented "stuff it all in the prompt" with extra services.

Distributed / agentic

Multiple agents share memory through a blackboard, event bus, or shared store.

  agent A ──┐                  ┌──▶ [ shared store ] ◀── agent C
            ├─▶ event bus ─────┤
  agent B ──┘                  └──▶ [ blackboard ]

Good for multi-agent systems, long-running pipelines, and human-in-the-loop. Fails when concurrent writers have no ownership rule: two agents writing conflicting "current plan" entries produce a system that argues with itself. Assign ownership per key, or make writes append-only with a single reducer.

See references/memory-architectures.md for the full comparison with cost, latency, accuracy, and operational-complexity tradeoffs, plus patterns by agent type.

Context Window Management

The context window is the scarcest resource in the system. Manage it like a budget with line items, not a bucket you pour into.

Budget the window explicitly

Allocate before you fill. A workable default:

SegmentShareNotes
System prompt + tool schemas10–15%Fixed; trim tool descriptions before anything else
Retrieved memory / documents25–35%The part that actually makes the answer good
Conversation / working state20–30%Recent turns + structured scratchpad
Output reserve15–25%Never let generation get truncated mid-JSON
Safety margin5%Tokenizer drift, tool results larger than expected

If the sum exceeds 100%, cut retrieved memory first (rerank harder), then conversation (summarize) — never the output reserve.

Strategies

Sliding window. Keep the last N turns verbatim, drop the rest. Cheap, and it loses the beginning of the conversation — exactly where the user stated the requirement. Use it for chit-chat, never for task work; pin the first message and the system prompt outside the window.

Summarization. Replace old turns with a rolling summary at a fraction of the tokens. Lossy in precisely the details that matter (numbers, names, constraints), so keep a separate structured facts block that is never summarized — only appended to and deduplicated.

Selective retrieval. Keep no history in context; store every turn and retrieve on relevance to the current query. Scales to unbounded history and is the right default for long-lived assistants. It can surface a stale turn that contradicts the live conversation, so always blend in the last 2–4 turns unconditionally.

Priority-based retention. Score each candidate and keep the top of the budget: score = w1·relevance + w2·recency + w3·importance + w4·pinned. Pinning is the escape hatch — user-stated constraints ("never use my full name") are pinned and never evicted, whatever the score says.

Compaction order

When you must shrink, compact in this order — cheapest and least damaging first:

  1. Tool output → digest. Replace raw JSON/HTML results with a 3-line summary plus a pointer to the full artifact on disk.
  2. Old turns → summary. Keep the structured facts block intact.
  3. Retrieved chunks → fewer, better. Rerank and cut the tail; never truncate a chunk mid-sentence.
  4. Duplicate elimination. The same document retrieved twice, the system prompt restated in a tool result, the plan quoted in three places.
  5. Only then drop low-priority history.

Never compact the system prompt, the original task statement, pinned constraints, or the output reserve.

Vector Store Integration

Embeddings

  • Match the model to the task. General-purpose models are fine for prose; code, tables, and multilingual corpora need models trained for them.
  • Never mix embedding models in one index. Vectors from different models are not comparable. Changing the model means re-embedding the whole corpus — treat it as a migration, and store embedding_model and dim as metadata so a mismatch is detectable rather than silently wrong.
  • Normalize if the metric expects it. Cosine similarity on unnormalized vectors makes the ranking partly a length artifact.
  • Query and document asymmetry. Many models want different prefixes for queries versus passages; getting this wrong costs 5–15% recall and is invisible in testing.

Chunking

Chunking is the highest-leverage decision in RAG, and the least glamorous.

StrategyWhenWatch out
Fixed-size + overlapBaseline, unstructured proseSplits sentences and tables mid-row
Recursive / structure-awareMarkdown, HTML, codeNeeds a sane separator hierarchy
Semantic (sentence-group)Dense prose, long argumentsSlower to build, marginal gain over good recursive
Parent-child / small-to-bigNeed precision retrieval + broad contextRetrieve small chunk, inject its parent
Per-recordTabular data, tickets, API objectsOne row per unit beats any splitter

Practical defaults: 300–800 tokens per chunk with 10–20% overlap for prose; keep headings with their content; never split a table; prepend the document title and section path so an isolated chunk still says what it is about.

Similarity search

  • HNSW for low-latency approximate search at scale. Tune M and ef_construction for recall vs memory; ef_search for recall vs latency.
  • IVF-PQ when the index must fit in RAM and some recall loss is acceptable. Needs a training pass and periodic retraining as the corpus drifts.
  • Flat / exact under ~100k vectors. Fast enough, and it removes a whole class of "why did it miss that" bugs. Do not build an ANN index you do not need.
  • Measure recall against exact search before shipping an ANN index; 70% recall@10 will feel randomly bad and you will blame the model.

Hybrid search

Dense vectors miss exact identifiers (SKUs, error codes, function names); BM25 misses paraphrase. Combine them.

query ──┬─▶ dense  top-50 ──┐
        └─▶ sparse top-50 ──┴─▶ RRF fuse ─▶ cross-encoder rerank ─▶ top-5

Reciprocal Rank Fusion needs no score calibration between two incomparable scales: score = Σ 1/(k + rank_i) with k ≈ 60. Then rerank the fused list with a cross-encoder — usually the biggest quality win per unit of effort after chunking.

Metadata filtering

Filter before or during vector search, never after. Post-filtering top-k means a restrictive filter can silently return 2 results.

Always store and filter on tenant_id (mandatory in multi-tenant — a missing tenant filter is a data-leak bug, not a relevance bug), source, doc_type, effective_date, version, author, acl. Enforce scope in the query builder: a model must never be able to widen its own reach.

Memory Persistence Patterns

PatternLatencyDurabilityGood forLimits
In-process dictnsnoneWorking memory, cachesDies with the process
File / JSONLmsstrong per-writeSingle-node agents, dev, small stateNo concurrency, no query, grows unbounded
Relational DBmsstrong, transactionalSessions, workflow state, auditSchema migrations; not a similarity index
Vector DB~10–100msstrongSemantic/episodic retrieval at scaleNot for exact state; eventual consistency
Distributed cache (Redis etc.)sub-msconfigurableSession cache, locks, rate stateEviction can silently drop state; not a source of truth
Object store10–100msstrongArtifacts, transcripts, embeddings backupNot queryable without a catalog

Rules that prevent most persistence bugs: one source of truth per fact (choose the system of record and treat every other copy as a cache — two writable stores always diverge); append, don't overwrite, for history (transcripts and event logs are append-only; current state is a materialized view derived from them); write-ahead the decision, not just the result (log intent before the side effect, so a crash is recoverable and deduplicable); persist the plan, not only the messages (resuming from a transcript requires re-deriving intent; resuming from a stored plan is deterministic); and namespace by agent, session, and tenant (retro-fitting tenancy into an existing store is a rewrite).

Memory Retrieval Strategies

Retrieval is where memory systems are won or lost. Measure it — recall@k, MRR, nDCG on a labeled set — rather than eyeballing answers.

StrategyHowBest forFails when
Last-NMost recent N turnsLive conversation continuityLong-range dependencies
SemanticEmbedding similarity to the current queryLarge corpora, paraphraseExact IDs, rare terms, stale-vs-live conflicts
HybridDense + sparse fused, then rerankedDefault for RAGOverkill for tiny corpora; needs tuning
Importance-weightedScore by salience (pinned, user-stated, decision)Long-lived assistantsImportance is hard to estimate at write time
Recency-weightedExponential decay on agePreferences that change, evolving factsStable facts get needlessly deprioritized
Graph traversalFollow entity relations from a seedMulti-hop questions ("who approves X for Y?")Costly to build and maintain

Bake the blend into one function so it is tunable and testable:

def rank(candidates, query_vec, now, budget_tokens):
    ranked = []
    for c in candidates:
        rel = cosine(query_vec, c.vec)                     # 0..1
        rec = math.exp(-DECAY * age_days(c.ts, now))       # 0..1
        imp = c.importance                                 # 0..1, set at write
        score = 0.55*rel + 0.25*rec + 0.15*imp + (1.0 if c.pinned else 0.0)
        ranked.append((score, c))
    ranked.sort(key=lambda t: t[0], reverse=True)
    return fill_budget(ranked, budget_tokens)              # dedupe by doc_id first

Two non-negotiables: deduplicate by source id before filling the budget, and always include the pinned set and the last 2 turns outside the ranking, so continuity survives a bad similarity result. Retrieve for the upcoming step, not the whole task — an agent about to edit a file needs that file, not the architecture doc. Query rewriting (2–3 retrieval queries) reliably beats a single raw query.

Memory Consolidation

Consolidation is the offline pass that keeps a memory store from becoming a junkyard. Run it on a schedule or at session end, never on the request path.

  1. Cluster. Group near-duplicate memories by embedding similarity plus entity overlap. Ten turns saying "user prefers metric units" are one memory.
  2. Deduplicate. Merge each cluster into one canonical entry, keeping the earliest timestamp as first_seen, the latest as last_confirmed, and the count as a confidence signal.
  3. Resolve conflicts. When two entries disagree (old address vs new), prefer the newer, mark the older superseded_by, and keep both for audit — never delete the loser silently.
  4. Promote. Recurring episodic patterns become semantic facts ("this user always asks for SQL, not prose"); frequent procedures become declarative playbooks.
  5. Summarize. Roll old episodes into coarser summaries that keep pointers to the raw records, so detail stays recoverable.
  6. Prune. Delete by documented rule: below an importance threshold and older than N days; never retrieved and older than M days; tool chatter immediately. Pinned and legally-required records are exempt. Pruning is a policy, not a cleanup script.

Consolidation needs an idempotency key (content hash + scope) so a re-run does not duplicate entries, and it must be reversible — soft-delete with a grace period, because a pass that hard-deletes user preferences is a support incident.

Summarization Approaches

ApproachHowCostRisk
ExtractiveSelect and keep original sentencesLowReads choppy; misses synthesis
AbstractiveRewrite in fewer wordsMediumInvents or drops detail; unverifiable
RecursiveSummarize, then summarize the summariesLow per levelError compounds each level
HierarchicalSection summaries + document summaryMediumNeeds structure in the source
Map-reduceSummarize chunks in parallel, then mergeHigh but parallelLoses cross-chunk relationships

Rules that keep summaries trustworthy:

  • Keep numbers, names, dates, and constraints verbatim. A summary that rounds "€4,250" to "about €4k" destroyed the only part a downstream agent needed.
  • Require provenance. Each summary line points to a source id; anything unverifiable is dropped rather than smoothed.
  • Never summarize the constraints block. User-stated rules live in a separate, never-compressed structure.
  • Bound recursion to 2–3 levels. Beyond that, information loss dominates and you cannot trace the error.
  • Summarize for a purpose. "Summarize this conversation" produces mush; "extract decisions, open questions, and stated preferences" produces something usable.

For conversation memory, keep two structures: a rolling narrative summary and an append-only facts ledger of decisions, preferences, and identifiers. The ledger survives every compaction pass; the narrative does not.

State Management

Working memory is what the model sees; state is what the system knows. Conflating them is a common source of "it resumed but did the wrong thing".

Session state

Per-conversation: a durable row keyed by session_id holding the message list (or its summary), the facts ledger, and the plan. Load on start, write on every turn that changes a decision — never lose an irreversible action.

Workflow state and checkpoints

Per-task execution: which step, which inputs, which outputs, which side effects have fired. This is what makes resumption possible. Persist at step boundaries, before any irreversible action, and on a timer for long steps. A checkpoint must be self-sufficient — everything needed to resume without replaying history: the plan, the cursor, accumulated results, and completed effects.

{
  "run_id": "run_8f21",
  "step": "match_payments",
  "status": "in_progress",
  "attempts": {"match_payments": 2},
  "completed_effects": ["email_sent:msg_44", "ledger_posted:tx_91"],
  "cursor": {"invoice_id": "INV-1042"},
  "updated_at": "2026-11-16T10:22:31Z"
}

Idempotent resumption

  1. Every side effect gets an idempotency key derived from run_id + step + target — not a random UUID, which defeats the purpose on retry.
  2. Check completions before acting: if key in completed_effects: skip.
  3. Pass the key through where the provider supports idempotent calls, so a crash between "called" and "recorded" is harmless.
  4. Fence concurrent runs with a lease per run_id and a TTL, so a dead worker does not block forever.
  5. Resume from the cursor, not the top. Re-running a 40-step workflow from step 1 because step 27 failed is how agents double-charge customers.

Two patterns to name: the phantom checkpoint (state written before the effect, so resumption re-fires it) and the optimistic checkpoint (written after the effect but before it succeeded, so resumption skips work that never happened). Write after confirmation, and make the effect idempotent so the window between is harmless.

Anti-Patterns

#Anti-patternDescriptionImpactFix
1Stuff-everything contextConcatenate all history and documents into the promptCost and latency explode; attention degrades; model loses the actual askBudget the window; retrieve and rerank; compact old turns
2Summarize-only memoryReplace all history with a prose summary, no facts ledgerNumbers, names, and constraints quietly vanishKeep an append-only facts ledger that is never summarized
3Unscoped multi-tenant retrievalQuery the vector store without a tenant filterCross-customer data leak; worst possible failureEnforce tenant_id in the query builder, not the prompt
4Post-filtering top-kRetrieve top-10 then filter by metadataRestrictive filters return near-empty results silentlyFilter during search; assert result count
5Mixed embedding models in one indexAdd vectors from a new model without re-embeddingSimilarity scores become meaningless; recall collapsesVersion the model; full re-embed as a migration
6Unbounded memory growthAppend everything, never prune or consolidateStore cost grows forever, retrieval quality decays, duplicate memories crowd out good onesScheduled consolidation with documented prune policy
7Rewriting the event logUpdate past entries in place to "correct" themLoses audit trail and the ability to explain past actionsAppend corrections with timestamps; supersede, don't overwrite
8Memory written before the effectCheckpoint state, then perform the side effectCrash causes double-send/double-charge on resumeWrite after confirmation; make effects idempotent with derived keys
9Random idempotency keysGenerate a fresh UUID per attemptRetries look like new work; dedupe impossibleDerive the key from run+step+target
10Sliding window on task workDrop the oldest turns of a long taskThe original requirement and constraints disappear firstPin the first message and constraints; summarize middle turns
11Similarity score as priorityOne cosine score decides across memory classesStale document beats a live user correctionBlend relevance, recency, importance; pin hard constraints
12Retrieving for the whole taskOne retrieval pass at task startLater steps lack the context they need; early context is noiseRetrieve per step, with query rewriting
13Trusting a summary over the sourceAct on a summary as if it were the recordCompounded errors, unverifiable claimsKeep pointers to raw records; verify before irreversible actions
14Memory in the prompt onlyStore "state" as text the model must re-read and trustModel can forget, contradict, or be prompt-injected into ignoring itKeep state in code/DB; inject only what the step needs
15No retrieval evaluationShip RAG without a labeled query setEvery change is a guess; regressions invisibleBuild 50–200 labeled queries; track recall@k and nDCG
16Chatter stored as memoryPersist tool retries, stack traces, and greetingsNoise dominates the index; retrieval quality dropsFilter at write time; store digests with artifact pointers

Reference Architectures

PatternWorking memoryLong-termRetrievalWatch
Personal assistant (dual-store)Scratchpad + last 6 turnsVector store (episodic) + facts ledger in SQLHybrid, scoped to user_id, per turnPreference drift: supersede, never overwrite
Support / RAG (dual-store)Last 4 turns + current ticketVersioned policy index, date-filtered chunksHybrid + cross-encoder rerankAnswering from a superseded policy version
Coding agent (hierarchical)Current file + plan + failing testsL1 symbol index; L2 code/docs chunked by function with path attachedSymbol reference and stack-trace frames beat similarityIndex staleness on edited files
Long pipeline (workflow state)Current step inputsSQL workflow row + effect log + document indexPer step, not per taskDouble-fired side effects on resume
Multi-agent (blackboard)Role-specific projectionAppend-only event bus + materialized viewRead your projection onlyTwo writers on one key

Budget shape for the assistant case: 30% retrieval, 25% conversation, 18% output, 12% system, 4% pinned. For coding, swap retrieval for file contents and keep the output reserve.

Tooling

scripts/memory_architect.py turns a requirements profile into a recommended architecture with tradeoffs. Stdlib-only, read-only, no network.

python3 scripts/memory_architect.py --print-schema > requirements.json
python3 scripts/memory_architect.py --config requirements.json
python3 scripts/memory_architect.py --self-test

Verification Checklist

Before calling a memory system done:

  • Every memory class has a stated lifetime and a delete rule; the context budget is explicit and the output reserve is protected
  • Retrieval is scoped by tenant/user in code, with a result-count assertion
  • Pinned constraints and the original task survive every compaction pass; the facts ledger is append-only and separate from the prose summary
  • Embedding model and dimension are stored as metadata; a mismatch fails loudly
  • Concurrency: lease per session/run with TTL
  • Side effects use derived idempotency keys; resume skips completed effects
  • Consolidation is idempotent, reversible, and off the request path; retrieval has a labeled eval set with recall@k
  • Cost per session and p95 retrieval latency are measured, not assumed
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 319kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 197kUpdated Claude CodeCodexVerifiedInstall