The Anatomy of Agent Memory

brgsk's essay (brgsk-blog) takes apart the word "memory" as agent-memory libraries use it. The libraries borrow vocabulary from cognitive science — episodic, semantic, procedural, working — but mostly don't build the machinery those words name. What they ship is one thing: a store of abstracted facts about past sessions, which is autobiographical semantic memory. The essay's point is not that this is fraud, it's that naming the real problem sharpens what you're trying to solve. The quote that carries the argument: "a store that can't answer what did I believe last month? isn't a memory system. it's a snapshot with a timestamp on it."

Three components

Every memory system decomposes into three parts, and each has one hard decision.

The extractor reads raw transcripts and emits abstracted "statements" — the things worth keeping. The design choice is timing: eager extraction runs per message (fresh, but pays LLM cost on every turn and can't see how a thread resolves), lazy extraction runs at end of session (cheaper, sees the whole arc, but exposed to lost-in-the-middle degradation when the transcript is long). Either way the extractor is an LLM summarizing, with all the failure modes that implies.

The store holds the statements — a vector index, a relational table, a knowledge graph, or some mix. The hard problem here isn't storage, it's contradiction. When a new statement conflicts with an old one, do you overwrite, append, or supersede (keep both, mark the old one stale with a pointer to the new)? Overwriting loses the history that would answer "what did I believe last month." Appending keeps everything but poisons retrieval with stale facts unless the retriever is told which is current. This is the memory-conflict-detection problem, and it's where most systems are weakest.

The retriever is RAG over the accumulated statements: vector similarity plus keyword match plus a reranker, ideally with time filters and presupposition checks (does the query assume a fact the store contradicts?). Retrieval quality is the thing that actually degrades as the store grows — not the storage, the finding.

The four canonical kinds, and which ones are missing

Cognitive science names four memory types. Working memory is the context window — brgsk sets it aside as already solved by definition. That leaves episodic (what happened, with time and place), semantic (facts stripped of their origin), and procedural (how to do things, skills). There's also a non-canonical fifth, prospective: remembering to do something later.

Three of the four are largely absent in production:

• Episodic gets compressed into semantic at extraction time. The extractor throws away the "when and in what session" and keeps the fact. So the episode is gone the moment it's stored.
• Procedural is mostly mislabeled semantic. The litmus test is whether a library actually evolves how the agent acts, not just what it knows. LangMem does it distinctly: score past trajectories, rewrite the system prompt from what worked. Mem0 only differs by tagging a statement metadata.memory_type="procedural" on the same vector index — a label, not a mechanism. Graphiti has no procedural concept at all.
• Prospective ("do Y the next time condition X appears") is unsolved by every production library. Schedulers handle "do Y at time T," which is the easy case; the conditional trigger is the hard one and nobody ships it.

What's left after you subtract the missing three is semantic memory about the agent's own past. Autobiographical semantic memory. That's the honest name for what these tools are.

Biology as vocabulary, not blueprint

The cognitive-science framing is useful for naming parts and dangerous as a design guide. Some biological ideas are worth importing: consolidation — compressing many episodes into stable patterns during downtime — shows up in Anthropic's "Dreams" work and Letta's sleep-time compute, and it maps cleanly onto agents. See agent-memory-decay for the decay-and-consolidate version. Others don't transfer. Emotional salience weights human memory, and a text-only agent has no emotions to weight with. And biological forgetting is something agents should not imitate: disk is cheap, keeping everything enables auditability, and the actual problem people blame on "not forgetting" — retrieval getting worse as the store grows — is a retrieval-and-adjudication problem, not a storage-pruning one. This is the same argument as ai-cannot-forget-forgive: machine memory has no native decay, and that's a feature to manage, not a bug to paper over with deletion.

References the essay leans on

Tulving (episodic vs semantic), Baddeley & Hitch (working memory), Joon Sung Park's Generative Agents, and practitioners Sarah Wooders (Letta) and Harrison Chase (LangChain).

Related

• agent-memory-components — the extractor/store/retriever decomposition as its own concept page
• agent-memory-decay — the consolidation-and-decay design brgsk partly endorses
• agent-memory-strategy-decision-tree — which layer each category of information belongs in
• memory-conflict-detection — the store's contradiction-handling problem in depth
• ai-cannot-forget-forgive — why forgetting is the wrong lever
• hippo-memory — a concrete library that does decay and consolidation