Memoryfields — agent memory as a file format

Cal Paterson's August 2026 essay proposes memoryfields, a portable file format for agent memory, and argues that the reason most agent-memory systems disappoint is that they treat memory as a process when it is better represented as data agent-memory-as-a-file-format. Paterson is the author of the format and ships tooling for it, so the piece is both an argument and a launch; he asks for review of the spec.

The case against the existing systems

He sorts current memory systems into three kinds that each fail their own way. Harness-locked systems, usually written by the lab renting you the harness, mine your conversation history and end up storing memories that are mostly about you, when information about the world is more useful. Over-built systems demand a lot of machinery — he cites one that needs pgvector, a Neo4j graph database and an LLM of its own just to decide what is worth remembering — which is hard to run and, he claims, actually confuses the models. And "High Modernist" systems impose an idealised rational structure, usually a graph and sometimes logical propositions, which strips facts out of their context and leaves the agent a list of distilled facts with the sense removed. What they share is treating memory as a multi-stage pipeline. He quotes Brooks on tables versus flowcharts to make the point that the data should be primary.

The format

A memoryfield is a directory (canonically archived as a zip) of markdown pages, each with optional YAML frontmatter, plus one optional SQLite file holding a vector index for semantic search. That is the whole thing. The design rests on four decisions:

Use prose, not chunks or extracted facts. RAG pipelines are complicated because they are trying to make existing human documents (big PDFs and the like) legible to an agent. A memory is different: at the moment it forms it is occurring to an agent that can already write prose, so it should write the memory directly in markdown, with no chunking, enrichment or double-summarising. The one constraint is that a page must fit in a vector embedding, a soft limit around 8 KB or roughly 1,300 words, which he argues is a healthy limit anyway; to add detail you add pages.

Semantic jump, not graph walking. He names Karpathy wikis as the key prior art and then rejects their central mechanism. Having an agent walk a hyperlinked knowledge graph is slow, because reaching information N links deep costs N+1 serial tool calls each pausing a few seconds, and it is unreliable, because the agent judges relevance only from link text or page titles, which pressures every page toward SEO-style snappy metadata and punishes the digressions and incidental lore that make a corpus valuable. Semantic search over page content replaces the walk: one call to search, one call to read all the hits in parallel, at most two tool calls regardless of depth.

More model, less mechanism. High-mechanism systems make the agent navigate an interface maze — a big API loads a lot of openapi.json into context, a small one is limiting, and someone else's API rarely fits your need. Being just a file format, memoryfields lets agents invent their own access patterns; he has watched agents use perl for corpus-wide find-and-replace and put inline CSV inside a memory to query it later with SQLite. Being low-mechanism also means the format scales with the model frontier: models are accidentally very good at bash, markdown and SQLite because those are in their training data, so they "get" a memoryfield in a way a disembodied call inside a memory pipeline never can, and they write better memories as they improve.

Open, interchangeable, transport-invariant. As a memory collection grows it becomes precious and you do not want lock-in, so there is an RFC-style spec and the format is deliberately open about where it lives — local files, S3, GitHub, HTTP; he personally uses Syncthing for private fields and S3 for shared ones. The static zip form also lets you review a field from someone else and pin it by sha256sum.

The objections he answers

Is it just RAG? In the loose modern sense, yes, but none of the usual RAG machinery is present: no chunking, no re-ranking, no hybrid search, and unlike most RAG it is written to as well as read. On embedding age, he defends nomic-embed-text-v1.5 as a small (270 MB), CPU-runnable, widely used default, while the spec allows other embeddings. On memory quality he argues the common fear of "filling memory with crap" mostly does not apply, because irrelevant pages are simply never surfaced by semantic search; his one real tip is to include citations, ideally URLs, so later passes can strengthen or fact-check a memory. On security he is blunt: you must not share your context window, including via memories, with parties you don't trust, which is why the static reviewable zip exists, and there is still no way for an agent to tell a good prompt from an evil one.

How it sits in the vault

This vault is itself a Karpathy-style wiki, and memoryfields is a direct critique of the mechanism Karpathy wikis run on — graph walking — so the disagreement is worth holding rather than smoothing. In practice the vault already sidesteps his objection: its warren tool provides semantic and keyword search over the pages (hybrid-search, agentic-search), so an agent here jumps to relevant pages by content rather than walking wikilinks, which is exactly Paterson's design decision 2. The page-sized-unit and write-in-prose arguments match llm-wiki-as-agent-memory, which makes the same case for markdown over a vector store on grounds of auditability and portability. Read through agent-memory-components, memoryfields is a specific answer to the store question that pushes work onto the extractor: a page-shaped memory needs the agent to write real prose.

Where it differs from the vault's practice is the graph. This wiki keeps wikilinks and catalogs deliberately, and treats orphans and cross-references as health signals; memoryfields drops links as a navigation mechanism entirely and relies on the embedding index. His own caveat that irrelevant memories cost only space, never quality, is the opposite of this vault's memory-conflict-detection discipline, where contradictions between pages are surfaced and resolved rather than left for the search to ignore.

proposition-identity-memory-tool is the useful contrast case. Paterson's whole thesis is that memory should carry no reasoning machinery, that "the less fixed machinery we put between the agent and that data, the better"; that essay is a first-hand account of someone trying to build the machinery — automatic contradiction detection over stored memories — and hitting a 130-year-old open problem in the process. The two agree that memory is data and disagree on how much the system around it should try to reason: Paterson says almost nothing, and the abyss essay is what happens when you say a lot. memorizing-session-transcripts is the third position, a negative result arguing that for coding agents with good commit and doc hygiene the whole retrieve-old-context move nets to zero regardless of format.