Mintlify's ChromaFs: A Virtual Filesystem for AI Assistants

Mintlify ran into a common problem with their documentation assistant: traditional RAG only returns chunks matching a query, so it fails when an answer spans multiple pages or when the exact syntax a user needs doesn't rank in the top-K results. Their first fix — cloning repos into sandboxes so the agent could browse real files — worked but was slow (~46 seconds P90 to create a session) and expensive ($70K+/year at 850K monthly conversations).

Their solution, ChromaFs, is a virtual filesystem that makes a Chroma vector database look like a UNIX directory tree. The agent issues normal shell commands (ls, cat, grep, find, cd) and ChromaFs translates each call into a database query. No actual files exist on disk.

How it works

ChromaFs builds on Vercel Labs' just-bash, a TypeScript reimplementation of common shell commands with a pluggable IFileSystem interface. just-bash handles parsing; ChromaFs handles storage.

Directory tree. All paths and their metadata (visibility, group permissions) are stored as gzipped JSON. On session start this decompresses into two in-memory structures: a path set and a directory-to-children map. Commands like ls, cd, and find resolve entirely in memory with no network calls.

Access control. Each path entry carries isPublic and groups fields. Before the agent sees the tree, ChromaFs filters it against the user's session token. Unauthorized paths are pruned completely — the agent can't reference files it shouldn't see. This replaces Linux-style chmod/group management with a single metadata check.

File content (cat). Documentation pages are stored as chunks in Chroma. When the agent cats a file, ChromaFs fetches all chunks for that page slug, sorts them by chunk_index, and concatenates them. Results are cached so repeated reads (common during grep workflows) don't hit the database again. Large OpenAPI specs sitting in customer S3 buckets use lazy file pointers — they appear in listings but only fetch on access.

Grep. Recursive grep uses a two-stage filter. First, Chroma's $contains (fixed strings) or $regex operators narrow the candidate set. Then matched chunks get prefetched into Redis, and just-bash runs the actual regex in memory against only those files. This keeps grep latency in the milliseconds even over large doc sets.

Writes. All write operations return EROFS (Read-Only File System). The filesystem is stateless and the agent can't modify anything.

Results

Session creation dropped from ~46 seconds to ~100 milliseconds. Marginal compute cost went to zero since it reuses existing database infrastructure. The system handles 30,000+ daily conversations across hundreds of thousands of users.

What makes this interesting

The core insight is that LLM agents don't need real filesystems — they need something that behaves like one. A virtual filesystem backed by a database gives you access control, caching, and search optimization that would be much harder to bolt onto actual file I/O. It also keeps the system stateless and read-only, which eliminates a whole class of agent-safety problems.