zerofs

ZeroFS is a log-structured filesystem that uses an S3-compatible bucket as its only durable storage. It serves that storage as a POSIX filesystem over NFS and 9P, or as a raw block device over NBD, so ordinary applications get a normal mount while the bytes live in object storage. Unlike Amazon's S3 Files, ZeroFS does not keep one file per S3 object — the bucket is an internal persistence layer, not a set of readable objects. See zerofs-vs-s3-files for the head-to-head with S3 Files.

How it works

Filesystem metadata lives in an lsm-tree. File contents are split into extents, compressed, and encrypted, then packed into immutable segment objects. Metadata and file data take separate paths and only converge at the object store, landing as metadata SSTs plus segment objects. Because small files are packed together into segments, they don't each cost one object and one PUT — a real problem for object stores when you have millions of tiny files. The bucket and the raw local cache are ciphertext, so mounting or recovering requires ZeroFS and the encryption password; only some structural metadata stays visible.

A local RAM-and-disk cache sits in front of the object store. It fills from reads and from newly sealed segments, so read-after-write doesn't need a GET. The cache is not write-back — writes still reach object storage when segments are sealed and metadata is flushed. Cold misses resolve extents through the LSM tree and coalesce adjacent frames into ranged GETs, with prefetch within and across segments for sequential traffic.

Durability is explicit. Over 9P a successful fsync returns only after object storage acknowledges the data segment and its LSM metadata, which is strong enough for sqlite (plain NFS is not). The design is copy-on-write internally, with immutable segments and checkpoints. It does not currently write a WAL or journal for un-flushed data, so anything not yet flushed can be lost on a crash. It's single-writer with no horizontal write scaling, and there's no deduplication or reflinks (left out deliberately to preserve locality).

Usage

Point it at a bucket and mount:

# start ZeroFS against an S3-compatible bucket
ZEROFS_ENCRYPTION_PASSWORD=... zerofs s3://my-bucket/prefix

# then mount over NFS or 9P from the client side
mount -t nfs -o ... localhost:/ /mnt/zerofs

It works against Amazon S3, S3-compatible stores (Garage, MinIO, and similar), Azure Blob, and Google Cloud Storage. Conditional PUT support uses Redis/Valkey for coordination; the default configuration is fine. Inotify works for changes made through the local mount but not for changes made by another client.

Testing and maturity

For a young storage project, the test story is unusually developed. A deterministic simulation suite injects storage faults and crashes at arbitrary points, then checks the recovered data against reference models, running hourly with fresh seeds. CI additionally runs pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen crash/failover tests.

Limitations

No WAL for un-flushed data, single-writer only, no dedup or reflinks, and NFSv4 is not planned (the author prefers the 9P extensions). It's a single-author project and still early, so it's on the watchlist — worth re-checking for a WAL, multi-writer support, and release cadence before depending on it.

Repo: github.com/Barre/ZeroFS — ~2,880 stars, AGPL-3.0, Rust.