ZeroFS vs. Amazon S3 Files

Both Amazon S3 Files and ZeroFS mount a POSIX filesystem over an S3 bucket, so from a distance they look interchangeable. The zerofs.net blog post (written by ZeroFS's author) argues they make opposite promises about what the bucket contains. S3 Files keeps each file as an ordinary S3 object you can still read with GetObject. ZeroFS treats the bucket as an internal persistence layer whose contents are meaningless without ZeroFS and an encryption password. Everything else — cost, cold-read behavior, rename cost, when data becomes visible — follows from that one difference.

The two layouts

S3 Files is built on Amazon EFS. A path like images/cat.jpg on the mount corresponds to the key images/cat.jpg in the bucket, and changes propagate in both directions. Active data and metadata sit in a low-latency tier AWS calls "high-performance storage," and the mount speaks NFS 4.1/4.2.

ZeroFS breaks the one-file-one-object mapping on purpose. Filesystem metadata lives in an lsm-tree; file contents are split into extents, compressed, encrypted, and packed into immutable segment objects. Metadata and file data travel separate paths and only meet at the object store, where they land as metadata SSTs and segment objects. An S3 client staring at the bucket sees opaque segments, not files. ZeroFS serves that filesystem over 9P or NFS for files and NBD for raw block devices, and unlike S3 Files it isn't tied to Amazon — it runs on any S3-compatible store, Azure Blob, or Google Cloud Storage.

Where the designs diverge

Object interoperability is the headline. With S3 Files, a file written through the mount becomes a normal object after an asynchronous export that starts 60 seconds after the last write; continued writing postpones visibility, and if both the S3 side and the file side edit the same object before they sync, S3 wins and the filesystem copy goes to lost+found. ZeroFS never exposes mounted files through the S3 API at all — the tradeoff is that small files no longer cost one object and one PUT each, since their extents are packed together, and the whole bucket is ciphertext at rest.

Cold reads work differently. The first S3 Files access to a directory imports every object's metadata and asynchronously copies files under a threshold (128 KiB by default) into high-performance storage; a first listing of 1,000 objects can take several seconds, and reads of at least 1 MiB go straight to S3. ZeroFS resolves the requested extents through the LSM tree and coalesces adjacent frames into ranged GETs, paying one round trip without dragging in the rest of the file or the rest of the directory. Its local RAM-and-disk cache is populated from reads and from newly sealed segments, so read-after-write needs no GET. Once the working set fits locally, data reads generate few S3 requests.

fsync semantics differ in a way that matters for databases. On S3 Files, durability and S3 visibility are separate events: a write to high-performance storage is durable immediately, but fsync does not make the object visible through the S3 API — that waits for the 60-second export. ZeroFS has no second visibility event because there is no export; over 9P a successful fsync returns only after object storage acknowledges the data segment and its LSM metadata, so the file survives a cold restart. This is what a self-hoster in the discussion was after: fsync durability strong enough for sqlite, which rules out plain NFS.

Rename is cheap on one side and expensive on the other. S3 has no directories and no atomic rename for general-purpose buckets, so S3 Files renames on the mount and then copies each affected object to a new key and deletes the old one — AWS says synchronizing 100,000 renamed files takes a few minutes, and during a directory rename both prefixes can be visible. ZeroFS just rewrites its LSM directory entries: no per-descendant copy, nothing exposed to object clients.

Cost

Both pay S3 storage and request charges. S3 Files adds its high-performance storage tier on top: roughly $0.30 per GB-month resident, $0.03 per GB read, $0.06 per GB written, with a 10 KiB minimum billable file size and required bucket versioning. How much lands in that tier depends on the import threshold and expiration window, so bucket size alone can't forecast the bill. ZeroFS's cost is object storage plus requests plus the compute and cache running it — one server, or two for high availability, each needing about 2 GB of RAM beyond its configured cache. The post's own illustrative model (store 10,000 GiB, read once) puts ZeroFS around $115–230/month plus infrastructure and S3 Files anywhere from ~$231 to ~$3,530 depending on how much data stays resident and small; the author is careful to call it illustrative, not a general comparison, and notes S3 Files may win for large-file streaming.

How to choose

The post's own framing is the cleanest summary: pick S3 Files when files must stay ordinary S3 objects that other tools read directly, and pick ZeroFS when the bucket can be a private substrate you trade direct access for compression, packing, and client-side encryption. This is the same "object storage as a POSIX substrate" pattern that Ursa applies to Kafka logs — the durability and capacity of S3 underneath an interface that isn't S3. It's also the practical counterpart to the virtual filesystem idea: a real kernel-visible mount rather than an in-process VFS, but the same instinct of putting a filesystem API in front of storage that isn't a disk.

Interesting from the discussion

The author confirmed ZeroFS does not write a WAL or journal for un-flushed data, so anything not yet flushed can be lost on a crash — a commenter (the_duke) flagged this as a gap worth closing. It's single-writer, with no horizontal write scaling yet. There's no deduplication, whole-file or block-level, and no reflinks — left out deliberately because both hurt locality, even though ZeroFS is internally copy-on-write with immutable segments and checkpoints. On trust, the author points to a deterministic simulation suite that injects storage faults and crashes at arbitrary points and checks recovered data against reference models, running hourly with fresh seeds, plus pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen in CI. One request that didn't exist yet: DuckDB-style scanning of the filesystem as columnar data for AI/ML training, which ZeroFS's packed-segment layout doesn't currently expose.