Ursa: Iceberg-First Storage Engine for Kafka

topicpartition.io's April 2026 post covers Ursa, a storage engine StreamNative bolted onto a Kafka 4.2 fork that replaces local disk with direct object-storage writes in Iceberg or Delta Lake format. The engine won VLDB 2025's best industry paper. StreamNative has promised to open-source it twice and hasn't yet.

What Ursa changes

Standard Kafka stores data in local log segments on each broker's disk. Replication copies those segments across brokers (typically three copies across availability zones). This is the dominant cost driver — cross-AZ network traffic is roughly 90% of Kafka operational expense, and the triple EBS storage bill compounds it. Every partition has a leader that owns the writes, which creates hot spots, makes rebalancing expensive, and means brokers are stateful — you can't just add or remove one without shuffling data.

Ursa makes topics "diskless." Writes go directly to object storage (S3, GCS, or equivalent) instead of local EBS. No cross-AZ replication because object storage handles durability natively. No ISR (in-sync replica) management because there are no replicas to keep in sync. Brokers for diskless topics are stateless — add one, remove one, no rebalancing. StreamNative claims 10x lower infrastructure costs.

The trade-off is latency. Object storage writes are slower than local disk writes. For high-throughput, latency-tolerant workloads (analytics pipelines, log aggregation, CDC streams), this is fine. For low-latency event processing, it's not. Ursa handles this by making topic type configurable: a single cluster can run both diskless topics (cheap, higher latency) and regular disk-backed topics (expensive, lower latency). You pick per topic.

The lakehouse integration

The more interesting piece is that Ursa doesn't just dump bytes to S3 — it writes Iceberg or Delta Lake tables with proper catalog registration as part of the core write path. This means a Kafka topic is an Iceberg table. No Kafka Connect sink reading from the topic and writing to a lakehouse. No connector lag. No double storage. No schema divergence between the streaming layer and the analytical layer.

Auto-DLQ (dead letter queue) for bad schemas is a small but load-bearing feature. In a connector-based pipeline, a malformed message can halt the entire sink. With Ursa, bad messages get routed to a DLQ automatically, and the lakehouse pipeline continues. The error handling is in the write path rather than in a separate connector's retry logic.

Only one other vendor — Bufstream — offers this kind of first-class lakehouse integration in the critical write path. Everyone else (Confluent included) bolts it on via connectors, which have different SLA guarantees. A connector can lag, lose messages on restart, or silently drop records that fail schema evolution. A native write path either succeeds or fails the produce request.

Kafka protocol as de facto standard

The post's meta-argument is that the Kafka wire protocol has won while the Apache Kafka codebase is losing relevance. Over 20 vendors now compete by implementing the Kafka protocol — WarpStream, Bufstream, Redpanda, AutoMQ, and others. The protocol has network effects: when a critical mass of clients speaks the wire format, it's cheaper for every new vendor to implement that format than to invent a new one.

IBM, Confluent, and Red Hat collectively control about 80% of open-source Apache Kafka activity including the Project Management Committee. The open-source repo's gravitational pull comes from governance rather than technical lock-in. StreamNative, coming from the Pulsar world, chose to fork Kafka rather than try to get Ursa merged upstream — which says something about their confidence in the open-source process versus the protocol's staying power.

Adaptable topics

The ability to mix topic types in one cluster (diskless vs. disk-backed) is the operational feature that matters most for adoption. Most Kafka deployments have a mix of workloads: some topics need sub-10ms latency, others just need cheap high-throughput. Running separate clusters for each is operationally expensive and fragments monitoring, ACLs, and schema registries.

Three systems support this today: Aiven Inkless, StreamNative Ursa, and Redpanda 26.1. Apache Kafka's own KIP-1150 pursues the same goal but is realistically 1–2 years from landing.

Connections

Kafka's existing storage is log-structured in the same family as lsm-tree — append-only segments with periodic compaction. Ursa replaces the segment storage with object-store writes but keeps the log abstraction. ingodb is another storage engine rethinking write paths, though for a different problem (document store with reactive indexing).

StreamNative's pedigree is relevant: they built Apache Pulsar (which pioneered compute-storage separation in streaming) and Oxia (a ZooKeeper/etcd replacement now in CNCF incubator). Ursa extends the compute-storage separation idea from Pulsar into the Kafka protocol ecosystem.