Map

Ursa - a new Iceberg-first storage engine for Kafka

Sources ↳ show in map Markdown

Ursa - a new Iceberg-first storage engine for Kafka

Author: topicpartition.io (unattributed) Date: April 10, 2026 Source: https://topicpartition.io/blog/ursa-a-new-lakehouse-first-storage-engine-for-kafka Tags: kafka, iceberg, delta-lake

Summary

StreamNative (creators of Apache Pulsar, Oxia) forked Kafka 4.2 and added Ursa, an optional diskless storage engine that writes directly to object storage in Iceberg/Delta Lake format. Won VLDB 2025 best industry paper. Not yet open-sourced despite two public commitments.

Ursa's design

  • Diskless topics: no local EBS, no cross-AZ replication (which is ~90% of Kafka operational cost), no ISR management
  • Claims 10x lower infra costs
  • Instant elasticity: brokers are stateless for diskless topics, scale up/down without rebalancing
  • No under-replicated partitions, no hot spots, rapid broker startup/shutdown

Lakehouse-native integration

  • Topics store directly as Iceberg/Delta tables with catalog registration on the write path
  • Auto-DLQ (dead letter queue) for bad schemas — malformed messages don't halt lakehouse pipelines
  • Single data copy: no Kafka + separate lakehouse storage duplication
  • Unified governance across one system

Compatibility

  • Coexists with classic Kafka paths (KRaft log, local segments, Tiered Storage)
  • Topics switchable between diskless and regular via config
  • Rolling back to standard Kafka possible, but Ursa-backed topics would be lost

Competitive landscape

  • 20+ vendors now compete on Kafka protocol implementations
  • Only Bufstream and Ursa provide first-class lakehouse integration in the critical write path (vs bolted-on connectors)
  • Adaptable topics (mix diskless and regular in one cluster): Aiven Inkless, StreamNative Ursa, Redpanda 26.1
  • Apache Kafka's KIP-1150 is 1-2 years out

Meta-argument: protocol vs codebase

  • Kafka wire protocol has won as de facto standard through network effects
  • Apache Kafka open-source repo's gravitational pull is weakening
  • IBM, Confluent, Red Hat control ~80% of open-source Kafka activity including PMC
  • Vendors find it cheaper to implement the protocol than create new APIs

Open questions

  • Can the protocol maintain dominance without simplification?
  • Will users move to minimalistic alternatives?
  • When will StreamNative actually open-source Ursa?