Designing Microkernel IPC

Seiya Nuta is the author of Resea and the in-progress FTL operating system. This post is a working diary of the IPC redesign he did in FTL over a few weeks, written as a guided tour from the textbook starting point ("IPC is memcpy(2) across processes") through the concrete design decisions FTL ended up with. The framing is useful because every section pairs a textbook approach with the specific failure mode that pushed FTL toward a different one.

The four hops of the argument

1. Message passing is the microkernel substrate. Each message has a type, a payload, and some handles. Drivers, the TCP/IP stack, and the filesystem are user-mode processes that talk through mailboxes — ipc_send / ipc_receive, the same API on both ends. A file read becomes an RPC: client sends FILE_READ_MSG, server replies with the bytes.

2. Synchronous IPC is easier to debug, but it deadlocks server-to-server. Asynchronous IPC means the kernel queues messages, which forces dynamic allocation, exposes backpressure, and opens a DoS surface. Hubris and Resea both went synchronous for that reason. But synchronous IPC alone deadlocks the moment two services try to send to each other — the canonical example is a TCP/IP server and an Ethernet driver where each sends before the other is in receive state. Nuta's term for the workaround is the notify-and-pull pattern: the server fires a non-blocking notification (a flag in a mailbox, no payload, no count), the client reacts by pulling with a normal synchronous request. This is covered as a separate concept page: notify-and-pull-ipc.

3. IDLs are the convention; FTL skips them. Fuchsia has FIDL, others have similar generators. FTL replaces the entire IDL layer with five fixed operations — open, read, write, getattr, setattr — mapped directly into syscall arguments. No serialization. The send syscall takes two inlined args, an optional body buffer, and an optional handle, and the operation type tells the kernel what each slot means. Renaming a file becomes setattr; opening a TCP listener becomes open("tcp:0.0.0.0:1234", MODE_LISTEN). Nuta calls this an accidental reinvention of Plan 9, and points out that Go's os.Rename on Plan 9 is literally a file-attribute write. FTL drops the strict "everything is a file" rule — paths can be arbitrary handles, not necessarily filesystem-shaped.

4. Pull, not push; peek, then receive. A naive driver pushes packets to the TCP/IP server as they arrive — and now the driver owns the backpressure problem. FTL inverts it: the TCP/IP server sends read requests when ready, the driver replies with what's buffered. The message-queue length is bounded by in-flight requests, not by data volume. Nuta draws the parallel to io_uring, which is also pull-shaped and bounded by SQE count. The receive path is two-step: a channel_peek returns the message header and an opaque recv_token, then channel_receive(token, buf) copies the body straight into the caller's destination — no intermediate copy, and if the receiver isn't ready the message stays parked in the kernel queue, which propagates backpressure to the client without the kernel needing a separate flow-control protocol. In real code the peek call is folded into the event-stream API (analogous to epoll), so there's no extra syscall.

Why this matters in 2026

The post lands in a year where everyone is rediscovering message-passing isolation under the names "agent sandbox" and "microVM." See microvm-2026 and sandboxing-ai-agents — the substrate arguments are the same shape: you replace shared mutable state (kernel namespace, shared memory) with explicit message-passing across a hardware boundary. FTL is microkernel-domain rather than container-domain, but the pull-based and notify-and-pull patterns translate directly to any system where a producer and consumer cross a trust boundary.

The post also slots cleanly into the wider message-passing-vs-shared-memory argument. Lee's 2006 framing was that message passing inherits the same nondeterminism as shared state. FTL's design is mostly an attempt to eliminate the specific failure modes message passing inherits: deadlock (notify-and-pull), backpressure (pull-based), serialization overhead (no IDL), and double-copy (peek-then-receive). It doesn't eliminate the underlying nondeterminism — it just narrows the surface where it can hurt.

The closing note is that the whole design was shaped by wanting async Rust to feel natural, especially around cancellation safety. Nuta says a follow-up post on that integration is coming.

See also

• notify-and-pull-ipc — the synchronous-plus-notification pattern as a concept page
• message-passing-vs-shared-memory — broader concurrency-coordination framing
• message-passing-shared-mutable-state — Lee/Tu source page behind the above
• microvm-2026 — same substrate argument at the VM-isolation layer
• sandboxing-ai-agents — message-passing-as-isolation in the agent-sandbox space
• seiya-nuta-blog — the author's blog and project context (Resea, FTL)
• microkernel-ipc-design (source)