Mesh LLM: Distributed Inference over iroh

Mesh LLM pools the GPUs and memory across whatever machines you already have and exposes the lot as a single OpenAI-compatible endpoint at localhost:9337/v1. You start one node, add more when you want, and the mesh decides where each request runs. It sits in the same territory as titit-local-ai and lucumr-local-models — the argument that you should run models on hardware you control instead of renting a metered API — but it answers a different half of the problem. Those pieces are about one machine doing local inference; Mesh LLM is about making several machines act like one so you can run models no single box could hold.

A request gets served one of three ways. It runs locally on the current machine's GPU, routes to a peer that already has the model loaded, or splits a too-large model across several machines as a pipeline. The client never sees which path was taken. It just talks to localhost.

Splitting a model across nodes

The split mode is called Skippy. A model is partitioned by layer ranges into stages — layers 0–15 on one node, 16–31 on the next, and so on — and activations flow down the pipeline from stage to stage. Several modest machines run a model none of them could hold alone. Skippy is a patch queue on top of llama.cpp that reaches into the engine's internals to pull out activations and filter tensors at load time. Pre-computed splits for popular models are published to a HuggingFace org; a job dices models by layer and posts the pieces.

The reason this works at all is what crosses the network. Only small activation vectors move between stages — a few kilobytes per token, not the gigabytes of weights. The weights stay resident in each node's VRAM. That makes the setup latency-bound rather than throughput-bound, and it can beat streaming weights off local RAM or disk, where you re-read gigabytes per token from slow storage. The cost is one network round-trip per pipeline stage per token: split across N nodes, you pay (N-1) latencies for every token generated. On a fast LAN with sub-millisecond hops that ceiling is high; over the open internet it collapses, which is why token generation is meant for local or enterprise networks. Prefill is different — the per-token latency doesn't accumulate there, so distributed prefill wins even on slower links.

Reported numbers are modest but real. Qwen 235B-A22B is listed at 16 tok/s across two nodes; a contributor got ~10 tok/s on GLM 5.2 across two Mac Studios over plain 1GbE using a custom Q2 quantization. If one node drops mid-inference the topology is recomputed and the request retried on a new split.

Why iroh

Every node — server or client — boots an iroh endpoint, which is its identity (a public key) and its only network surface. There is no central server. iroh does the hole-punching, NAT traversal, and relay fallback to open a direct, authenticated QUIC connection between any two nodes wherever they sit, and Mesh LLM runs two iroh relays in separate regions as the fallback path. Because nodes are addressed by public key rather than IP, "route to a peer" and "stream activations to the next stage" become the same primitive as "talk to localhost," just with a different endpoint ID. This is the same key-addressed, location-independent idea behind yggdrasil-network, and the transport-name-not-a-server property that stateful-agent-routing identifies as the missing piece for addressing a specific durable process.

The protocol uses three QUIC ALPNs: mesh-llm/1 for the main mesh (gossip, routing, HTTP tunnels, plugin channels), mesh-llm-control/1 for the owner control plane, and skippy-stage/2 for the latency-sensitive activation transport. Inside the main connection everything is a bidirectional stream tagged with a single leading byte — gossip (0x01), HTTP tunnels (0x04), route queries (0x05), peer-down and peer-leaving events, plugin RPC, direct-path requests — all demuxed off that first byte. iroh supplies the secure transport; Mesh LLM builds its own gossip layer on top so it controls who is admitted, which versions are compatible, and which peers to trust. The runtime itself is plugin-based: plugins declare capabilities in a manifest and are exposed over MCP, HTTP, inference, and mesh events.

What's unsettled

Encryption is in-transit only. iroh's key-based dialing gives authenticated, encrypted QUIC between endpoints, so snoopers can't read the traffic, but it is not end-to-end or in-process — whoever runs the inference sees the plaintext prompt and response. On a mesh shared with friends or family that's awkward for anything personal (a medical question to a shared model, say), and the contributors were candid that privacy from participating nodes and defense against poisoned activations are both unsolved. Their answer for now is to build a private mesh of trusted peers. There is also no fairness or incentive mechanism for the public mesh yet; the team sees it as unnecessary while the real use case is private meshes you own. This is a different distributed-systems posture than log-distributed-llms, which treats multi-agent coordination as a consensus problem with Byzantine actors — Mesh LLM sidesteps Byzantine trust by assuming a mesh you control.

Related prior art raised in discussion: exo (Mesh LLM's docs have a comparison page), cocompute (another iroh-based effort), and Colibri. The client is about 18 MB; a mobile app on iroh's Swift SDK is planned, along with support for ACP so other agent clients can join.

Mesh LLM: Distributed Inference over iroh

Splitting a model across nodes

Why iroh

What's unsettled

See also