Distributed Model Inference on a Mesh of iroh Endpoints

When people picture running a large language model, they picture a data center. Racks of GPUs that belong to someone else, a metered API, and a bill that grows every month you succeed. You send your prompts off to a black box and hope the price, the model, and the privacy policy all stay the way they were when you signed up.

For a lot of teams that is a bad trade. You give up control over when models change, where your data goes, and what hardware runs your workloads. And as usage grows, so does the bill, with no lever to pull except "pay more."

Mesh LLM is a different shape. It pools the GPUs and memory you already have, across as many machines as you want to add, and exposes the whole thing as one OpenAI-compatible API. Start one node. Add more later. Let the mesh decide whether a model runs on the box in front of you, routes to a peer, or splits across several machines.

The problem: AI is expensive, and it is somebody else's

The popular models are monoliths. Most people reach them through a UI or an API key and pay a large provider to run everything. That is convenient, and it is also a surrender. You do not control when the model gets updated, what memory it runs in, or what hardware sits underneath.

Plenty of businesses and services that depend on these models want the opposite: more control, more pluggability, lower cost. They have GPUs sitting in offices, in closets, under desks. What they are missing is a way to make those machines act like one.

Mesh LLM: run the models yourself

The pitch is simple. Run bigger models without buying bigger GPUs. Share compute privately with your team, or publicly with the world, to power agents and chat. Point any OpenAI client at http://localhost:9337/v1 and stop caring where the work actually happens.

Under the hood, Mesh LLM distributes model compute across a mesh of iroh endpoints. A request can be served three ways:

• Run it locally, on this machine's GPU.
• Route it to a peer that already has the model loaded.
• Split a model too big for any single box across several machines, as a pipeline.

How it works

The architecture is pluggable. Plugins declare what they provide in a manifest, the runtime starts them, routes calls, and exposes their capabilities over MCP, HTTP, inference, and mesh events. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants.

For the giants, Mesh LLM has a split mode (internally, "Skippy"). A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so several modest machines can run a model none of them could hold alone. The OpenAI client never sees any of this. It still just talks to localhost.

How it uses iroh

Every node, whether it serves models or only sends requests, boots an iroh endpoint. That endpoint is the node's identity, a public key, and its only network surface. There is no central server. iroh handles the hole-punching, NAT traversal, and relay fallback needed to open a direct, authenticated QUIC connection between any two nodes, wherever they sit.

To keep that working across the open internet, Mesh LLM runs two iroh relays in different regions, so nodes that cannot reach each other directly always have a fallback path nearby.

The whole protocol rides on QUIC's ALPN negotiation. There are three:

Inside the main mesh-llm/1 connection, everything is a bidirectional QUIC stream tagged with a single leading byte that says what kind of stream it is. One connection carries gossip, inference, route queries, and peer-lifecycle events, all demuxed by that first byte:

The neat part is what this buys you. iroh gives authenticated, NAT-traversing QUIC between any two machines, addressed by public key. So "route to a peer" and "stream activations to the next pipeline stage" become the same primitive as "talk to localhost," just with a different endpoint ID. The networking stops being something you have to think about.

iroh provides the secure transport. Mesh LLM builds its own gossip layer on top, so it controls exactly who gets admitted to the mesh, which versions are compatible, and which peers to trust.

Getting started

Users can install the lightweight software (about 18 MB) and either join the public mesh or configure private deployments. The system presents itself as localhost:9337/v1 to any standard OpenAI client.

A mobile app is coming, built on iroh's Swift SDK. The plan is to speak ACP, the emerging agent standard, so other clients can join the mesh too. The throughline is the same one that motivated the whole project: more peer to peer, fewer closed servers, and no lock-in.

Iroh is a dial-any-device networking library that just works. Compose from an ecosystem of ready-made protocols to get the features you need, or go fully custom on a clean abstraction over dumb pipes. Iroh is open source, and already running in production on hundreds of thousands of devices. To get started, take a look at the docs, the code, or the discord channel.

Notes from the Hacker News discussion (item 48876505)

Contributors i386, sig_kill, and michaelneale answered questions in the thread.

• Performance is latency-bound, not throughput-bound. Only small activation vectors (a few kilobytes) cross the network per token, not the weights, so raw bandwidth is not the bottleneck. Each pipeline stage adds one network round-trip per token, so splitting across N devices adds (N-1) latencies per token. A commenter worked out that ~10ms combined latency caps token generation near 33 tok/s in the ideal case; a fast 1GbE LAN with sub-millisecond hops is much higher. Over the open internet, latency makes token generation impractical, but prefill/prompt-processing still wins because there the latency does not accumulate per token.
• Real numbers. Qwen 235B-A22B is listed as "proven at 16 tok/s across 2 nodes." Contributor i386 reported ~10 tok/s on GLM 5.2 on a home lab of two Mac Studios (M3 Ultra 256GB + M1 Ultra 128GB) over 1GbE, using a custom Q2 quantization that keeps sensitive tensors at Q8, with 5ms simulated latency/jitter. The pitch is running a large model on several machines you already own, without RDMA or NVLINK fabric.
• Why it can beat weight-offloading to RAM/disk. With offload you re-stream gigabytes of weights per token from slow storage; in a split setup the weights stay resident in each node's VRAM and only kilobytes of activations move. So the distributed setup can be faster than single-box offload despite the network.
• Fault tolerance. If a node drops mid-inference, the topology is recalculated and the request is automatically retried on a new split.
• Underlying engine. Skippy is a patch queue on top of llama.cpp that exposes internals like activations and lets them filter tensors at model load. Several commenters criticized the blog post for not naming the inference engine. Pre-computed model splits are published to a MeshLLM HuggingFace org.
• KV cache. Each stage holds its own KV cache for the layers it hosts. When a stage is idle waiting its turn, that slack can be used for more parallelism (planned: n-gram speculative decoding for token verification).
• Encryption and privacy. Transport is encrypted end-to-end at the QUIC layer via iroh's key-based dialing (in-transit only). It is not in-process encrypted: whoever runs the inference sees the plaintext input/output. Contributors were candid that privacy from participating nodes and defense against activation poisoning are unsolved; the recommended answer today is a private mesh of trusted peers.
• Public-mesh incentives. No fairness/reward mechanism yet. The team considers it unnecessary while the real use case is private meshes you own; incentive structures may come once a public mesh grows large enough.
• Comparisons raised. Commenters mentioned exo (Mesh LLM docs have an exo-comparison page), cocompute (a commenter's own iroh-based project), and Colibri.