Petals
- title
- Petals
- type
- toolbox
- summary
- BitTorrent-style distributed LLM inference across volunteer GPUs β working idea, dormant code since 2024
- tags
- python, llm, distributed, p2p, inference, watchlist
- language
- Python
- license
- MIT
- created
- 2026-07-29
- updated
- 2026-07-29
Petals splits an LLM across machines that don't know each other. You load a handful of consecutive transformer blocks onto your own GPU, join a public swarm of people hosting the other blocks, and a forward pass hops from node to node until it reaches the output layer. The BitTorrent analogy is apt in the way that matters: nobody holds the whole file, serving is how you earn the right to consume, and the swarm's capacity is whatever its members happen to have plugged in. It came out of the BigScience research workshop, the same effort that produced BLOOM, and got a TechCrunch write-up in December 2022.
The pitch beyond "big model, small GPU" is that you are inside PyTorch rather than behind an API. Because your process holds real transformer blocks, you can run arbitrary fine-tuning and sampling methods, take custom paths through the model, and read hidden states β things a hosted completions endpoint will never expose.
Read the landing page with a date attached
petals.dev advertises Llama 3.1 (up to 405B), Mixtral 8x22B, Falcon 40B+, and BLOOM 176B, and quotes single-batch inference at up to 6 tokens/sec for Llama 2 70B and up to 4 tokens/sec for Falcon 180B. That is a 2023β2024 model lineup and a 2023β2024 set of measurements, presented in the present tense on a page with no date on it.
The repository backs that up. The last commit landed on 2024-08-25 ("Upgrade Pydantic to >= 2.0.0"), the last tagged release is v2.2.0 from 2023-09-06, and 113 issues are open. The repo is not archived and the swarm monitor at health.petals.dev is still linked, but the code has not changed in roughly two years as of July 2026, and both the throughput figures and the model list should be read as historical. Whether the public swarm still has hosts on it is a question the landing page cannot answer β check health.petals.dev, not the marketing copy.
Two years is a long time in this particular corner. The MoE architectures that now dominate open-weight releases already only touch a fraction of themselves per token (see mixture-of-experts), which changes the arithmetic Petals was designed around, and the single-machine offloading approaches it was built to beat have moved on considerably β deltafin streams a 2.8T-parameter model's experts off disk on one Mac, badly but locally. Quantization has moved too (llm-quantization), shrinking the class of models that genuinely need more than one consumer GPU.
Where the idea still stands
The throughput numbers are the honest part of the pitch and they were never good: 6 tokens/sec is about reading speed, adequate for a chatbot and useless for anything batch. That is the same ceiling Alex Ellis runs into arguing about local models generally, arrived at from the opposite direction β Petals trades latency and privacy for capacity you don't own, where local inference trades capacity for latency you control. Neither substitutes for a frontier subscription, and Petals adds a trust problem local inference doesn't have: your prompt and your hidden states pass through strangers' GPUs, and the project has always been explicit that the public swarm is not for sensitive data.
The design is still the clearest existing answer to "what would volunteer-computing LLM inference look like," which is why it is worth a page even dormant. A private swarm across machines you control is the deployment that survives the trust objection, and that use case doesn't care whether the public network is alive.
Tagged watchlist β see watchlist. The thing to re-check is binary: whether upstream resumes or the repo gets archived, and whether health.petals.dev shows any hosts.
MIT, ~10.5kβ as of 2026-07-29, 634 forks. Repository is dormant, not archived.