local-llm (jamesob)

jamesob's write-up of everything he knows about running current open-weights models locally: the bill of materials for his own machine, the BIOS and kernel settings that took a lot of fiddling to find, ready-to-run docker-compose.yml serving configs under runners/, and a speech-to-text setup. The README notes that nothing in it except the tables was written by AI, which is visible in the tone — it reads like someone showing you their rig, not like documentation.

Three price points

At roughly $2k you buy two RTX 3090s for 48GB of VRAM, which runs Qwen3.6-27B and leaves room for local speech-to-text. He rates the STT setup as the surprise of the build: cohere-transcribe (replacing whisper-large-v3) needs about 11GB of VRAM, and there is a ready config in runners/stt. He says he is comfortable using it precisely because it is local, in a way he is not with the hosted equivalent.

The $20k tier is a TODO. He lists 2x RTX 6000 Pro, four networked DGX Sparks, and Apple hardware as the options and says he has no experience there.

At roughly $40k you buy four RTX PRO 6000 Blackwell Workstation cards for 384GB of VRAM, about $46,000 of silicon on its own. His current model at that tier, as of 1 July 2026, is GLM-5.2-Int8Mix-NVFP4-REAP-594B served through vLLM with DCP4 and MTP5, hitting roughly 80 tokens/sec at 460k context. The REAP in that filename is reap-expert-pruning — the weights have already had their least-active experts cut.

The build

The unusual choice is spending on VRAM and refusing to spend on the host. The base system is a last-gen DDR4 EPYC box assembled almost entirely from eBay parts: ASRock Rack ROMED8-2T, EPYC Milan 7313P, 128GB of DDR4 ECC RDIMM, two 1700W PSUs, a 4TB boot NVMe and two 8TB NVMes for weights, all in an open frame. Total $5,687, which he estimates saves about $10,000 against a PCIe5/DDR5 host in the July 2026 market. The reasoning is that spilling into system RAM makes inference unusably slow for agentic workloads anyway, so host generation buys little once the model fits in VRAM.

The part that makes this more than a parts list is the PCIe switch. He runs a c-payne Microchip Switchtec PM40100 Gen4 switch (~$1,330 including the redriver host adapter and SlimSAS cables) so the four GPUs talk peer-to-peer at wire speed during the allreduce step of tensor parallelism, instead of routing everything through the PCIe root complex. GPU and switch live in a wooden enclosure he fabricated in a day, with the switch's own fan unplugged for being loud and useless.

The settings that took the longest

This is the section to come back to, because it is a list of things that silently do not work by default.

In BIOS: force the switch slot to x16, because bifurcation was splitting it and training the upstream link at Gen4 x8 (both SlimSAS 8i cables carry x8 each, so both must be connected). Force Gen4 rather than Auto, because Blackwell Gen5 devices negotiating down through a Gen4 switch can fail training and drop to Gen1. Disable ASPM, which drops idle links to 2.5GT/s — the links were actually running at Gen4 under load, but ASPM produced the alarming "downgraded" reading in lspci. Enable Resize BAR for full 96GB BAR exposure and P2P. Disable SR-IOV.

In the kernel: iommu=off amd_iommu=off nomodeset on the GRUB command line, plus options nvidia_uvm uvm_disable_hmm=1. Without iommu=off, NCCL hangs on multi-GPU P2P.

At runtime: ACS must be disabled, or peer-to-peer traffic gets bounced through the CPU root port and the switch does nothing at all. pcie_acs_override needs a patched kernel, so the repo ships a setpci loop run at boot from a systemd oneshot. Verification is lspci -vvv | grep ACSCtl showing all minus signs and nvidia-smi topo -m showing PIX between all four GPUs rather than PHB or NODE.

The measured result is Gen4 line rate: 27.5 GB/s unidirectional, 50.4 GB/s bidirectional, 0.37–0.45 µs latency. tools/measure-gpu-speed.sh reproduces the measurement.

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 350    # down from a 600W default

That power cap exists because the whole rig runs on a single 110V circuit, which he admits is probably unwise. 350W across four cards is 1,400W of GPU load sized to the PSU budget; during an earlier single-PSU phase the cards ran at ~260W each.

How he actually uses it

Model weights live on a ZFS filesystem replicated across the two 8TB drives, pulled with hf download <model> --local-dir ~/storage/<model>. Each model gets a directory with its own docker-compose.yml, and every container mounts ~/storage/models read-only. Serving lands on http://clank.j.co:5000 behind an internal DNS name.

The agent side runs on a separate VM: a small application that opens a tmux session per directory under ~/src, each running an opencode instance pointed at the inference machine's HTTP API. He credits tooling for most of what makes open models usable — camoufox plus a Kagi API key plus searXNG for browsing and search, a Telegram bot for alerts, and a private Gitea instance the agent can file PRs against. The VM only reaches the host through a shared filesystem mount, so the agent can install whatever it wants.

Caveats

No license is stated in the README. Everything here is Nvidia-specific and mostly AMD-EPYC-specific, and the ACS and IOMMU workarounds trade away isolation guarantees for bandwidth — reasonable on a dedicated bare-metal inference box, not something to copy onto a shared machine. Prices are July 2026 and the DDR5 argument depends on a RAM market that is currently distorted, which is hold-on-to-your-hardware.

The "$40k gets you something pretty close to Claude Opus" claim should be read against local-ai-is-not-opus, where Alex Ellis argues that the near-Opus framing is wrong even for well-tuned local setups. The two are not quite in conflict — Ellis is talking about far smaller hardware — but the vault should not treat "close to Opus" as settled.

titit-local-ai is the general case for running local models; local-llm-16gb-vram-tests is the same exercise at the other end of the budget, on one 16GB card. mesh-llm pools GPUs across machines instead of packing them into one chassis. For the model-side vocabulary the runner configs assume: mixture-of-experts, llm-quantization, kv-cache-sizing, moe-cpu-offload. Adjacent tooling: llama-cpp and ik-llama-cpp for the GGUF path this repo mostly bypasses in favor of vLLM.

Repo: jamesob/local-llm. Related repos it points at: local-inference-lab/rtx6kpro for multi-card RTX 6000 Pro tuning, and c-payne.com for the switches.