#inference

8 pages search “inference” ↳ all tags

Wiki 4

  • GLM5.2 on AMD MI355X Wafer's vendor benchmark claiming 2626 tok/s/node for GLM5.2 on MI355X, and how they got there
  • Neutrino-1 8B Fermion Research's Apache-2.0 8B: a 3.88 GB ternary-coded container for H100, MacBook, and CPU
  • Prompt Caching in Agents How KV-cache reuse sets the cost, latency and tool design of a coding agent, and what Pi shows
  • Speculative Decoding Predict multiple tokens per forward pass, verify in batch — four llama.cpp variants (draft model, MTP, EAGLE3, ngram-mod) trade off draft quality, memory, and applicability

Toolbox 4

  • deltafin Runs Kimi K3's 2.8T weights on one Apple Silicon Mac, streaming MXFP4 experts over HTTP to disk
  • ik_llama.cpp ikawrakow's llama.cpp fork with IQK quants, Q6 KV cache and faster long-context prompt processing
  • llama.cpp C/C++ inference engine for GGUF open-weights models, the base most local-LLM tools wrap
  • Petals BitTorrent-style distributed LLM inference across volunteer GPUs — working idea, dormant code since 2024