# 80 tok/sec and 128K context on 12GB VRAM with Qwen3.6-35B-A3B + MTP
# 80 tok/sec and 128K context on 12GB VRAM with Qwen3.6-35B-A3B + MTP
**Author:** janvitos (Reddit r/LocalLLaMA)
**URL:** https://www.reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/
**Posted:** 2026-05-09
**Hardware:** AMD Ryzen 7 9700X, RTX 4070 Super 12GB, 48GB DDR5-6000, CachyOS
**Stack:** llama.cpp master + MTP PR #22673
---
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec with 80%+ draft acceptance rate on the benchmark found here: [https://gist.githubusercontent.com/am17an/228edfb84ed082aa88e3865d6fa27090/raw/7a2cee40ee1e2ca5365f4cef93632193d7ad852a/mtp-bench.py](https://gist.githubusercontent.com/am17an/228edfb84ed082aa88e3865d6fa27090/raw/7a2cee40ee1e2ca5365f4cef93632193d7ad852a/mtp-bench.py)
Here's my PC specs:
```
OS: CachyOS (HIGHLY recommended)
CPU: AMD Ryzen 7 9700X
RAM: 48GB DDR5-6000 EXPO I
GPU: RTX 4070 Super 12GB
```
Results with other hardware may vary.
To run llama.cpp with MTP support, you need to build it from source and add a draft PR that hasn't yet been merged with the master branch. You can find a very nice guide on how to do that here and also download the Qwen3.6 MTP GGUF: [https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF) - Thanks [u/havenoammo](https://www.reddit.com/u/havenoammo)!
llama.cpp command:
```
llama-server \
-m Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \
-fitt 1536 \
-c 131072 \
-n 32768 \
-fa on \
-np 1 \
-ctk q8_0 \
-ctv q8_0 \
-ctkd q8_0 \
-ctvd q8_0 \
-ctxcp 64 \
--no-mmap \
--mlock \
--no-warmup \
--spec-type mtp \
--spec-draft-n-max 2 \
--chat-template-kwargs '{"preserve_thinking": true}' \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
```
The most important parameter here is -fitt 1536. Since part of the model is offloaded to CPU because of its size and , this tells llama.cpp to properly balance the load on the GPU/CPU to get the best possible performance, and leaves 1536 MB of free memory for the MTP draft model and KV cache. Since I'm running my dGPU as a secondary GPU (monitor plugged in the iGPU), I can use all the available 12GB VRAM for inference. 1536 might be too small if you use your dGPU as your primary GPU, so test it out first.
You can also try different values for -spec-draft-n-max. I got slightly better tok/sec with 3, but a much better acceptance rate with 2, so the trade off was not worth it. With MTP, you want to maximize speed AND acceptance, so you need to find the best balance between both.
Benchmark results:
```
mtp-bench.py
code_python pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=80.8
code_cpp pred= 58 draft= 40 acc= 37 rate=0.925 tok/s=81.8
explain_concept pred= 192 draft= 152 acc= 114 rate=0.750 tok/s=70.0
summarize pred= 53 draft= 40 acc= 32 rate=0.800 tok/s=75.4
qa_factual pred= 192 draft= 144 acc= 119 rate=0.826 tok/s=77.8
translation pred= 22 draft= 16 acc= 13 rate=0.812 tok/s=81.9
creative_short pred= 192 draft= 160 acc= 111 rate=0.694 tok/s=69.2
stepwise_math pred= 192 draft= 144 acc= 119 rate=0.826 tok/s=76.5
long_code_review pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=73.2
```
If you have any questions, feel free to ask :)
Cheers.
---
## Comments
> **zulutune** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okta396/) · 50 points
>
> Hey OP thank you so much for this. I have an underutilized 5070ti and I’m going to try this out. Hopefully this weekend.
>
> > **zulutune** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktadje/) · 8 points
> >
> > Btw did you try DeepSeekV4? I’m kinda curious for this model too.
> >
> > > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktfh8r/) · 23 points
> > >
> > > I've tried DeepSeek V4 cloud for coding and didn't like it at all. It was overthinkig way too much and seemed confused and paranoid. But that's me. I'm sure others would debate this 😄
> > >
> > > When using cloud models for coding, GPT 5.5 is my top choice. In my opinion, its deterministic behavior makes it extremely apt at one shotting large and complex code additions/modifications.
> > >
> > > To be honest, I found Qwen3.6 35B A3B local to be in the same league as most other and bigger open LLMs, except GLM 5.1, which can debug and resolve issues that Qwen3.6 cannot.
> > >
> > > > **rz2000** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuudyz/) · 4 points
> > > >
> > > > Have you tried DeepSeek v4 with different thinking parameters? Using the flash version locally, I’ve found that completely turning off thinking gets good results.
> > > >
> > > > I’ve only used it with chat. In Kagi Assistant which uses fireworks.ai, together.ai, or deepinfra, it can be extremely slow with either the pro or flash version. However the quality of the written analysis is very good with ot without websearch enabled.
> > > >
> > > > Locally, I have used [https://github.com/antirez/ds4](https://github.com/antirez/ds4) to run the flash version. This custom engine achieves pretty excellent performance, and here is where I have found a lot of benefit to simply switching of the reasoning step with `\nothink`.
> > > >
> > > > I can’t run the full pro version, but it is pretty amazing to get better performance from the flash version than I can get from cloud providers, albeit with Kagi in between.
> > > >
> > > > > **zulutune** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okv149s/) · 2 points
> > > > >
> > > > > Does that mean you have a macbook with 128GB?
> > > > >
> > > > > > **rz2000** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okv3gqp/) · 2 points
> > > > > >
> > > > > > A Mac Studio with 256 GB. I think Mac Studios with 192GB+, or the maxed out M5 MacBook Pro is what antirez was targeting with this inference engine.
> > > > > >
> > > > > > In a couple years this sort of performance will likely be cheap, and it would worry me more if I were Google, OpenAI, Anthropic than some of the other open model releases that suddenly made AI briefly crash.
> > > > > >
> > > > > > I haven’t gotten Gemma 4 with MTP acceleration to work very reliably yet, but that is another way that local inference is becoming viable for much more than just hobbyist use.
> > > > > >
> > > > > > > **zulutune** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okv44hr/) · 1 point
> > > > > > >
> > > > > > > Gratz, you’re in a different league.
> > > > > > >
> > > > > > > So how does Qwen 3.6 and DS4 compare, what’s your favorite? Do you ever feel the need to use cloud models, or does that level of GB’s really give you the raw power of a Opus/Codex?
> > > > > > >
> > > > > > > > **rz2000** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okwa1ae/) · 2 points
> > > > > > > >
> > > > > > > > I haven’t used either for much code assistance.
> > > > > > > >
> > > > > > > > The “personality” of DeepSeek v4 is much more like GLM 4.6 or 4.7, which I think is pretty good, but without the need to quantiize it down to 4bits which can result is strange errors. DeepSeek v4 flash fits in 160GB of memory at full precision.
> > > > > > > >
> > > > > > > > For tasks other than coding I find Qwen pretty unbearable. It seems very incurious and very worried about anything that might be innovative.
> > > > > > > >
> > > > > > > > > **zulutune** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oky9u80/) · 1 point
> > > > > > > > >
> > > > > > > > > Interesting observation haha :) thanks for sharing your insights
> > >
> > > > **zulutune** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktg339/) · 2 points
> > > >
> > > > Thanks for posting this!
> **StupidScaredSquirrel** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt5pyv/) · 24 points
>
> Why -no-mmap?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt6pvb/) · 41 points
> >
> > It's a general llama.cpp recommendation when using --mlock (prevents swapping to disk). --no-map loads the entire model into RAM instead of loading parts when needed. As I understand it, it prevents disk I/O and makes memory usage more predictable. It might result in slower loading times, but better stability during inference.
> >
> > > **sh4rk1z** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktjueu/) · 29 points
> > >
> > > benchmarked mmap and no-mmap less than an hour ago on rtx2070S/ryzen3950x/64ram with vram limited to 6.25GiB for qwen-3.5-9B-ud-q4-k-xl so I can use my desktop while using the local model. Result over 3 runs with no-mmap gave improvements:
> > >
> > > \- ~1.5% decode speed improvement.
> > > \- ~5.2% prompt processing improvement.
> > > \- 28 MB vram less used.
> > > \- standard deviation between runs droped by 10-20x
> > > \- no disk io so less wear/tear
> > >
> > > I'm still experimenting with some things (turboquant, trellis) and will post once done and then try the Qwen 3.6.
> > >
> > > > · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktl09n/) · 3 points
> > > >
> > > > \[deleted\]
> > > >
> > > > > **sh4rk1z** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktlokw/) · 3 points
> > > > >
> > > > > no-mmap
> > > > >
> > > > > > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktlzpq/) · 1 point
> > > > > >
> > > > > > Thanks!
> > >
> > > > **letsgoiowa** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktkw5o/) · 1 point
> > > >
> > > > Std?
> > > >
> > > > > **CircularSeasoning** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktx8c2/) · 53 points
> > > > >
> > > > >
> > > > >
> > > > > The user has entered three letters, "Std", with a question mark, possibly hoping to elicit more information about (Something To Do?) with "Std"? I'm not sure what that means. I should ask for clarification.
> > > > >
> > > > > Wait! The user's name is 'letsgoiowa' (i.e., "Let's go, Iowa!") so let me research what happens in Iowa in connection with the letters or acronym, "STD"...
> > > > >
> > > > > \[web search content omitted\]
> > > > >
> > > > > Ah.
> > > > >
> > > > > I should helpfully advise the user to test for: Chlamydia.
> > > > >
> > > > > All good.
> > > > >
> > > > > Proceed.
> > > > >
> > > > >
> > > > >
> > > > > > **sh4rk1z** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktxfrf/) · 6 points
> > > > > >
> > > > > > 😂😂😂
> > > > > >
> > > > > > > **CircularSeasoning** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oku1fjr/) · 4 points
> > > > > > >
> > > > > > > letsgoiowa looking at me all σ\_σ
> > > > >
> > > > > > **letsgoiowa** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okwbhyk/) · 5 points
> > > > > >
> > > > > > Standard deviation lol
> > > > > >
> > > > > > But thanks
> > >
> > > > **theowlinspace** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okujete/) · 1 point
> > > >
> > > > —mmap with —mlock shouldn’t use disk io after you’ve loaded the model because it locks the mapped pages in RAM
> >
> > > **BitGreen1270** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okxkrru/) · 1 point
> > >
> > > I have a 780m igpu and adding --no-mmap makes it use 2GB extra RAM with nothing else changing. My prompt is just a 500 word story in the style of Roald Dahl. Since I only have 32GB, that's a pickle. No difference in tps though - still getting exactly the same. This is for non-MTP though. I'm downloading the MTP version to try out with your params (thanks so much!)
> > >
> > > **StupidScaredSquirrel** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt706q/) · 1 point
> > >
> > > But when is it useful to have mmap then? If having -no-mmap still loads what is needed?
> > >
> > > > **farkinga** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okta5s8/) · 10 points
> > > >
> > > > When the model is big, and when the weights will be in system ram anyway (e.g. a moe) , use mmap (on Linux) to avoid loading the whole model into ram. With mmap, Linux will load the weights into ram as needed. However, use no-mmap if you have a performance reason to keep the weights in ram anyway. It should run a little faster with no-mmap but it takes longer to start.
> > > >
> > > > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt7kr9/) · 4 points
> > > >
> > > > To be honest, I would say try both and see what works best for ya 😄
> > > >
> > > > **Marksta** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktebur/) · 1 point
> > > >
> > > > Because mmap is faster to load the model on Linux where it has a real system level mmap. And if you were to turn off the server and turn it back on again, the model would already be in mmap. Restarting something big like Deepseek without mmap would mean waiting a few minutes each time to load it, unload it, load it again...
>
> > **dark-light92** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt6lka/) · \-1 points
> >
> > So it doesn't mmap.
> **Still-Notice8155** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktobbo/) · 17 points
>
> Qwen3.6-35B-A3B-MTP-UD-Q2\_K\_XL.gguf on GTX 1070 8GB + i7-11700 16GB
>
> Config: turboquant+MTP | n-cpu-moe 32 | turbo4/turbo3 KV | ctx 131K | ctx-checkpoints 8
>
> \---
>
> Gen t/s degradation (attention O(n) cost):
>
> 0K: 48 t/s ████████████████████████████████
>
> 10K: 31 t/s █████████████████████
>
> 30K: 28 t/s ██████████████████
>
> 50K: 23 t/s ███████████████
>
> 80K: 23 t/s ███████████████ ← DeltaNet plateau
>
> 100K: 19 t/s ████████████
>
> 125K: 13.6 t/s █████████
>
> Curve flattens 30-80K thanks to 30 DeltaNet O(1) layers. Only 10 attention layers drive degradation.
>
> PP t/s (batch-driven, unaffected by context):
>
> Short prompt (<20 tok): 41 t/s avg — overhead bound
>
> Batched prompt (50+ tok): 135 t/s avg — GPU parallel
>
> At 125K ctx: still 78-95 t/s PP
>
> Draft acceptance: 58-86% depending on task predictability. Lifetime: ~90%.
>
> VRAM: 7.5 GB used, 633 MB free at 131K. Turbo4/turbo3 KV = 590 MB (vs 720 MB q4\_0).
>
> RAM: 12 GB used (model no-mmap = 13.2 GB + MoE CPU offload + 500 MB prompt cache). 2 GB free with checkpoints=8.
>
> Improvement over non-MTP baseline:
>
> Non-MTP MTP+turbo Speedup
>
> 5K: 27.4 → 48 = 1.8x
>
> 80K: ~7 → 23 = 3.3x
>
> 125K: ~3 → 13.6 = 4.5x
>
> The gap widens at high context — MTP saves ~constant time per token regardless of context, while attention cost grows linearly.
>
> > **DunderSunder** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktwkvg/) · 5 points
> >
> > > Qwen3.6-35B-A3B-MTP
> >
> > which quant is this?
> >
> > > **Still-Notice8155** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktx0pk/) · 4 points
> > >
> > > Qwen3.6-35B-A3B-MTP-UD-Q2\_K\_XL.gguf. I would love to test the Q4\_K\_M, but I don't have enough RAM for now.
> > >
> > > \--n-cpu-moe 32 --no-mmap --parallel 1 \\
> > >
> > > \--ctx-checkpoints 8 \\
> > >
> > > \--spec-type mtp --spec-draft-n-max 3 \\
> > >
> > > \--cache-type-k turbo4 --cache-type-v turbo3 \\
> > >
> > > \--jinja -c 131072 -fit off
> > >
> > > **Still-Notice8155** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okyig96/) · 1 point
> > >
> > > Qwen3.6-35B-A3B IQ4\_XS + MTP on GTX 1070 8GB
> > >
> > > Hardware
> > >
> > > CPU: i7-11700 (8c/16t)
> > >
> > > RAM: 32 GB DDR4-3200
> > >
> > > GPU: GTX 1070 8GB (Pascal, stock clocks, no OC)
> > >
> > > OS: Ubuntu 26.04, CUDA 12.4, driver 580.142
> > >
> > > Model
> > >
> > > Name: Qwen3.6-35B-A3B (MoE, 256 experts, 8 active, 3B active params)
> > >
> > > Quant: IQ4\_XS (19.4 GB, 4.37 BPW)
> > >
> > > MTP: Q8\_0 draft heads, 3-token speculative decoding
> > >
> > > Arch: 30 DeltaNet (O(1)) + 10 quadratic attention (O(n)) layers
> > >
> > > Context: 131,072 tokens
> > >
> > > Server Flags
> > >
> > > \--n-cpu-moe 35 --no-mmap --parallel 1 --ctx-checkpoints 32
> > >
> > > \--spec-type mtp --spec-draft-n-max 3
> > >
> > > \--cache-type-k turbo4 --cache-type-v turbo4
> > >
> > > \--jinja -c 131072 -fit off
> > >
> > > Build: llama.cpp master + PR #22673 (MTP) + turboquant cache patches
> > >
> > > Turbo4 KV cache: 4-bit WHT quantization for K and V
> > >
> > > Gen Speed vs Context
> > >
> > > 0–15K: 32.1 t/s
> > >
> > > 15–40K: 28.1 t/s
> > >
> > > 40–70K: 24.3 t/s
> > >
> > > 70–100K: 23.0 t/s
> > >
> > > 100–131K: 18.1 t/s
> > >
> > > Prompt Processing
> > >
> > > 0–15K: 148 t/s
> > >
> > > 40–70K: 107 t/s
> > >
> > > 100–131K: 64 t/s
> > >
> > > Draft Acceptance (MTP)
> > >
> > > Per-task: 42–89% (varies by difficulty)
> > >
> > > Global: 75–80% lifetime
> > >
> > > VRAM at 131K
> > >
> > > GPU model: 4,578 MB
> > >
> > > KV cache: 1,122 MB (turbo4 compressed)
> > >
> > > Recurrent: 251 MB
> > >
> > > Compute: ~493 MB
> > >
> > > Total: ~7.6 GB / 475 MB free
> > >
> > > RAM
> > >
> > > 22 GB used / 9 GB free (32 GB total, --no-mmap)
> > >
> > > Have retested in 32GB ram. Still good performance. I'm not sure about the quality degredation.
> > >
> > > > **Still-Notice8155** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okyzn16/) · 1 point
> > > >
> > > > I have tried this benchmark [https://github.com/alexziskind1/codeneedle](https://github.com/alexziskind1/codeneedle)
> > > >
> > > > \## Qwen3.6-35B-A3B IQ4\_XS + MTP — CodeNeedle Positional Recall
> > > >
> > > > Tests exact line-by-line recall: stuff entire source into context, reproduce
> > > >
> > > > functions verbatim. Pass = ≥8/20 lines match exactly including whitespace.
> > > >
> > > > MTP speculative decoding at n=3, turbo4 quantized KV cache.
> > > >
> > > > \### Results
> > > >
> > > > HTTP no-think: 10/11 PASS (91%), 187/220 lines (85%), 50 total hallucinations
> > > >
> > > > HTTP think: 9/11 PASS (82%), 186/220 lines (85%), 66 total hallucinations
> > > >
> > > > jQuery no-think: 14/16 PASS (88%), 283/320 lines (88%), 319 total hallucinations
> > > >
> > > > jQuery think: 14/16 PASS (88%), 271/320 lines (84%), 43 total hallucinations
> > > >
> > > > \### MTP Draft Acceptance
> > > >
> > > > Global Per-task range
> > > >
> > > > HTTP no-think 94% 86-100%
> > > >
> > > > HTTP think 93% 86-100%
> > > >
> > > > jQuery no-think 91% 51-100%
> > > >
> > > > jQuery think 87% 62-100%
>
> > **FirefoxMetzger** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvn7nx/) · 2 points
> >
> > what does the turboquant refer to here? K/V cache or or model quantization?
> >
> > > **Still-Notice8155** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okwyx46/) · 5 points
> > >
> > > KV cache.
>
> > **Ok\_Jury\_8311** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/olc7b7q/) · 2 points
> >
> > can you please share steps to have turboquant+MTP running?
> >
> > > **Still-Notice8155** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/olc8lik/) · 1 point
> > >
> > > git clone [https://github.com/jmpangilinan/llama-cpp-turboquant.git](https://github.com/jmpangilinan/llama-cpp-turboquant.git)
> > > cd llama-cpp-turboquant
> > > git checkout mtp-turboquant
> > > cmake -B build -DGGML\_CUDA=ON
> > > cmake --build build --config Release --target llama-server
> **FrostWolfDota** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktd09y/) · 13 points
>
> I have a 16GB AMD cpu, will try to reproduce it when I find some time. Never tried using llama.cop directly, only through LM studio.
>
> > **house\_monkey** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuhnds/) · 14 points
> >
> > Wish I could reproduce my 16GB AMD cpu
> **429\_TooManyRequests** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktr0gf/) · 7 points
>
> Wow this post is perfect timing. I have a 3080 Ti and was depressed I couldn’t get this exact model working last night. I’ll try it out today and send results!
> **ai\_without\_borders** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oku503f/) · 6 points
>
> the 80 tok/s is with 128K context loaded — at shorter contexts (4-8K) you would be pushing 100+ easily. MTP overhead shows up more in prompt processing than in token generation, so the win is biggest on long generation runs vs short QA bursts. good config though, -no-mmap with mlock is the right call for sustained throughput.
> **slimdizzy** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktayef/) · 4 points
>
> I have a 3080 12gb I will try this on. Thanks muchly OP!
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktgja2/) · 4 points
> >
> > Awesome! Share your results 😄
> **Independent-Flow3408** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktt87f/) · 5 points
>
> This is a really useful writeup, thanks.
>
> The "-fitt 1664" detail is the part I would have missed. For long-context coding workflows, did you notice the speed dropping mainly from KV/cache pressure, or from CPU/GPU balancing once the context gets large?
>
> Also curious if you tested this with an agent workflow like OpenCode/Continue, or only direct llama.cpp prompting.
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuhmfq/) · 4 points
> >
> > Speed drops towards 50 tok/sec when context has filled up near 128K. But that's still very reasonable and usable. Didn't notice quality degradation.
> >
> > I've been using this with Opencode for the past few days without any issues. I can analyze the entire codebase of a small project, which fills up the context near 75K, and continue working on it normally no problem.
> >
> > So yeah, I would consider this as pretty stable 😄
> **MistingFidgets** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okx0uke/) · 6 points
>
> Spec Decode and MTP are really awesome. I have some benchmark data i want to share but can't post yet, need some upvotes on comments before localllama will let me.... help me out here
> **ElChupaNebrey** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktesm4/) · 4 points
>
> What is you speed on 27b
>
> > **twiddlebit** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okticji/) · 10 points
> >
> > 27b wont fit on 12gb of vram so probably not very good
> >
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktgz0b/) · 5 points
> >
> > I haven't even tried it after seeing other people's benchmarks. I know it wouldn't be fast enough for real-world coding anyways, so I'll wait until some miracle happens or I buy a new GPU 😄
> **ducksoup\_18** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktk0t6/) · 5 points
>
> I have 2 3060s for a total of 24gb vram. I'd love to see these kind of numbers with that setup. Will try.
> **HavenTerminal\_com** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oku0m0x/) · 5 points
>
> the spec-draft-n-max 2 vs 3 finding is the kind of thing you only figure out by running both. appreciate you logging it.
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okui605/) · 1 point
> >
> > And I recommend that everyone test their own values, as I've seen others find success with 3 or 4 😄
> **Fuzilumpkinz** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt9nsy/) · 3 points
>
> I’ll try this for sure. I’m getting 40 atm but I’m on a 6700 xt. Curious if I can find any increases
> **burdzi** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktcfch/) · 3 points
>
> Nice 🤩 does MTP also work for vision? If I give it images?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktdr9z/) · 5 points
> >
> > There seems to be some issues with vision at the moment. You can read about it here: [https://github.com/ggml-org/llama.cpp/issues/22867](https://github.com/ggml-org/llama.cpp/issues/22867) and on the official PR thread: [https://github.com/ggml-org/llama.cpp/pull/22673](https://github.com/ggml-org/llama.cpp/pull/22673)
> **masterlafontaine** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktihyu/) · 3 points
>
> What is the prompt speed? Usually this is what makes agentic code the most boring and slow. It's usually about reading, say 50k, then writting 3k.
> **sirnixalot94** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oku2evx/) · 3 points
>
> I haven’t tried MTP yet, but I have that same model running on an RTX 4080 16GB with —cpu-moe=20 (Ryzen 9 5950X and 64GB system RAM) and I’m getting 105t/s pp and right at 50t/s generation speed. I’m going to check this out and see if adding this in addition to that will improve my performance even more. Thanks for the findings!
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuoa40/) · 3 points
> >
> > Definitely try the -fitt flag. It replaces --cpu-moe and the guessing work. The only thing you need to figure out is the right amount of reserved RAM. So for non-MTP, I started with -fitt 256, but ran into OOM errors here and there. It was rock solid with -fitt 512. You can check your VRAM usage with nvidia-smi. For me, 11800MiB / 12282MiB is pretty much the max I can push.
> **cognitium** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okucfap/) · 3 points
>
> Are you actually getting good output from that model though? It's the fastest local model I've ever used because only 3B are active at a time but it'll use half of it's context endlessly soliloquizing about how it's a good model that follows the rules and then doesn't follow them.
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okupcd7/) · 3 points
> >
> > Try it with thinking disabled: --chat-template-kwargs '{"enable\_thinking": false}' (might be slightly different for Windows).
> >
> > I feel it's pretty darn good at coding this way. Make sure you use the right launch parms for instruct / non-thinking mode though: temperature=0.7, top\_p=0.80, top\_k=20, min\_p=0.0, presence\_penalty=1.5, repetition\_penalty=1.0
> >
> > > **cognitium** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okur2xv/) · 1 point
> > >
> > > Alright, I'll try those. I spent most of yesterday playing with qwen3.6 35B and 27B and they both have issues with over thinking. The speed of 35B is what's most impressive.
> > >
> > > > **Substantial-Thing303** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okz739k/) · 1 point
> > > >
> > > > you can also try deepseek's scratchpad grammar on qwen3.6 to cut down on the thinking: [https://github.com/noonghunna/club-3090/blob/master/docs/STRUCTURED\_COT.md](https://github.com/noonghunna/club-3090/blob/master/docs/STRUCTURED_COT.md)
> **\_bones\_\_** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okyzgg6/) · 3 points
>
> Getting 60t/s on an RTX3080 12GB with this setup. So quite useful!
>
> I am getting a huge preprocessing time in an existing session, which is a bit weird, as I didn't have that with regular Qwen 3.6 before this, a Q3 that got me 45t/s.
>
> Definitely interesting stuff, thanks for posting.
> **alchninja** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktteu3/) · 2 points
>
> Hey, thanks for the info! Could I ask what your CPU and RAM specs are? I'm on a Ryzen 5700x and 32GB DRR4-3600, just trying to get a feel for how much people are able too benefit from having newer CPUs and DDR5.
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okun2hr/) · 3 points
> >
> > Here's my specs:
> >
> > AMD Ryzen 7 9700X
> > 48GB DDR5-6000
> >
> > I'm surprised I'm not encountering more issues with the 3 x DIMM RAM config. It's actually running great even with EXPO I 😄
> >
> > I was able to run the same model (non-MTP) with 32GB, but it was tight. That's why I stole a 16GB DIMM from my son's gaming PC. With 48GB, I have a 10-12 GB buffer at all times when the model is loaded.
> >
> > One thing to note, since installing CachyOS, I noticed it's way less RAM hungry than Windows. And to be honest, once everything is setup properly, CachyOS is pretty incredible. It's actually my daily driver now. I haven't switched back to Windows in days.
> >
> > > **alchninja** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvjcny/) · 2 points
> > >
> > > Thanks! I bet your son is super happy about his missing RAM stick lol
> > >
> > > Yep, getting into local LLMs and seeing how Kubuntu breathed new life into my 9 year old Dell XPS (I don't know how I lived without KDE Plasma for so long) finally pushed me away from Windows for good on all my machines. I still keep it on a partition just for the occasional gaming session with a friend (unfortunately the stuff we play needs Windows) but I can't imagine ever using it as my daily again.
> **Sufficient\_Sir\_5414** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okus80a/) · 2 points
>
> How are you balancing the KV cache for the 128k context window alongside the MTP draft model on only 12GB? Did you have to aggressively tune the -fitt parameter or sacrifice context depth to maintain that 80% acceptance rate?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuua4v/) · 1 point
> >
> > That's the magic of -fitt: Once you find the sweet spot that doesn't cause any OOM, you get a rock solid local inference setup that can perform very well even on a hybrid GPU/CPU config. No long tuning. No sacrifices. Just a few code analysis / creation runs with the agent to fill the context and test the VRAM limit.
> **coolaznkenny** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okux048/) · 2 points
>
> Going to utilize this guide once i get my hands on a steam machine!
> **FirefoxMetzger** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvmtom/) · 2 points
>
> Hm, so the reason this works as well as it does is that you offload layers to host memory (i.e. your total footprint is >12GB) and you increase decode tok/s with speculative decoding using a draft model?
>
> > **janvitos** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okwxn9k/) · 2 points
> >
> > Exactly!
> **oviteodor** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvyyf0/) · 2 points
>
> Thank you OP
> **BitGreen1270** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okyfeai/) · 2 points
>
> This is very cool, thanks for sharing. I used the same prompt on the non-MTP and the MTP version and got the following:
>
> Non-MTP - \[ Prompt: 80.3 t/s | Generation: 21.6 t/s \]
>
> MTP - \[ Prompt: 71.9 t/s | Generation: 28.1 t/s \]
>
> Prompt speed seems to have gone down, but token generation has gone up significantly. This is on my 780m iGPU.
> **pwmcintyre** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okzty96/) · 2 points
>
> legend! i'm finally getting useful results on my 4070 12GB
> **chille9** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol0tc9i/) · 2 points
>
> 50 t/s with rtx 4060Ti 16Gb and 32gb ram! Also using the q5 quant at a 98k context! Magnificent.
>
> > **Loouiz** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol7d9yr/) · 1 point
> >
> > Is it stable? Did you make any other adjustments? I'm trying this with a 16gb 4080 super an 32gb ram and I'm gettin oom here and there...
> >
> > > **chille9** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ola7wfw/) · 1 point
> > >
> > > I´ve made very small adjustments. I also recompiled using the instructions that op had listed.
> > >
> > > Here´s the bat file i run in my llama dir where you can see my settings.
> > > [https://pastebin.com/dSkkKX60](https://pastebin.com/dSkkKX60)
> > >
> > > It´s been pretty stable for me. I hope you can solve it! Only getting oom or errors on using text files and pdfs. Pdfs and text works great on the Q4 qwen 35B MTP model.
> > >
> > > Edit: put --spec-draft-n-max to 3 instead of 2 and no crashes with pdfs.
> **b0ts** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol1d2jx/) · 2 points
>
> On my 3070 (8GB) with a Ryzen 9 7900x and 64GB DDR5 6400:
>
> [](https://preview.redd.it/cmoup89zoc0h1.png?width=920&format=png&auto=webp&s=476c05c9755c2fa1cdd915062a6c7b92cdb14f0f)
> **RaspNAS** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol4byqa/) · 2 points
>
> I tried the MTP benchmark on llama.cpp too after seeing your post.
> Thanks a lot! This ultra-high-speed LLM is insane !!!!
> Hardware: - GPU: RTX 3060 12GB - CPU: Ryzen 9 5950X (16 threads) - RAM: DDR4-3200 40GB - OS: Windows 11 Pro (on Proxmox with PCIe Passthrough)
>
> \`\`\`powershell Administrator in 🌐 letwir-main in ~\\Documents via v24.14.0 via 🐍 v3.14.2 (.venv) ❯ curl [https://gist.githubusercontent.com/am17an/228edfb84ed082aa88e3865d6fa27090/raw/7a2cee40ee1e2ca5365f4cef93632193d7ad852a/mtp-bench.py](https://gist.githubusercontent.com/am17an/228edfb84ed082aa88e3865d6fa27090/raw/7a2cee40ee1e2ca5365f4cef93632193d7ad852a/mtp-bench.py) -o mtp-bench.py % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 7709 100 7709 0 0 77194 0 --:--:-- --:--:-- --:--:-- 79474
>
> Administrator in 🌐 letwir-main in ~\\Documents via v24.14.0 via 🐍 v3.14.2 (.venv) ❯ sd "8080" "11434" .\\mtp-bench.py
>
> Administrator in 🌐 letwir-main in ~\\Documents via v24.14.0 via 🐍 v3.14.2 (.venv) ❯ py .\\mtp-bench.py code\_python pred= 192 draft= 156 acc= 138 rate=0.885 tok/s=38.9 code\_cpp pred= 192 draft= 180 acc= 131 rate=0.728 tok/s=35.0 explain\_concept pred= 192 draft= 189 acc= 128 rate=0.677 tok/s=33.7 summarize pred= 53 draft= 48 acc= 36 rate=0.750 tok/s=37.4 qa\_factual pred= 192 draft= 180 acc= 131 rate=0.728 tok/s=35.2 translation pred= 22 draft= 24 acc= 13 rate=0.542 tok/s=31.6 creative\_short pred= 192 draft= 207 acc= 122 rate=0.589 tok/s=31.1 stepwise\_math pred= 192 draft= 174 acc= 133 rate=0.764 tok/s=35.8 long\_code\_review pred= 192 draft= 192 acc= 127 rate=0.661 tok/s=32.8
>
> Aggregate: { "n\_requests": 9, "total\_predicted": 1419, "total\_draft": 1350, "total\_draft\_accepted": 959, "aggregate\_accept\_rate": 0.7104, "wall\_s\_total": 46.07 } ` build options: ` .\\vcpkg install pthreads openssl curl\[core,http2,http3,openssl,ssh,zstd\] --triplet x64-windows git fetch origin pull/22673/head:mtp-clean cmake -B build -DGGML\_CUDA=ON -DGGML\_CUDA\_F16=ON -DGGML\_CUDA\_GRAPHS=ON -DCMAKE\_TOOLCHAIN\_FILE="C:/PATH/vcpkg/scripts/buildsystems/vcpkg.cmake" `` add options: `--threads 16 --threads-batch 16` change options: `--spec-draft-n-max 3` ``powershell llama-server --port 11434 --host 0.0.0.0 --threads 16 --threads-batch 16 -m "A:\\LLM\\Qwen3.6-35B-A3B-MTP-UD-Q3\_K\_XL.gguf" -fitt 1736 -c 131072 -n 32768 -fa on -np 1 -ctk q8\_0 -ctv q8\_0 -ctkd q8\_0 -ctvd q8\_0 -ctxcp 64 --no-mmap --mlock --no-warmup --spec-type mtp --spec-draft-n-max 3 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --jinja --webui-mcp-proxy \`\`\`
> **eliko613** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol4gks6/) · 2 points
>
> Great writeup — the -fitt tuning is genuinely underappreciated. Most people just set -ngl 99 and wonder why their CPU is saturated.
>
> A few things that helped me squeeze out a bit more on a similar split setup:
>
> - Bumping --ctxcp slightly (128 worked better for me than 64 at longer context) — worth benchmarking your specific use case
> - \--spec-draft-n-max 2 is conservative; if your draft model is fast you can push to 3–4 and get meaningful throughput gains
> - With preserve\_thinking: true the KV cache fills up fast at 131k context — make sure you're actually using that window or trim -c to free headroom
>
> Also been using zenllm.io for quick parameter testing before committing to long runs — handy for dialing in temp/top-p without burning local resources. Not affiliated, just a useful scratch pad.
>
> What's your tok/s looking like on this config?
> **zerozero023** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol5qcwv/) · 2 points
>
> Nice write-up. The -fitt flag is something I never paid attention to before — makes sense for hybrid GPU/CPU setups. Did you notice any quality difference with Q4\_K\_XL vs higher quants at this context size?
> **q-admin007** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol6oe2g/) · 2 points
>
> Awesome work. I have a 5070 Ti 16GB connected via Oculink with a Strix Halo. Will give it a go later with UD-Q6\_K\_XL. It seems to be the sweetspot in terms of precision on smaller systems. I also would rather half my context and use f16 there.
> **admajic** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt628g/) · 4 points
>
> Huh? On a 3090 I'm getting average 150 tok/s and tops at 200 tok/s. Amazing how offloading destiny's u
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okt76rl/) · 12 points
> >
> > That's awesome! It's actually because the entire model fits into your VRAM, which is impossible on a 12GB GPU.
> >
> > **PrometheusZer0** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okta310/) · 1 point
> >
> > what's your setup? Lucebox?
> >
> > > **admajic** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okxey8q/) · 2 points
> > >
> > > Using mtp you need to pull it from git i did a write-up about it
> **yoomiii** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okzdcxt/) · 2 points
>
> wake me up when MTP PR is merged
> **damianzoys** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktdf1u/) · 1 point
>
> I got some nice tok/s too, but the hallucinations make it almost impossible to use. It hallucinates tools and directories which aren’t there, even with low temperature. Any idea how to fix this?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktedoe/) · 2 points
> >
> > Are you getting these hallucinations with MTP only?
> >
> > To be honest, I haven't noticed any issues with MTP and have been using it for a few days to do some code work, but no major project yet. No tool issues at all. For my setup, Qwen3.6 is actually much more stable with tools than Gemma 4.
> **mindinpanic** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktuxcl/) · 1 point
>
> Promising! Did you get any issues with the coding agent context?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okunizx/) · 2 points
> >
> > Nope 😄
> **feik696** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okueox7/) · 1 point
>
> I'm not too experienced with PCs, so I've mostly been using LM Studio, which has the same graphics card as yours. However, where LM Studio shows 30 tokens per second, I'm getting half that amount here. It's possible that I've made a mistake with the compilation, but then again, it wouldn't have started in the first place, right?
>
> > **feik696** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuexux/) · 1 point
> >
> > code\_python pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=14.1
> >
> > code\_cpp pred= 192 draft= 138 acc= 121 rate=0.877 tok/s=13.4
> >
> > explain\_concept pred= 192 draft= 152 acc= 114 rate=0.750 tok/s=13.2
> >
> > summarize pred= 53 draft= 40 acc= 32 rate=0.800 tok/s=14.1
> >
> > qa\_factual pred= 192 draft= 140 acc= 121 rate=0.864 tok/s=14.7
> >
> > translation pred= 22 draft= 16 acc= 13 rate=0.812 tok/s=14.7
> >
> > creative\_short pred= 192 draft= 156 acc= 113 rate=0.724 tok/s=13.1
> >
> > stepwise\_math pred= 192 draft= 140 acc= 121 rate=0.864 tok/s=14.5
> >
> > long\_code\_review pred= 192 draft= 146 acc= 117 rate=0.801 tok/s=13.7
> >
> > Aggregate: {
> >
> > "n\_requests": 9,
> >
> > "total\_predicted": 1419,
> >
> > "total\_draft": 1060,
> >
> > "total\_draft\_accepted": 877,
> >
> > "aggregate\_accept\_rate": 0.8274,
> >
> > "wall\_s\_total": 113.26
> >
> > }
> >
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuq9xv/) · 1 point
> >
> > I also started with LM Studio, but to be frank, I never got good results with it. When I switched to llama.cpp, it was a night and day difference. LM Studio is a wrapper around llama.cpp that seems to add latency to the process. And you can never really be sure which parameters it passes to llama.cpp. If you can run llama.cpp directly, I'm pretty confident you'll get much better tok/sec!
> **ItsRektTime** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuk1v3/) · 1 point
>
> I got the following benchmark results on a 3060 12GB and R5 5600 with 32GB RAM:
>
> ```
> // python3 mtp-bench.py
> code_python pred= 192 draft= 148 acc= 116 rate=0.784 tok/s=40.3
> code_cpp pred= 58 draft= 40 acc= 37 rate=0.925 tok/s=49.3
> explain_concept pred= 192 draft= 148 acc= 116 rate=0.784 tok/s=41.6
> summarize pred= 53 draft= 40 acc= 32 rate=0.800 tok/s=44.4
> qa_factual pred= 192 draft= 144 acc= 119 rate=0.826 tok/s=45.6
> translation pred= 22 draft= 16 acc= 13 rate=0.812 tok/s=40.8
> creative_short pred= 192 draft= 166 acc= 108 rate=0.651 tok/s=38.1
> stepwise_math pred= 192 draft= 138 acc= 122 rate=0.884 tok/s=46.7
> long_code_review pred= 192 draft= 146 acc= 118 rate=0.808 tok/s=43.5
>
> Aggregate: {
> "n_requests": 9,
> "total_predicted": 1285,
> "total_draft": 986,
> "total_draft_accepted": 781,
> "aggregate_accept_rate": 0.7921,
> "wall_s_total": 36.04
> }
> ```
>
> Also, I ran with `-fitt 1736`, since I use the 3060 as the primary GPU
>
> > **Resident\_Worker\_5807** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/olbjx38/) · 1 point
> >
> > Do you run on windows or other OS?
> >
> > > **ItsRektTime** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/old8ojr/) · 1 point
> > >
> > > It was on a WSL2 Debian distro
> **EmelineRawr** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okulx0s/) · 1 point
>
> Interesting, I also have a 4070 SUPER and was happy with a 40 tk/sec, I'll try your thing, thanks!!
> **OsmanthusBloom** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuryw6/) · 1 point
>
> Thanks a lot, this is inspiring! I'm trying to see if I can use MTP on my poor 3060 Laptop with just 6GB VRAM.
>
> One stupid question though: how did you get mtp-bench.py working with current llama-server? What command did you use to run it?
>
> For me it just gives 400 Bad Request errors regardless of how I try to run it. I suspect the problem is the call to "/completion" (I think it should be "/v1/completions"?)
>
> EDIT: Nevermind, I found the problem. I was using llama-server with --models-preset, as I'm used to. But apparently it doesn't provide the exact same API that way, so the mtp-bench.py didn't work. I switched to running llama-server with separate CLI options and now it works!
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuusg1/) · 1 point
> >
> > You shouldn't have to pass anything to [mtp-bench.py](http://mtp-bench.py/). It just starts, connects to your server and runs the benchmark.
> >
> > If you look at the end of [mtp-bench.py](http://mtp-bench.py/), you see the following line:
> >
> > ap.add\_argument("--url", default="[http://127.0.0.1:8080"](http://127.0.0.1:8080%22))
> >
> > If your server is not already running on [http://127.0.0.1:8080](http://127.0.0.1:8080/), you can either modify [mtp-bench.py](http://mtp-bench.py/) to match your server host/port, or change your server port to match [mtp-bench.py](http://mtp-bench.py/), and it should work 😄
> >
> > > **OsmanthusBloom** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okv5yci/) · 2 points
> > >
> > > Yeah. My problem was that I was using llama-server with the --models-preset option, which means it will run a proxy server on port 8080 and start separate workers for the requested model. In this mode the REST API is more limited and mtp-bench didn't work. As soon as I switched to the traditional CLI mode (lots of cli options) mtp-bench started working without any options.
> > >
> > > > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvex9r/) · 2 points
> > > >
> > > > Awesome! Glad you found the issue :)
> **Due\_Steak\_1249** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuvi3n/) · 1 point
>
> Have you observed any performance degradation as the context window reaches capacity? Historically, a 32k token limit appeared to be the optimal threshold for maintaining accuracy; for instance, Qwen3 reportedly showed a decline from 95% to 75% accuracy when scaling toward 128k.
>
> Conversely, some users suggest that operating significantly below the 128k mark may increase the model's susceptibility to repetitive loops. I am interested in the current state of the art regarding this architecture and your practical experiences using it. It appears that users are currently forced to balance significant trade-offs between context volume and output reliability.
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuzp56/) · 1 point
> >
> > I've coded quite a bit with Qwen3.6, not as much with MTP though. Did lots of code additions, debugging and refactoring on ~10,000 line projects. Never noticed any degradation at all. Context was filling up quite fast though, so at 128K, I had to work on specific parts to prevent constant compaction.
> >
> > Unfortunately, I realized Qwen3.6 cannot compete against larger models like GPT 5.5 for more demanding coding tasks, and often simply cannot produce any working code. But I still feel like Qwen is very capable for small projects where logic isn't pushed too far. I've had much more success with Qwen than Gemma 4.
> **leonbollerup** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okv3uij/) · 1 point
>
> Have you run any test to compare the quality against a “normal” model ?
> **Plastic\_Use\_4610** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okvybcf/) · 1 point
>
> Seems really high for the hardware - well done
> **the\_masel** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okwrdjr/) · 1 point
>
> Interesting, thank you. Did you compare it without MTP? With my 5060 Ti 16GB, I get around +15% tok/s and up to 66tok/s. Is this normal? (Tested on Windows 11)
> **Weird\_Night\_2176** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okxikcw/) · 1 point
>
> Been self-hosting AI for the past few months and finally got it to a point worth sharing. The stack:
>
> \- Jetson Orin Nano Super: CrewAI orchestration, 14 AI agents
>
> \- Orange Pi 5 Plus: Ollama model server
>
> \- Odroid XU4: PostgreSQL memory layer
>
> \- Jetson Nano 4GB: Tailscale mesh, network services
>
> Total monthly cost: $8 (electricity + Claude API for final decisions only) The agents run a paper trading desk, generate SEO content for a local business client, write YouTube scripts, and send me a morning briefing every day via WhatsApp. All local, all private, zero cloud dependency.
>
> Documenting the whole build on YouTube if anyone wants to follow along: [https://www.youtube.com/@BlackBoxAILab](https://www.youtube.com/@BlackBoxAILab)
>
> Happy to answer questions about the hardware setup or the agent architecture.
> **PeteInBrissie** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okyl6wr/) · 1 point
>
> I’ve done this today and for some reason OpenCode is looping weirdly compared to the non-MTP setup. If I work it out I’ll share here
>
> > **PeteInBrissie** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol2uksd/) · 1 point
> >
> > OK My setup is R5 5060G, 64GB RAM, RTX4060Ti. In OpenCode it was looping like mad until I set my context to 65576. Unfortunately OpenCode is also pushing 18,000 tokens at it which means an initial reaction time of about 3 minutes - after which it's really quick. Pretty sure I was seeing 90t/s at one stage last night.
> **Snoo40301** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol0a8un/) · 1 point
>
> Is this using the official llama.cpp or a fork for the MTP ?
> **zabadey** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol0z1qu/) · 1 point
>
> Sorry for my dumb question, but does it mean that I can also use it with my 16gb ram mbp m5?
> **trialbuterror** · [2026-05-10](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol3111i/) · 1 point
>
> Will this work for 9060xt 16gb 16gb ddr4 5600g processor ?
>
> How effective is coding softwares ?
> **Resident\_Worker\_5807** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol3xcn8/) · 1 point
>
> can i run it on Windows + Vulkan?
> gpu is 4070 12gvram
>
> 32g ram on DDR4
> **Loouiz** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol7g8pf/) · 1 point
>
> I've been running your config with a 16gb 4080 super, 7800x3d, 32gb ram. It is amazing, but I still get an occasional oom here and there. Any tips?
>
> > **janvitos** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol9vj65/) · 1 point
> >
> > Raise -fitt to something higher. Try 128 increments. If you're using 1536, try 1664 😄
> >
> > > **Loouiz** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ola4w66/) · 1 point
> > >
> > > Oh, I've also been trying to come up with a way to use a 1080 I have gathering dust, but couldn't come up with anything, mainly because of pascal architecture. My only goal is agentic coding. Any ideas or resources you recommend?
> **leonbollerup** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol95prh/) · 1 point
>
> Sadly.. the quality in the answer... goes to hell.. atleast in tests:
> \--
>
> This is the prompt:
> \---
> A city is planning to replace its diesel bus fleet with electric buses over the next 10 years. The city currently operates 120 buses, each driving an average of 220 km per day. A diesel bus consumes 0.38 liters of fuel per km, while an electric bus consumes 1.4 kWh per km.
>
> Instructions:
>
> 1. Verify your data
> 2. Use tables to represent data where you can
>
> Relevant data:
>
> \- Diesel emits 2.68 kg CO₂ per liter.
>
> \- Electricity grid emissions currently average 120 g CO₂ per kWh, but are expected to decrease by 5% per year due to renewable expansion.
>
> \- Each electric bus battery has a capacity of 420 kWh, but only 85% is usable to preserve battery life.
>
> \- Charging stations can deliver 150 kW, and buses are available for charging only 6 hours per night.
>
> \- The city's depot can support a maximum simultaneous charging load of 3.6 MW unless grid upgrades are made.
>
> \- Electric buses cost $720,000 each; diesel buses cost $310,000 each.
>
> \- Annual maintenance costs are $28,000 per diesel bus and $18,000 per electric bus.
>
> \- Diesel costs $1.65 per liter; electricity costs $0.14 per kWh.
>
> \- Bus batteries need replacement after 8 years at a cost of $140,000 per bus.
>
> \- Assume a discount rate of 6% annually.
>
> Tasks:
>
> 1. Determine whether the current charging infrastructure can support replacing all 120 buses with electric buses without changing schedules.
> 2. Calculate the annual CO₂ emissions for the diesel fleet today versus a fully electric fleet today.
> 3. Project cumulative CO₂ emissions for both fleets over 10 years, accounting for the electricity grid getting cleaner each year.
> 4. Compare the total cost of ownership over 10 years for keeping diesel buses versus switching all buses to electric, including purchase, fuel/energy, maintenance, and battery replacement, discounted to present value.
> 5. Recommend whether the city should electrify immediately, phase in gradually, or delay, and justify the answer using both operational and financial evidence.
> 6. Identify at least three assumptions in the model that could significantly change the conclusion.
>
> \---
>
> Result:
>
> [](https://preview.redd.it/4ln2q02iqk0h1.png?width=1670&format=png&auto=webp&s=c555458251bdbd0350e64243cd21cf90cf055b1a)
>
> > **leonbollerup** · [2026-05-11](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/ol95rey/) · 1 point
> >
> > [](https://preview.redd.it/j81vr2omqk0h1.png?width=1480&format=png&auto=webp&s=c9ca5c09f37801bfb24e544aaedd59cd5662c482)
> **Creative-Type9411** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/olaaeg0/) · 1 point
>
> the guide link is missing? for the "You can find a very nice guide on how to do that here and also download the..."??
> **mdda** · [2026-05-12](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/olcmss9/) · 1 point
>
> I've got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand rig (i7-6700 w 32 GB RAM + GTX 1080 w 8GB VRAM) : But I apparently I need >4 upvotes before I can post the story...
> **singlegpu** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktvr4i/) · 1 point
>
> Any recommendations on where to learn more about this parameters?
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okuqqre/) · 2 points
> >
> > Here you go: [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
> >
> > You can also ask any AI to explain them more in detail 😄 I got some pretty good answers from Gemini.
> **iamapizza** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktw993/) · 1 point
>
> Does it work if you use --fit, --fit-target, and --fit-ctx? Supposedly these args should be taking care of using as much vram as possible.
> **unrevealedpains** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktxo20/) · 1 point
>
> how would It run on my 4GB VRAM, RTX 3050? I know this might be a stupid question but I am new to all of this
>
> > **janvitos** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okunmnu/) · 1 point
> >
> > Not stupid at all! You should try it 😄 I'd be curious and happy to see the result!
> **IrisColt** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okux1rm/) · 1 point
>
> Thanks a lot!!!
> **evilbarron2** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/oktrfdy/) · 0 points
>
> Hmm. I get 100+ tok/sec (as measured by the llama-serve WebUI) with qwen3.6 35b A3b on my 3090 with my prompt.
> **WithoutReason1729** · [2026-05-09](https://reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/okucqby/) · \-1 points
>
> Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW)
>
> You've also been given a special flair for your contribution. We appreciate your post!
>
> *I am a bot and this action was performed automatically.*