Qwen4: The Architecture of the Future (Qwen3.8-Flash-Next)

On 2026-08-26 the Qwen team released an open-weight multimodal MoE model that it calls an "experimental preview of the architecture that will underpin Qwen4". This Russian-language Habr article, published 2026-08-30 by Master_AI on gptunnel's company blog, walks through the architecture, the benchmarks and a set of local runs habr-gptunnel-1076382. gptunnel resells model API access. The article's links to "Qwen3.8 Max", "Opus" and, in the last line, "Qwen4" all go to gptunnel's own model pages with utm_medium=seo, and the "Qwen4" link lands on the Qwen3.8-Max page because there is no Qwen4 to sell. Read it as marketing that happens to contain a technical summary.

The article gets one thing right early and in capitals: Qwen4 does not exist. There are no weights, no model card and no date, only rumours of a September 2026 window. The model is named inconsistently. The quoted announcement says "Qwen3.8-Flash", the rest of the article says Qwen3.8-Flash-Next, and the Unsloth page it links is titled "qwen3.8-next". It has 125 billion parameters, plus 51 billion more in a separate embedding layer, with 6 billion active per token.

The article sets it against a precedent: Qwen3-Next was an intermediate release that tried out the Gated DeltaNet plus gated attention hybrid, which then went into every model from Qwen3.5 to Qwen3.8 Max. Its dates for that precedent contradict each other, putting Qwen3-Next both "a year and a half ago" and alongside Qwen3.5 in February 2026. Its lineage runs from Tongyi Qianwen on a Llama architecture in April 2023, through Qwen3 in April 2025 (hybrid thinking, 235B-A22B trained on 36 trillion tokens in 119 languages), to Qwen3.8-Max on 2026-08-03 at 2.4 trillion parameters, which the article calls the second-largest open model after kimi-k3, and a dense Qwen3.8-27B under Apache 2.0 eleven days later.

Four changes

The official blog, as the article relays it, describes changes along four axes: attention, residual, embedding and optimization.

Attention is a hybrid. The 48 layers form 12 identical blocks, each with three Gated DeltaNet layers, which compress history into a fixed-size state (see kimi-delta-attention for the DeltaNet family), and one layer of full attention routed through a new Qwen Sparse Attention. Where DeepSeek's DSA scores every token, QSA's lightweight indexer compresses keys into micro-blocks with a factor of 4, keeps the top 512 blocks (about 2,051 logical token positions), and runs the final softmax over the original uncompressed keys and values. At 1M tokens of context, the article gives the QSA kernel up to 7.6× faster prefill and 4.9× faster decode, and, at a 90% cache hit rate, 8.6× the prefill throughput of Qwen3.7-Plus.

Gated Residual widens the residual stream into four parallel branches with an element-wise dynamic gate deciding how much each layer reads from and writes to each branch. One branch ended up as a highway from the first attention layer to most later layers without being designed to. The stream state can be stored in FP8.

The n-gram embedding looks at the current token plus a few previous ones and looks the combination up in a table of about 20 million bigram and trigram entries, crediting Per-Layer Embedding from Gemma 3n and DeepSeek's Engram. Because the lookup positions can be computed ahead of time, the table can live in host RAM and load asynchronously rather than occupy VRAM. The article quotes and then qualifies the slogan "works like 6B, knows like 180B": these parameters need almost no compute, but they still have to be stored somewhere, so what was saved is compute, not memory. That matches factual-capacity-scaling, where stored knowledge costs parameters however cheaply they are read.

Training switched from AdamW to Muon, with AdamW kept for embeddings and the MoE router. The team also found that batch-size warmup was no longer useful, costing 18.8% more optimizer steps for no gain in quality, so the final recipe starts at the target batch size.

Benchmarks

The official comparison is against Qwen3.7-Plus (May 2026), DeepSeek-V4-Flash-0731 (July 2026) and Claude Opus 4.6 at max effort (February 2026). Qwen picked the comparison set, and the Claude model in it was six months old at release.

Opus 4.6 keeps the lead on Humanity's Last Exam. On visual and agentic tasks the article gives Qwen 84.5 against 62 on AndroidWorld, 88.5 against 73.9 on RealWorldQA and 95.7 against 65.5 on MathVision. Artificial Analysis scores it 56 on its Intelligence Index against a class median of 29, and calls it very verbose: 200 million tokens to run the index against a median of 110 million. That verbosity shows up in any per-task cost comparison, such as the one in intelligence-vs-cost-linear.

Zhipu released GLM-5.3-Flash the same day. The article's comparison: 320B total and 18B active against Qwen's 125B/6B; native 1M context against 262K (1M with YaRN); DeepSWE 63.4 against 58.7; an input price of about $0.075 per million tokens on promotion against $0.16. GLM wins on DeepSWE and context, while Qwen's threefold smaller active parameter count is what lets it fit on a home machine.

The article also points to a Hugging Face discussion reporting that Russian-language quality regressed noticeably against earlier Qwen models, a problem it traces back to 3.5.

Running it locally

Unsloth's memory table, as quoted: 75 GB at 1-bit, 79 GB at 2-bit, 90 GB at 3-bit, 112 GB at 4-bit, 200 GB at 5-bit, 270 GB at 8-bit, 355 GB at BF16. The 1-bit build is far above the naive ~22 GB because the n-gram table is never quantized below 4 bits, being a precision-sensitive lookup. In that build the MoE experts take 37.11 GB (54.9%), the n-gram table 26.82 GB (39.7%), attention and QSA 1.71 GB, and everything else 1.91 GB, so nearly 40% of the file serves a single GGML_OP_GET_ROWS copy that does no arithmetic. See unsloth and llm-quantization for the quant families.

The article does not say who ran the following tests. On a Core Ultra 9 285K with 24 cores, 95.3 GB of RAM and no GPU, generation plateaued at about 11 tokens per second from 8 threads on, while prefill kept scaling to 53 t/s at 24 threads, showing decode bound by memory speed and prefill by compute. Adding one RTX 5090 (32 GB) raised generation to 52 t/s, by pinning the n-gram table to system RAM and moving expert layers onto the card step by step:

-ot per_layer_token_embd=CPU   # keep the n-gram table in host RAM
-ncmoe N                       # number of MoE layers whose experts stay on CPU

One group of layers too many and speed fell from 52 to 3.7 t/s with no error, because the driver spilled VRAM into system memory over PCIe. The article's rule: when a setting that should be faster is slower, suspect VRAM overflow first. This is the offload trade-off moe-cpu-offload and habr-local-llm-quantization-deep-dive describe, made worse by a failure that raises no error. Generation slowed by 15–17% at 32K context.

On wikitext-2 perplexity, the 1-bit Flash-Next build scored 4.01 against 5.68 for the 4-bit Qwen3.8-27B. That compares two different models, so it shows the ordering and not a like-for-like quantization loss. On 12 practical tasks, four of five builds scored a perfect 1.000 and the 1-bit 27B dropped to 0.309; asked who invented the transistor, that 27B build named three nonexistent scientists and got the date wrong by 22 years, while the 1-bit Flash-Next answered correctly.

Takeaway

Set the benchmarks aside and the article's point holds: the per-token cost of a model is being cut by architecture as well as by hardware, by splitting memory into cheap addressable storage and expensive compute, attending only to selected fragments, and widening the residual stream rather than deepening it. Whether Qwen4 uses this design is, for now, the Qwen team's stated intention and not a release. For how far local Qwen models are from frontier ones in practice, see local-ai-is-not-opus; for evidence about where Qwen3.8's reasoning traces come from, see qwen-gpt-reasoning-prefills; for the general split between total and active parameters, see mixture-of-experts.