Flash-Next on Strix Halo

Qwen3.8-Flash-Next is a hybrid: 48 layers, of which 36 are gated-delta-net recurrent layers and 12 are sparse-attention (QSA) layers that select the top 2048 KV cells per token with a learned indexer. It has a 512-expert MoE (10 active per token), 4-wide hyper-connections instead of a plain residual stream, per-layer embeddings, and a built-in MTP head for self-speculative decoding. Weights are UD-Q4_K_XL (~103 GB loaded).

Strix Halo pairs 40 RDNA3.5 CUs with 128 GB of unified LPDDR5X at ~256 GB/s. We run Vulkan/RADV rather than ROCm: on this APU Vulkan wins decode by 18–35%, and ROCm 10's gains target datacenter parts. Decode is memory-bandwidth-bound (the fork reaches ~80% of peak DRAM bandwidth); prefill is compute- and latency-bound, which is where most of the kernel work below lives.

For scale: mainline llama.cpp on the same machine decodes this model at ~25 t/s shallow, 15 t/s at 32k, and 8–9 t/s past 90k. An independent fork of the same lineage (apepojken/llama.cpp, lighter Q3 quant) reaches 50 t/s shallow but trails this branch at every prefill depth and at temp-1 decode; the two forks found the same bugs independently, which we treated as mutual validation.

Prefill — cumulative average

Prefill — marginal window rate

The rate over just the window between adjacent rungs — the speed you actually get at that depth. Both tiers floor near ~185 t/s through 128–192k: the sparse-attention selection caps how much of the KV each token attends, so the per-token cost stops growing instead of decaying toward zero.

Decode

The two tiers are identical within noise — decode never sees the batch size. The 8k–98k plateau is draft acceptance holding; the wobble (±15%) tracks content draftability.

Prefill vs the base model (both UB=2048)

Two ceilings, precisely mapped

• The base model's ub2048 ceiling is memory: host OOM at ~192k of fill (see above).
• IQ4_XS's ub2048 ceiling is a GPU watchdog, not memory — and it's now root-caused: with RAM to spare, prefill deterministically dies at ~227k with an amdgpu ring timeout, reproduced across runs. The cause is a submission-concurrency hazard, not a single slow kernel: forcing one submission in flight at a time (GGML_VK_SERIALIZE_SUBMISSIONS=1) makes the exact killer config complete a full 200k prefill+decode cleanly, where the overlapped run wedges the ring at ~194k. It fits the signaled N / emitted N+2 dmesg signature and explains why UB=1024 survives with full overlap (its smaller submissions stay under the threshold). Resolved by the guard, which is already optimal: its automatic UB=1024 step above 192k is both safe and faster than a serialized-ub2048 workaround — measured 209 t/s (ub1024) vs 160 t/s (serialized ub2048) at 200k. So ub2048 at full depth is a losing trade even when made to work; the fast path was already the one shipping. (Two earlier theories — an oversized dispatch, then bounding in-flight submissions to two — were both disproven on the box.)
• Spending the headroom doesn't buy prefill: A/B-tested moving the 27.5 GB per-layer-embedding table back to the GPU (a wash — and the placement flag proved inert: the tensor stays in an anonymous host buffer either way) and UB=3072 (flat to slightly negative). The headroom's real value is fill safety and zero swap. Also settled an old mystery: the "cold PLE" prefill tax on the base model was anonymous-page swap under RAM pressure, not page-cache misses — on this quant it simply doesn't exist.
• Template quirk: the finetune's chat template enforces reasoning efforts xhigh|medium|low (xhigh is its default) and rejects high; the launcher remaps automatically.

Full-context (262k) enablementprefilldecode

The interesting discovery: at this scale the wall is host RAM, not GPU memory — on a UMA APU the GTT is host RAM, so the "free VRAM" the tools report past ~120 GB doesn't physically exist. And the wall moves during use: KV pages acquire physical backing as context fills, so a configuration that boots cleanly can still be OOM-killed twenty minutes into a deep prefill. Boot-fit ≠ fill-fit.

• Target KV at q8_0. Quantizing the 12 attention layers' KV (6144→3264 MiB) plus the indexer cache (2304→1224 MiB) frees ~3.9 GB of host RAM for a measured <1% throughput difference, shallow and deep — 36 of 48 layers are recurrent and never touch it. KV=q8_0
• The fill ceiling, measured. With the speculative draft loaded, UB=2048 survives to ~192k of processed context and then the kernel OOM-killer fires mid-prefill. UB=1024 returns ~4 GB of compute buffers and completes the full 252k fill with 3.1 GB to spare — 209 t/s average prefill (20.6 min), 28.9 t/s decode at full depth.
• Automatic tiering. The launcher's deep-context guard applies all of it: above 96k it swaps to the Q4 draft (the Q8 draft OOMs host RAM past ~120k), above 128k it sets KV=q8_0, above 192k it steps to UB=1024 so a full-depth fill survives. Every step prints what it did and every step can be overridden. DEEPMTP=0

Speculative decoding (MTP)decode

• MTP draft head. Ported the model's next-token-prediction block as a llama.cpp draft model. +72% decode at temp 0 shallow, growing to +155% at 128k, because base decode collapses with depth and drafted decode doesn't.
• Stochastic speculative sampling. Exact-match verification makes speculation a wash at temp > 0, so the draft now samples with the target's sampling parameters and the server verifies by rejection sampling — lossless for the target distribution. Temp-1 chat: 37.9 t/s vs ~26 without (+46%). SPEC_STOCH=0
• Draft micro-batch decoupling. The draft used to inherit the target's ubatch 2048 and duplicate its scaled buffers, which walled prefill at ~82k. --spec-draft-ubatch 512 removes the wall and gains +11–13% prefill with MTP on.
• Partial recurrent rollback. Rejected draft tokens used to restore a 112 MiB checkpoint; now the recurrent state rolls back in place. +19% decode at temp 1. The first version corrupted the conv state by snapshotting only plane 0 — fixed, and acceptance matches the checkpoint path exactly. RSROLL=0
• Draft depth by temperature. Draft length 6 wins at temp 0, 4 wins under sampling (deep sampled drafts diverge and rejections discard more). The launcher switches on TEMP.

Sparse attention (QSA)prefilldecode

• Block-level indexer top-k. Selecting whole indexer blocks instead of expanding scores to per-cell resolution deletes a ~2 GB score buffer and, incidentally, a real bug: the per-cell sort exceeded Vulkan's k < 1024 top-k limit, so every step past ~1k context silently sorted on the CPU — about 14k dispatches per run, now zero. pp +25%, tg +11% at 32–64k. BLKTOPK=0
• Gathered sparse prefill. At depth, the QSA layers streamed the whole KV cache through flash attention only to mask most of it. Tokens near each other select mostly the same cells (a 128-token window's union covers 30–50% of the cache), so three small custom ops deduplicate each window's selections, gather those K/V rows and mask columns, and run standard flash attention on the gathered subset. Mathematically equal to the masked-dense path. +6% marginal prefill at 32–49k, +10% at 49–63k. GP=0
• Pooled-key summary cache. One f32 summary row per position block feeds the indexer instead of recomputing pooled keys per step; writes are deferred so prefill never pays for them. tg +6–9% at 32–41k, byte-identical. POOLED=0

Expert GEMMs (MoE)prefill

The 512-expert matmuls are ~34% of prefill and the hardest problem on this GPU. What the measurements established: they are not bandwidth-bound (tiles stream 67–110 GB/s of the available 256) and not dequant-bound (trivial-dequant q5_1 ties heavy-dequant q4_K at ~9.5 TFLOPS). The limiter is latency structure — the indirect row-ID gather plus ~40 rows per expert, too little work per tile to hide memory latency. Wins here are correspondingly modest and additive:

• Row-list prepass + compact dispatch. A GPU prepass builds per-expert row lists once, so tile workgroups read their rows directly instead of re-scanning the ID tensor, and only live (expert, tile) pairs launch — the old grid was ~97% dead workgroups. +1% (dead launches turn out to be nearly free on RDNA3.5). Upstream later built the same prepass independently (#26686); both paths now coexist. GGML_VK_MMID_COMPACT=0
• 48-wide tile. BN=64 padded ~38% of lanes at 40 rows/expert; a 3-fragment 48-wide warp grid pads 17%. +1.4%, byte-identical. GGML_VK_MMID_BN48=0
• Per-expert tile selection. Pick the tile by expected rows per expert, not batch width, so decode and speculative-verify batches use the small tile that suits them. Also covers routing-weight fusion into the tile store (+2.9% prefill) and f16 activation staging.

Kernel fusionprefilldecode

• Matmul epilogue fusion. The scale → sigmoid/silu → scale → multiply chains that follow the wide dense GEMMs (hyper-connection gates, attention out-gate) now run in f32 on the accumulators inside the matmul store instead of as separate passes. +3.3% prefill total; the fused multiply is bit-exact. GGML_VK_MM_TILE_EPILOG=0
• Fused hyper-connection ops. The combine step (residual + gated block output) is one custom op instead of a repeat/mul/add chain, and the mix tail (sigmoid, gate, mean-collapse over 4 streams) is another — that one replaces ~7 dispatches per call, 96 calls per decoded token. +0.5–0.9% decode. Q4X_HC_MIX_NOFUSE=1
• Transposed inject projection. The hyper-connection inject matmul has M=4 — pathological for a tiled kernel, and it cost ~16% of prefill. Computing it transposed dispatches the efficient GEMV path instead; the op now costs 1.7%.
• Contiguous conv concat + batch 2048. A one-line ggml_cont on the delta-net conv input (+3% shallow), and raising the micro-batch to 2048 (+25% prefill), the single best config flip.

Correctness fixesbugs

• Vulkan queue deadlock. Any device-side wait on the shared queue could be enqueued ahead of its signal and wedge the GPU under MTP's small batch shapes. Cross-context waits now happen on the host via vkWaitSemaphores, so the queue always drains.
• Speculative replay livelock. With non-bit-reproducible redecode, a rejected draft could re-reject forever at deterministic slot-reuse boundaries. The server forces one plain decode step after three same-position restores, which makes livelock impossible regardless of numerics.
• Shared-memory scrambling at BK > 32. The q4_0/q4_1/q5_0/q5_1 matmul loaders hard-coded the split-nibble +8 offset for a 32-wide K block; any deeper K block makes two thread groups race on the same shared slots. Found while testing BK=64 (which is a loss anyway); the block-relative fix is bit-identical at BK=32 and defuses the trap for future tile work. Likely relevant upstream.
• Context-shift crash with unquantized K cache (upstream fix, merged): the Hadamard rotation matrix was copied to a tensor that may have no buffer.

Upstream sync

The branch tracks ggml-org master by merge; the latest sync brings in 54 commits including upstream's own Flash-Next work (radix top-k for k ≥ 1024, mat-vec tuning for Strix Halo, fewer graph splits in qwen4exp) alongside this fork's machinery. Where the two overlap — upstream's row-ID hoisting is nearly the same prepass as ours — both paths coexist and the fork's validated defaults stay in charge.

Validated on-box against the pre-merge build with the identical bench script: recall, stability, and draft acceptance are unchanged (needles 3/3, harness 10/10, acceptance 0.871 vs 0.873); prefill is equal or better through 32k; shallow decode gains +6.7% (upstream's RDNA3 mat-vec dispatch wins on this APU too — GGML_VK_RDNA3_MMV_TUNE=0 reverts). One residual: decode at 32k reads −6.6%, of which ~2 points trace to a slower single-token flash-attention dispatch (222 → 257 µs at 33k KV, same shape). The working branch runs the fully-validated pre-merge state; the merged tree is preserved on flashnext-v2-merged with the depth-decode residual as the open item before re-adoption.

Tooling, packaging & diagnosticsinfra

• Full context is the default. The launcher now serves the whole 262k context out of the box; the deep-context guard makes the maximum safe automatically (Q4 draft, KV=q8_0, UB=1024 at depth). CTX=131072 narrows it.
• One reproducible toolbox image. A single CI-built container — ghcr.io/wdenejko/llama.cpp/llama-vulkan-wdenejko — pins the exact environment these numbers were measured in (Fedora 43, gcc 15.3.1, Mesa 25.3.6 RADV, znver5). Its binaries serve at parity with the box build (596.8 vs 596.5 t/s at 2k, decode identical), so what the community pulls is what the box runs.
• Launcher, unbloated. run-flashnext.sh refactored 277 → 193 lines with the behavior proven byte-identical (comment-stripped, continuation-joined token streams diff empty).
• Job-time telemetry that survives a hung graph. GGML_VK_JOB_LOG=1 reports per-submission GPU time; GGML_VK_SUBMIT_TRACE=1 prints each submission's ops as it is dispatched, flushed — the one instrument that can name the culprit when a graph never returns (timestamp read-back can't). This is what root-caused the ub2048 ring hang as a concurrency hazard rather than a slow kernel.
• Uncensored IQ4_XS template fix. The finetune's chat template rejected mid-conversation system messages, which agent harnesses (e.g. opencode) send routinely — every such request 500'd. A relaxed copy renders them in place, byte-identical on normal inputs. TPLFIX=0 reverts.

In progress

• Agentic coding eval (pi-bench). Throughput is only half the story — this measures the served model as an actual coding agent on SWE-bench-style tasks (clone a repo, fix a real bug, run its test suite), scored by an independent cloud LLM judge. Agent runs on the local model, judge in the cloud so it never contends for the single serving slot. Results to be added here.
• IQ4_XS depth-tax profile. The i-quant's prefill trails the base quant by an amount that grows with depth (−5% shallow, parity to 96k, then −21% at 160k / −29% at 192k) while decode stays at parity — backwards from what weight-dequant cost predicts, so something KV/attention-adjacent responds to the quant. One GGML_VK_PERF_LOGGER session at 128–192k names the dispatch; then it's a targeted i-quant kernel fix or a documented envelope.

Open perf levers

• Expert-GEMM latency structure. The MoE matmuls are the hardest problem on this GPU (~34% of prefill, latency-bound not bandwidth-bound). Ranked next attempts: mul_mat_id BK64, epilogue-fusion phase B (fuse the MUL-by-tensor routing weight into the tile store), and flash-attention-at-depth tuning. Each is expected to be small and additive, in keeping with everything shipped in this area.
• ub2048 concurrent at depth (low priority). The ub2048 ring hang is resolved — the guard's UB=1024 is safe and already faster at depth. The only remaining upside is hunting the specific unsynchronized buffer so ub2048 could run with full overlap instead of falling back, worth ~5% deep-prefill at best and uncertain kernel-debugging work. Parked behind the profiling above.

Before upstream re-adoption

The ggml-org merge is built and validated on flashnext-v2-merged (correctness clean, acceptance matched). The one open item blocking re-adoption on the working branch is the −6.6% decode-at-32k residual, ~2 points of which trace to a slower single-token flash-attention dispatch at 33k KV — chase that kernel, then re-merge.

• Marginal windows, not point probes. A single 2048-token probe at depth understates both legs and can swallow a real 10% difference — the gathered-prefill win only appeared in ≥6k-token windows. The chain's own per-task rates are free; read them.
• Engagement evidence before belief. Every knob prints a one-shot line or is probed; several early "results" were legs where the environment never reached the binary (containers don't forward env, and one launcher was a stale copy for half a day).
• Unexpected output differences are bug reports. Changes that should be bit-identical get compared byte-for-byte; when BK=64 "won" but outputs drifted, the drift was the bug. Quality gates are cloze needles at 32k/64k plus a 10-task code harness, run before any default flips.
• Boot-fit ≠ fill-fit. On unified memory, KV pages acquire physical backing as context fills; a server that boots with gigabytes "free" can be OOM-killed twenty minutes into a deep prefill. Fill validation — an actual full-depth prefill — is part of every deep-context default now.
• Deep-context failures that look like model damage are often state accumulation. A per-token FFN error too small to affect short outputs compounds through the recurrent layers over 32k tokens into retrieval collapse with perfectly coherent shallow behavior.
• GPU-memory contention hides from the obvious counter. This APU exposes only 2 GB of VRAM; a model lives entirely in GTT (amdgpu.gttsize ≈ 124 GB). A competing ROCm/HIP job allocates through KFD, which does not register in mem_info_gtt_used — so a gtt=0 reading is not proof the box is clear. An 87 GB model that loads fine solo dies at load under contention with radv: Not enough memory for command submission → device-lost — a pin-budget failure, not host RAM. Clear the box (stop other GPU users, not just the systemd services) or offload before trusting a large-model boot.

Everything is driven by scripts/run/run-flashnext.sh, which starts llama-server with the validated defaults (ubatch 2048, coopmat on, block top-k, gathered prefill, pooled cache, fusion) and forwards the revert knobs listed above. Recommended sampling: code = temp 0, penalties off, thinking off (MTP=1 TEMP=0 REASONING=none); chat = temp 1 with stochastic speculation (auto-on).

# chat / creative (temp 1, stochastic speculation auto-on with MTP=1)
MTP=1 ./run-flashnext.sh

# fastest deterministic code generation
MTP=1 TEMP=0 REASONING=none ./run-flashnext.sh

# full 262k context — the deep-context guard handles the rest automatically:
#   >96k: Q4 draft · >128k: KV=q8_0 · >192k: UB=1024 (so a full fill survives)
MTP=1 CTX=262144 ./run-flashnext.sh

# pin the fast batch tier if your jobs stay under ~190k of fill
MTP=1 CTX=262144 UB=2048 ./run-flashnext.sh

Source: wdenejko/llama.cpp · master. Or skip the build — the prebuilt toolbox image is the exact environment these numbers were measured in (Fedora 43, Vulkan/RADV, znver5):

# one command, then run the launcher from inside or via toolbox run
toolbox create llama-vulkan-wdenejko --image ghcr.io/wdenejko/llama.cpp/llama-vulkan-wdenejko:latest