CUDA vec FA kernel: turbo4 V-cache branch consumes 4 of 8 elements per iteration — half of every head's V output is zero on batch-1 decode (patch attached)

[morpheos-llc]

I've been using this turbo quant llama.cpp branch and generally loving it. I stumbled upon a bug and hopefully this investigation and patch is a meaningful contribution.

Summary

On the CUDA backend, -ctv turbo4 produces wrong attention output during single-token decode while producing correct output during batched prefill. Perplexity (batched) is fine, which is why this doesn't show up in PPL-based validation. It does show up in generation: greedy continuations loop and lose context, and agentic runs fail to terminate.

Measured with a teacher-forced top-1 agreement probe (fraction of decode steps whose greedy argmax matches an f16-KV reference, same prompt, same model):

model	attention	backend	prompt depth	K=q8_0 / V=turbo3	K=q8_0 / V=turbo4
Llama-3.2-3B-Instruct Q4_K_M	GQA 3:1, hd128	CUDA	2K	0.958	0.577
Llama-3.2-3B-Instruct Q4_K_M		CUDA	8K	0.987	0.689
Llama-3.2-3B-Instruct Q4_K_M		CUDA	45K	0.969	0.507
Llama-3.2-3B-Instruct Q4_K_M		CUDA	61K	0.963	0.415
Qwen3-4B Q8_0	GQA 4:1, hd128	CUDA	30K	0.987	0.681
Qwen3-4B Q8_0		Vulkan (Radeon 8060S)	30K	0.982	0.990
Qwen3.6-35B-A3B UD-Q4_K_M		Vulkan (Radeon 8060S)	45K	0.977	0.987
Qwen3.6-35B-A3B UD-Q4_K_M		Vulkan (Radeon 8060S)	61K	0.950	0.956
Qwen3.6-35B-A3B UD-Q4_K_M	GDN hybrid, GQA 8:1, hd256	CUDA	45K	0.974	0.822
Qwen3.6-35B-A3B UD-Q4_K_M		CUDA	61K	0.950	0.783
Nemotron-3-Nano-30B-A3B Q3_K_M	Mamba-2 hybrid, 6 attn layers	CUDA	45K	0.990	0.950
Nemotron-3-Nano-30B-A3B Q3_K_M		CUDA	61K	0.990	0.990

f16 scored against its own reference gives 0.990–1.000 in every case (determinism floor). The gap is not depth-dependent (present at 2K), not architecture-dependent, and not present on Vulkan. Nemotron is nearly immune, consistent with only 6 attention layers being exposed.

Environment

• Fork: TheTom/llama-cpp-turboquant @ 7ebcbb0b6 (b10513, merged 2026-08-17), TURBO4_USE_4BIT=1 (4-bit PolarQuant turbo4, no QJL code in the CUDA backend)
• CUDA build, GGML_CUDA_FA_ALL_QUANTS=OFF, CMAKE_CUDA_ARCHITECTURES includes 86
• GPU: RTX 3090 (Ampere, cc 8.6) over OCuLink; Vulkan comparison on AMD Radeon 8060S (Strix Halo, RADV)
• OS: Fedora 43, kernel 7.1.3; CUDA toolkit 13.1 (V13.1.115); NVIDIA driver 580.173.02
• Server flags: -fa on (the fork auto-enables FA for turbo types), -np 1, -c 8192 (short-depth runs) / -c 65536 (long-depth runs), -ctk q8_0 -ctv turbo4 (explicit; the auto-asymmetric rule also produces q8_0-K)

What is and isn't affected

• Prefill is correct. llama-perplexity at -c 65536 on Qwen3.6-35B-A3B, K=q8_0: f16-V 5.654, turbo3-V 5.685, turbo4-V 5.663. turbo4 is better than turbo3 on PPL, as expected from the format.
• Decode throughput is normal. llama-bench -d 65536: turbo4-V 82 t/s vs turbo3-V 74 t/s vs f16 93 t/s. Not a speed cliff.
• Write path is correct. A split probe writes all V rows via batched prefill, then reads them (a) through the prefill kernel and (b) through the batch-1 kernel. Same stored rows: (a) matches turbo3 (0.967 vs 0.967); (b) does not (0.672 vs 0.966).
• K=f16 / V=turbo4 also fails (0.28 distinct-4gram on greedy generation vs 0.68 for f16), so it is not the q8_0×turbo4 cross-type combination.
• turbo3-V is fine everywhere (0.95–0.99 agreement).
• Vulkan turbo4-V on AMD is fine (0.956–0.990 across three models, including Qwen3.6-35B-A3B at 61K).

Root cause (found with a standalone harness — details below)

ggml/src/ggml-cuda/fattn-vec.cuh, lines 573–606, the inline type_V == GGML_TYPE_TURBO4_0 V-accumulation branch of flash_attn_ext_vec (the #else float2-accumulator path, which is what CUDA takes since V_DOT2_F32_F16_AVAILABLE is HIP-only).

The branch is a copy of the turbo3 branch just above it. For turbo3/turbo2, line 125 pins V_rows_per_thread = 4; for turbo4 it falls through to 2*cpy_ne, which is 8 on Volta+ (ggml_cuda_get_max_cpy_bytes() == 16). The loop stride was widened accordingly, but the body still reads two qs bytes (4 nibbles) and writes two float2 accumulator slots per iteration — half of what each iteration owes. Elements i % 8 ∈ {4,5,6,7} of every V row are never read, the corresponding VKQ slots stay at their zero-init, and the writeback at lines 707–720 copies all four slots per iteration, so the zeros propagate into the attention output.

Net effect: on every batch-1 decode step, output dimensions d % 8 ∈ {4,5,6,7} of every attention head are forced to zero, while the softmax weights (K path) are computed correctly. That is why the failure is "degraded, not random" (0.42–0.82 agreement), depth-independent, architecture-independent, and worse on models with more attention layers.

Why perplexity never sees it: with a quantized K or V on Ampere, ggml_cuda_get_best_fattn_kernel routes Q->ne[1] == 1 to VEC and everything larger to MMA-f16, which converts V to f16 through convert.cu (dequantize_block_cuda<QK_TURBO4, QR_TURBO4, dequantize_turbo4_0>) — a correct path. Batched prefill never executes the defective branch; single-token decode always does. Ada/Blackwell send Q->ne[1] <= 2 to VEC as well.

Harness evidence (fork device functions #included and executed, not transcribed; 512 random rows, head dims 128 and 256, fixed seed):

path	mismatches vs CPU dequantize_row_turbo4_0
dequantize_turbo4_0 (dequantize.cuh:457, prefill conversion)	0 (bit-exact)
dequantize_V_turbo4_0<float, 2/4/8> (fattn-common.cuh:902)	0 (bit-exact)
live vec-kernel turbo4 V branch (fattn-vec.cuh:573–606, ncols=1, KQ=1)	50.000% — every mismatch is exactly 0.0, period-8 pattern at offsets 4–7, both head dims
same branch with the patch below	0 (bit-exact)

vec_dot_fattn_vec_KQ_turbo4_0 was read but not executed (K was q8_0 in every run); on inspection its striding covers D fully. turbo3/turbo2 bodies cover their 4 elements and are unaffected.

Patch

One hunk, ggml/src/ggml-cuda/fattn-vec.cuh: loop the existing body over V_rows_per_thread/4 byte-pairs and offset the VKQ slot index by 2*b. Patch and repro script: https://gist.github.com/morpheos-llc/349f1115bfa1e7bc14d2d2f392f26d46

End-to-end validation — same commit, patch applied in a separate worktree, CUDA-only build, RTX 3090, decode agreement vs f16 (teacher-forced greedy, K=q8_0):

model	depth	unpatched turbo4	patched turbo4	turbo3 (same binary)	f16 self-check
Llama-3.2-3B Q4_K_M	2K	0.546	0.979	0.958	1.000
Llama-3.2-3B Q4_K_M	45K	0.428	0.997	0.969	0.995
Qwen3.6-35B-A3B UD-Q4_K_M	45K	0.778	0.987	0.974	1.000

Patched turbo4 matches or beats turbo3 in every cell, which is what the format's bit budget predicts and what AMD-Vulkan turbo4 already showed on this build.

• Perplexity (Llama-3B, -c 2048 --chunks 8, device-pinned): f16 8.660; q8_0/turbo4 8.713 patched and 8.713 unpatched — byte-identical, confirming prefill never touched the defective branch.
• llama-bench q8_0/turbo4, Llama-3B: tg128 @ d0 254 vs 249 t/s (noise); tg128 @ d32768 90.9 patched vs 101.9 unpatched — an ~11% decode cost at depth, which is the price of reading the half of V the kernel was skipping.
• Greedy 512-token continuation of a 45K prefix on Qwen3.6: unpatched enters a verbatim repetition loop by ~token 60 and emits an invalid partial-UTF-8 token (the server then returns HTTP 500 "output does not match the expected Content-only format" — another symptom people may have seen); patched stays coherent for all 512 tokens.

Reproduction

Script: turbo4_decode_agreement.py in the gist above. It only needs a running llama-server and a text file (wiki.test.raw or any long plain text).

# 1. f16 reference
./llama-server -m Llama-3.2-3B-Instruct-Q4_K_M.gguf -ngl 99 -fa on -c 8192 -ctk f16 -ctv f16 --port 8081 &
python3 turbo4_decode_agreement.py --server http://127.0.0.1:8081 --text wiki.test.raw \
    --prompt-tokens 2000 --ref-tokens 384 --make-ref ref.json
# 2. under test
./llama-server -m Llama-3.2-3B-Instruct-Q4_K_M.gguf -ngl 99 -fa on -c 8192 -ctk q8_0 -ctv turbo4 --port 8081 &
python3 turbo4_decode_agreement.py --server http://127.0.0.1:8081 --ref ref.json
# expect ~0.58 on CUDA; ~0.96 with -ctv turbo3; ~1.00 with -ctv f16

A quicker qualitative check: greedy-continue a ~45K-token wiki prefix with -ctv turbo4 on CUDA. On Qwen3.6-35B-A3B I get a repetition loop within ~120 tokens, then a hard context reset (the model starts regenerating an article from 39K tokens earlier), and malformed tokens (@-$) that appear nowhere in the corpus. f16 and turbo3 continue coherently.

Separate observation (filing separately)

On the same RTX 3090 via the Vulkan backend, -ctv turbo3 and -ctv turbo4 both produce garbage in prefill and decode (PPL ~7000 for both; GGML_VK_DISABLE_COOPMAT2=1 fixes it). Different cause — the coopmat2 FA pipeline has no turbo dequant — so it gets its own issue; I'll link it here once filed.

How this was found

I was running an agentic coding benchmark (multi-turn tool use, 600s wall) across Qwen3.5/3.6 models served with turbo4 KV. Every turbo4-served model collapsed to ~0.1–0.3 pass rate with timeout-dominated failures, while models on f16 or MLA caches were fine. The completed runs had normal scores, so the models weren't wrong, they just didn't stop. A KV A/B (f16 vs turbo4, same context) isolated it, and the probes above narrowed it to the CUDA decode read path. Happy to share the full probe scripts and raw outputs.

Disclosure: the investigation, the dequant harness, and the patch were developed with Claude (Anthropic, Fable 5.1) working under my direction on my hardware; all measurements above are from real runs I can re-execute on request, and I've reviewed the patch and understand the defect (the turbo4 branch's loop stride is 8 elements but its body consumes 4). This is my first bug report here, so let me know if I need to adjust anything to match repo etiquette. Happy to restructure or run additional tests if that would help.

[morpheos-llc]

Fix branch with a single commit on top of 7ebcbb0b6: https://github.com/morpheos-llc/llama-cpp-turboquant/tree/fix/cuda-fattn-vec-turbo4-v-rows - Happy to open a PR if you'd like it that way.