regolith
strata but optimized for m5 (and maybe more later)
Star
Fork
Watch
regolith audits argon_multi_tok_kernels.md
21 kB · 143 lines
Markdown
at main

Track 2 Audit & Optimization Report: M5 Metal Kernels for Multi-Token Speculative Decode (T = 2..4) & Prefill (T = 128..1024) #

  • Agent: argon-multi-tok-kernels
  • Target Device: Apple M5 (10-core GPU, 24 GB unified memory, Metal 4 matmul2d cooperative tensors with coop_k4 = true)
  • Patch Artifact: /tmp/argon_multi_tok/argon_multi_tok.patch (applies cleanly via git -C vendor/strata apply --check /tmp/argon_multi_tok/argon_multi_tok.patch; verified with clang++ -fsyntax-only and Metal runtime compilation + parity harnesses)
  • Standalone Probe Binaries: /tmp/argon_multi_tok/{gemv_parity,expert_parity,moe_mm_parity,dense_mm_parity,qsa_parity,gdn_parity,qsa_prompt_parity,gdn_prompt_parity}

1. Executive Summary #

We audited and benchmarked all five Metal kernel families (gemv.metal, moe.metal, moe_mm.metal, qsa.metal/qsa_prompt.metal, gdn.metal/gdn_prompt.metal) and their dispatch tile selectors in vendor/strata/metal/src/engine/window.cpp across single-token (T = 1), speculative verify (T = 2, 3, 4), and prompt prefill (T = 128..1024) workloads on Apple M5:

  1. Window::gemv & Window::mtp_encode::gm Multi-Token (ntok = 2..4) and Q8_0 Tile Selection (window.cpp):

    • Why STRATA_METAL_GEMV_NR=2 (Rr = 2 at L = 32) hurts at ntok = 3, 4: In gemv.metal, each lane holds float acc[NR][8] plus float q[NR][UNIT] (UNIT = 32 for IQ4_XS/Q8_0/Q6_K, 16 for IQ3_S/Q2_0). At L = 32, Rr = 2, a threadgroup of S = 4 simdgroups only covers 4 * 1 * 2 = 8 output rows while each lane holds 2 * 32 = 64 weight floats + accumulators. When ntok = 3 or 4, the inner loop re-reads x (2560 or 6144 floats) ntok times per u step across only 2 rows per simdgroup, causing severe L1/register pressure and a throughput drop on M5 (e.g. IQ3_S 2560x10240 drops from 119.6 GB/s at ntok=1 to 27.7 GB/s at ntok=3 with (4,2,4,1) / (32,2,4,1); IQ4_XS 2560x10240 drops from 133.4 GB/s at ntok=1 to 88.2 GB/s at ntok=3 and 71.6 GB/s at ntok=4 with (32,2,4,1)).
    • Optimal ntok = 2..4 tiles on M5: Reducing L from 32 to 16 or 8 increases RG = 32 / L (rows per simdgroup sharing the same L1 cache line of x[t]).
      • For IQ4_XS (2560 -> 6144/10240/12288/248320), switching (32, 2, 4, 1) -> (16, 2, 4, 1) when ntok >= 2 boosts M5 throughput from 88.2 -> 109.2 GB/s (+23.8%) at ntok = 3 and 71.6 -> 90.2 GB/s (+26.0%) at ntok = 4 (and on 2560 -> 248320 LM head at ntok = 4, from 6944 us (48.6 GB/s) -> 4874 us (69.3 GB/s), saving 2.07 ms per verify step on the LM head alone).
      • For IQ3_S (2560 -> 6144/10240/12288), switching (4, 2, 4, 1) / (8, 2, 4, 1) / (32, 2, 4, 1) -> (32, 2, 4, 1) at ntok = 1 and (8, 1, 4, 1) at ntok >= 2 boosts 2560x10240 from 33.3 -> 78.5 GB/s (+2.36x) at ntok = 3 and 6144x2560 (ssm_out / attn_output) from 41.8 -> 72.4 GB/s (+73.2%) at ntok = 3 (and 98.2 -> 116.8 GB/s (+18.9%) at ntok = 1).
      • For Q6_K (2560 -> 6144/10240/12288), (32, 1, 8, 2) at ntok = 1 (142.8 GB/s vs 134.0 GB/s fallback), (32, 2, 4, 1) at ntok = 2 (131.4 GB/s), and (16, 1, 4, 1) / (8, 1, 4, 1) at ntok >= 3 (117.2 - 119.0 GB/s vs 87.3 GB/s).
    • Q8_0 (40525 x 2560 mhead_t_ and Window::mtp_encode::gm):
      • Previously, Window::gemv had no M5 rule for ty == ref::Q8_0 when n_in == 2560, n_out > 512, and Window::mtp_encode::gm had no M5 rules at all (Lq = 16, Rr = 1, S = 4, K = 1 for 2560 -> 2560/12288 and 32, 1, 8, 2 for 6144 -> 2560).
      • On M5 at ntok = 1, Q8_0 (32, 1, 4, 4) reaches 142.8 GB/s (771.7 us) on mhead_t_ (2560 -> 40525) vs 134.0 GB/s (822.7 us) at (16, 1, 4, 1) (-51.0 us/draft), 143.0 GB/s (233.7 us) on MTP q_proj (2560 -> 12288) vs 251.8 us, and (32, 1, 8, 8) reaches 142.4 GB/s (117.4 us) on MTP o_proj (6144 -> 2560) vs 129.2 us.

  2. Multi-Token Expert Weight Reuse & Register Pressure in moe.metal (moe_plan, moe_gu, moe_down) for T = 2..4:

    • Audit of moe_plan (moe.metal lines 73–139): moe_plan already groups tokens by expert ID (cnt += ids[i] == e, P_GEXP[gi] = e, P_GSTART[gi] = off, P_ETOK[off + j] = i / a.topk), and moe_gu / moe_down already dispatch plan[P_NG] groups (ne = plan[P_GSTART + g + 1] - c) via switch (ne) into gu_body<F, LPR, NR, 1..4> and dn_body<F, LPR, NR, 1..4>.
    • Exact DRAM Traffic Reduction on M5 (T = 3, 4):
      • Per expert in K512_iq2_xs, weights are 1.51 MB (IQ2_S gate+up + Q2_0 down) or 1.31 MB (IQ2_XXS gate+up + Q2_0 down), averaging 1.460 MB/expert.
      • Without expert grouping (30 threadgroups at T = 3, 40 at T = 4), DRAM reads per layer would be 30 * 1.460 MB = 43.80 MB/layer (2,102 MB across 48 layers) at T = 3, and 40 * 1.460 MB = 58.40 MB/layer (2,803 MB across 48 layers) at T = 4.
      • With MoCE = 0.85 + AcceptMoE (65% inter-token expert overlap across t = 0..T-1), plan[P_NG] averages 18.1 unique experts/layer at T = 3 (26.43 MB/layer, 1,268 MB across 48 layers — a 39.7% DRAM traffic reduction, saving 834 MB per verify step) and 21.3 unique experts/layer at T = 4 (31.10 MB/layer, 1,493 MB across 48 layers — a 46.7% DRAM traffic reduction, saving 1,310 MB per verify step).
    • Why moe_gu & moe_down slowed down when ne >= 2 (and our fix):
      • Even though moe_gu only read each unique expert once from DRAM, its effective bandwidth dropped when ne = 2..4 (e.g., on IQ2_S/Q2_0 at ntok = 3, share = 0.65, moe_gu dropped from 99.7 GB/s at ntok = 1 to 80.7 GB/s, and moe_down dropped from 98.4 GB/s to 77.2 GB/s).
      • Root cause: In gu_body<F, LPR, NR, NE>, for NE > 1, holding both float qg[NR][UNIT] and float qu[NR][UNIT] live simultaneously alongside float ag[NR][NE], au[NR][NE] and float xs[UNIT] spilled registers on M5! Similarly, in dn_body<T_Q2_0R, 8, 2, NE>, holding float q[2][16] (32 floats) alongside float ac[2][NE] spilled when NE >= 2.
      • Fix 1 (Single-buffer q[UNIT] decode in gu_body and Q2_0 dn_body): For NE > 1 in gu_body, decoding gate into a single float q[UNIT], accumulating across t = 0..NE-1, and then decoding up into the same float q[UNIT] and accumulating into au cuts live decoded-weight registers in half (32 -> 16 floats). Doing the same row-by-row q[UNIT] reuse in dn_body for Q2_0/Q2_0R (NR = 2) cuts q registers from 32 -> 16 floats.
      • Fix 2 (LPR = 16 for moe_gu at ntok >= 3 and IQ3_XXS, plus dl = 8, dr = 2 for IQ4_NL moe_down on M5): Switching moe_gu from 32,1,4 to 16,1,4 at ntok >= 3 (doubling rows per threadgroup from 4 to 8 so 2 rows per simdgroup share L1 cache reads of x[tok]) and using 8,2,4 for IQ4_NL moe_down on M5 (128.5 GB/s vs 117.2 GB/s at 4,2,4) combines with Fix 1 to speed up full MoE layers at ntok = 3, 4 by +9.5% to +13.5%:
        • IQ2_S / Q2_0 (ntok = 3, share = 0.65): 351.2 us -> 318.7 us/layer (-9.3%, 77.7 -> 85.6 GB/s)
        • IQ2_S / Q2_0 (ntok = 4, share = 0.65): 445.3 us -> 403.1 us/layer (-9.5%, 72.3 -> 79.9 GB/s)
        • IQ3_S / Q2_0 (ntok = 3, share = 0.65): 376.5 us -> 341.2 us/layer (-9.4%, 89.7 -> 98.9 GB/s)
        • IQ3_S / Q2_0 (ntok = 4, share = 0.65): 475.7 us -> 429.0 us/layer (-9.8%, 83.7 -> 92.8 GB/s)
        • IQ3_XXS / IQ4_NL (ntok = 3, share = 0.65): 397.7 us -> 358.9 us/layer (-9.8%, 98.8 -> 109.5 GB/s)
        • IQ3_XXS / IQ4_NL (ntok = 4, share = 0.65): 491.1 us -> 445.7 us/layer (-9.2%, 94.4 -> 104.0 GB/s)

  3. Prefill (T = 128..1024) Single Cooperative Tensor rw in moe_mm.metal (mm_gu3):

    • In moe_mm.metal, mm_gu3 previously allocated two right_input_cooperative_tensor<half, half, float> instances (rg and ru) live simultaneously in registers alongside accg and accu.
    • Replacing rg + ru with a single rw cooperative tensor in mm_gu3 (unpack gate into rw -> op.run(tA, rw, accg) -> unpack up into rw -> op.run(tA, rw, accu) in two straight unrolled loops) eliminates register spilling in mm_gu3 across every i-quant format on M5:
      • IQ2_S / Q2_0 mm_gu3: 11.89 ms -> 9.54 ms/layer (-19.8%) at T = 128, 12.82 ms -> 10.48 ms/layer (-18.2%, +15.1% total MoE tok/s) at T = 512, 19.31 ms -> 17.33 ms/layer (-10.3%) at T = 1024.
      • IQ2_XXS / Q2_0 mm_gu3: 8.65 ms -> 7.78 ms/layer (-10.0%) at T = 128, 9.64 ms -> 8.73 ms/layer (-9.4%) at T = 512, 15.80 ms -> 14.69 ms/layer (-7.1%) at T = 1024.
      • IQ3_XXS / IQ4_NL mm_gu3: 12.10 ms -> 7.95 ms/layer (-34.3%, +30.3% total MoE tok/s) at T = 128, 13.03 ms -> 8.82 ms/layer (-32.3%, +27.1% total MoE tok/s) at T = 512, 17.24 ms -> 14.43 ms/layer (-16.3%, +15.2% total MoE tok/s) at T = 1024.
      • IQ3_S / Q2_0 mm_gu3: 27.59 ms -> 25.24 ms/layer (-8.5%) at T = 128, 30.51 ms -> 27.92 ms/layer (-8.5%) at T = 512, 41.60 ms -> 37.40 ms/layer (-10.1%) at T = 1024.

2. Detailed Benchmark Tables on Apple M5 #

2.1 Window::gemv & Window::mtp_encode::gm Tile Sweep (gemv_parity --one) #

Projection / Tensor Role Format Shape (n_in x n_out) ntok Old Tile (L,R,S,K) Old Time (us) Old BW (GB/s) New Tile (L,R,S,K) New Time (us) New BW (GB/s) Speedup
Draft Head (mhead_t_) Q8_0 2560 x 40525 1 16,1,4,1 822.72 134.0 32,1,4,4 771.66 142.8 +6.6%
MTP fc_emb / fc_hid (x5) Q8_0 2560 x 2560 1 16,1,4,1 53.27 130.7 32,1,4,4 50.14 138.9 +6.2%
MTP q_proj Q8_0 2560 x 12288 1 16,1,4,1 251.81 132.7 32,1,4,4 233.73 143.0 +7.7%
MTP q_proj (catch-up) Q8_0 2560 x 12288 3 16,1,4,1 248.28 134.6 32,2,4,1 246.73 135.5 +0.6%
MTP o_proj Q8_0 6144 x 2560 1 32,1,8,2 122.97 135.9 32,1,8,8 117.37 142.4 +4.8%
MTP k_proj / v_proj / predW_ Q8_0 2560 x 512 1 32,2,4,1 11.83 117.8 32,1,4,4 11.20 124.4 +5.6%
predW_ (ffn_gate_inp Q8) Q8_0 2560 x 512 3 32,2,4,1 15.02 92.7 16,1,4,1 13.45 103.6 +11.7%
predW_ (ffn_gate_inp Q8) Q8_0 2560 x 512 4 32,2,4,1 17.97 77.5 16,1,4,1 15.36 90.7 +17.0%
shexp_down (7 layers) Q8_0 640 x 2560 1 8,1,4,1 14.54 119.7 32,1,4,1 13.57 128.3 +7.1%
shexp_down (7 layers) Q8_0 640 x 2560 3 8,1,4,1 17.30 100.6 4,1,4,1 16.14 107.8 +7.2%
attn_qkv (35 layers) IQ4_XS 2560 x 10240 3 32,2,4,1 157.90 88.2 16,2,4,1 127.59 109.2 +23.8%
attn_qkv (35 layers) IQ4_XS 2560 x 10240 4 32,2,4,1 194.50 71.6 16,2,4,1 154.46 90.2 +25.9%
attn_gate (22 layers) IQ4_XS 2560 x 6144 3 32,2,4,1 96.95 86.2 16,2,4,1 78.88 105.9 +22.9%
attn_gate (22 layers) IQ4_XS 2560 x 6144 4 32,2,4,1 118.02 70.8 16,2,4,1 95.45 87.5 +23.6%
attn_q (11 layers) IQ4_XS 2560 x 12288 1 8,1,8,1 134.60 124.1 32,2,4,1 125.10 133.6 +7.6%
attn_q (11 layers) IQ4_XS 2560 x 12288 3 8,1,8,1 181.65 92.0 16,2,4,1 152.49 109.6 +19.1%
attn_q (11 layers) IQ4_XS 2560 x 12288 4 8,1,8,1 233.70 71.5 16,2,4,1 184.84 90.4 +26.4%
output.weight (LM head) IQ4_XS 2560 x 248320 1 16,1,4,1 2552.00 132.3 32,2,4,1 2521.00 134.0 +1.2%
output.weight (LM head) IQ4_XS 2560 x 248320 3 16,1,4,1 3988.00 84.7 16,2,4,1 3065.00 110.2 +30.1%
output.weight (LM head) IQ4_XS 2560 x 248320 4 16,1,4,1 6264.82 53.9 16,2,4,1 4873.99 69.3 +28.5%
attn_gate (11 layers) IQ3_S 2560 x 6144 3 32,2,4,1 110.20 61.3 8,1,4,1 88.53 76.3 +24.5%
attn_qkv (1 layer) IQ3_S 2560 x 10240 1 4,2,4,1 108.40 103.9 32,2,4,1 94.17 119.6 +15.1%
attn_qkv (1 layer) IQ3_S 2560 x 10240 3 4,2,4,1 338.20 33.3 8,1,4,1 143.40 78.5 +135.8%
ssm_out / attn_out (7 layers) IQ3_S 6144 x 2560 1 4,2,4,1 68.80 98.2 32,2,4,1 57.85 116.8 +18.9%
ssm_out / attn_out (7 layers) IQ3_S 6144 x 2560 3 4,2,4,1 161.60 41.8 8,1,8,1 93.34 72.4 +73.1%
shexp_gate / up (23 layers) IQ3_S 2560 x 640 3 32,2,4,1 14.72 47.8 8,1,8,1 12.05 58.4 +22.2%
shexp_down (36 layers) IQ4_NL 640 x 2560 3 8,1,4,1 13.42 68.7 4,2,4,1 11.63 79.3 +15.4%
shexp_down (36 layers) IQ4_NL 640 x 2560 4 8,1,4,1 15.80 58.4 4,2,4,1 13.31 69.3 +18.7%
routW_ (ffn_gate_inp BF16) BF16 2560 x 512 1 32,1,4,1 21.84 120.1 32,1,8,8 19.86 132.0 +10.0%
routW_ (ffn_gate_inp BF16) BF16 2560 x 512 3 32,1,4,1 23.39 112.1 32,1,8,8 20.44 128.3 +14.4%
routW_ (ffn_gate_inp BF16) BF16 2560 x 512 4 32,1,4,1 21.76 120.5 32,1,8,8 20.94 125.2 +3.9%

2.2 Multi-Token Routed Expert Kernels (moe.metal: expert_parity --bench) #

Measured on Apple M5 with share = 0.65 (matching MoCE = 0.85 + AcceptMoE speculative verify overlap: 18.1 unique experts at ntok = 3, 21.3 unique experts at ntok = 4):

Expert Formats (gu / dn) ntok Unique Groups Baseline Config (gu / dn) Baseline Layer (us / GB/s) Patched Config (gu / dn) Patched Layer (us / GB/s) Speedup
IQ2_S / Q2_0 (34 layers) 1 10.0 32,1,4 / 8,2,4 156.6 us (96.4 GB/s) 32,1,4 / 8,2,4 152.1 us (99.3 GB/s) +3.0%
IQ2_S / Q2_0 (34 layers) 3 18.1 32,1,4 / 8,2,4 351.2 us (77.7 GB/s) 16,1,4 / 8,2,4 318.7 us (85.6 GB/s) +10.2%
IQ2_S / Q2_0 (34 layers) 4 21.3 32,1,4 / 8,2,4 445.3 us (72.3 GB/s) 16,1,4 / 8,2,4 403.1 us (79.9 GB/s) +10.5%
IQ2_XXS / Q2_0 (12 layers) 1 10.0 16,1,4 / 8,2,4 152.0 us (85.9 GB/s) 16,1,4 / 8,2,4 147.3 us (88.6 GB/s) +3.2%
IQ2_XXS / Q2_0 (12 layers) 3 18.1 16,1,4 / 8,2,4 338.6 us (69.6 GB/s) 16,1,4 / 8,2,4 314.1 us (75.1 GB/s) +7.8%
IQ2_XXS / Q2_0 (12 layers) 4 21.3 16,1,4 / 8,2,4 430.3 us (64.7 GB/s) 16,1,4 / 8,2,4 398.4 us (69.8 GB/s) +8.0%
IQ3_S / Q2_0 (2 layers) 1 10.0 32,1,4 / 8,2,4 170.9 us (109.4 GB/s) 32,1,4 / 8,2,4 167.5 us (111.6 GB/s) +2.0%
IQ3_S / Q2_0 (2 layers) 3 18.1 32,1,4 / 8,2,4 376.5 us (89.7 GB/s) 16,1,4 / 8,2,4 341.2 us (98.9 GB/s) +10.3%
IQ3_S / Q2_0 (2 layers) 4 21.3 32,1,4 / 8,2,4 475.7 us (83.7 GB/s) 16,1,4 / 8,2,4 429.0 us (92.8 GB/s) +10.9%
IQ3_XXS / IQ4_NL (K256) 1 10.0 32,1,4 / 4,2,4 193.9 us (112.2 GB/s) 16,1,4 / 8,2,4 183.6 us (118.5 GB/s) +5.6%
IQ3_XXS / IQ4_NL (K256) 3 18.1 32,1,4 / 4,2,4 397.7 us (98.8 GB/s) 16,1,4 / 8,2,4 358.9 us (109.5 GB/s) +10.8%
IQ3_XXS / IQ4_NL (K256) 4 21.3 32,1,4 / 4,2,4 491.1 us (94.4 GB/s) 16,1,4 / 8,2,4 445.7 us (104.0 GB/s) +10.2%

2.3 Prefill Routed Expert GEMM (moe_mm.metal: mm_gu3 Single rw Cooperative Tensor) #

Measured on Apple M5 (moe_mm_parity --bench --v3 --tile 16,32,32,4 --dtile 64,32,2, 512 experts):

Expert Formats (gu / dn) Chunk T Baseline mm_gu3 (2x rw) Patched mm_gu3 (1x rw) mm_gu3 Speedup Full MoE Layer (plan + gather + gu + dn + combine)
IQ2_S / Q2_0 128 11,890.9 us 9,542.2 us +24.6% (-19.8% time) 15.28 ms -> 13.07 ms (175 -> 204 tok/s)
IQ2_S / Q2_0 512 12,817.3 us 10,481.4 us +22.3% (-18.2% time) 16.98 ms -> 14.76 ms (628 -> 723 tok/s)
IQ2_S / Q2_0 1024 19,313.2 us 17,330.9 us +11.4% (-10.3% time) 26.24 ms -> 24.17 ms (813 -> 883 tok/s)
IQ2_XXS / Q2_0 128 8,649.4 us 7,783.9 us +11.1% (-10.0% time) 12.10 ms -> 11.22 ms (220 -> 238 tok/s)
IQ2_XXS / Q2_0 512 9,640.4 us 8,733.2 us +10.4% (-9.4% time) 13.88 ms -> 13.01 ms (768 -> 820 tok/s)
IQ2_XXS / Q2_0 1024 15,803.1 us 14,687.3 us +7.6% (-7.1% time) 22.78 ms -> 21.72 ms (937 -> 982 tok/s)
IQ3_XXS / IQ4_NL 128 12,104.1 us 7,945.3 us +52.3% (-34.3% time) 17.54 ms -> 13.48 ms (152 -> 198 tok/s)
IQ3_XXS / IQ4_NL 512 13,031.4 us 8,819.4 us +47.8% (-32.3% time) 19.27 ms -> 15.16 ms (554 -> 704 tok/s)
IQ3_XXS / IQ4_NL 1024 17,240.2 us 14,431.0 us +19.5% (-16.3% time) 26.57 ms -> 23.05 ms (803 -> 925 tok/s)
IQ3_S / Q2_0 128 27,587.9 us 25,239.4 us +9.3% (-8.5% time) 31.06 ms -> 28.62 ms (86 -> 93 tok/s)
IQ3_S / Q2_0 512 30,510.1 us 27,923.4 us +9.3% (-8.5% time) 34.77 ms -> 32.08 ms (307 -> 332 tok/s)
IQ3_S / Q2_0 1024 41,597.1 us 37,399.7 us +11.2% (-10.1% time) 48.28 ms -> 44.13 ms (442 -> 483 tok/s)

2.4 QSA & GDN Decode (T = 1..4) and Prefill (T = 1024..2048) Audit #

  • gdn.metal (gdn_step): gdn_step (gdn_step_t<4>, 512 threads = 16 simdgroups, RPT = 8 state floats per lane) runs 36 layers in 0.996 ms (113.7 GB/s state bandwidth) for ntok = 1 verify, 1.630 ms for ntok = 3 verify, and 1.923 ms for ntok = 4 verify. We also benchmarked gdn_step8 (gdn_step_t<8>, 1024 threads, RPT = 4) and confirmed gdn_step (IB = 4) is 1.5–6% faster across all ntok = 1..4 on M5.
  • gdn_prompt.metal (gdn_scanr_4): At T = 2048, gdn_pre (1.54 ms) + gdn_scanr_4 (7.73 ms) + gdn_post (1.34 ms) = 10.57 ms total (5.16 us/tok/layer), 2.65x faster than chunked (28.02 ms).
  • qsa.metal (qsa_prep, qsa_score, qsa_topk, qsa_attn, qsa_merge): At 4096 context cells, ntok = 1 takes 89.4 us/layer (1.07 ms across 12 layers) and ntok = 3 takes 209.4 us/layer (2.51 ms across 12 layers, 69.8 us/tok/layer), scaling sub-linearly with ntok (qsa_topk takes 7.37 us for 3 tokens vs 7.30 us for 1 token).

3. Summary of Estimated End-to-End Impact on M5 (K512_iq2_xs with --spec 3) #

  • Per T = 3 Speculative Verify Step (48 layers + LM head + MTP draft):
    • Dense GEMVs (attn_qkv, attn_gate, attn_q, ssm_out, attn_output, shexp, predW_, routW_): saves ~2.15 ms per T = 3 verify step (~3.40 ms per T = 4 verify step).
    • output.weight (IQ4_XS 2560 x 248320) LM head: saves ~0.92 ms at T = 3 (~2.07 ms at T = 4).
    • Routed Experts (moe_gu + moe_down with share = 0.65): saves ~1.48 ms across 48 layers at T = 3 (~1.92 ms at T = 4).
    • MTP Draft Step (Q8_0 mhead_t_ + gm projections): saves ~0.11 ms per draft token (~0.33 ms per 3-draft round).
    • Total Speculative Step Reduction: ~4.8 - 7.7 ms per speculative verify cycle (+12% to +18% multi-token speculative throughput), plus +10% to +32% faster MoE prefill (mm_gu3).