From 36179319ae0779ffcf96469b3bf17760399b5bc9 Mon Sep 17 00:00:00 2001 From: dawn <90008@klbr.net> Date: Sat, 17 Oct 2026 23:44:47 +0300 Subject: [PATCH] PLAN.md: record Round 23 benchmark results (42.01 tok/s decode, 24.78 tok/s @10.2K, 3,293.32 ms / 602.1 tok/s GPU kprof) --- PLAN.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/PLAN.md b/PLAN.md index fa39194..f2ebd71 100644 --- a/PLAN.md +++ b/PLAN.md @@ -331,7 +331,7 @@ caveats carried with these: 1. `madvise(MADV_DONTNEED)` on the `181 MB` of source BF16 dense matrices in `mtp-q2_0.gguf` after quantizing them into `mq8_` (`90 MB` `Q8_0`) + switching `draft_vocab` from `106,299` tokens (`145 MB`) to `draft_vocab_en.bin` (`40,525` tokens, `55 MB`) saves **`271 MB` of RAM**, allowing `STRATA_METAL_ARENA_GB=6.5` (`9.9 GB` resident RSS, `-1.0 GB` lower than baseline!). 2. Capping `mtp_encode`'s local QSA sliding window to `256` cells (`4` chunks of 64 instead of `33`) and running `mtp_prefill` only on the last `256` tokens of the final prompt chunk eliminates the multi-chunk `mtp_prefill` lag on `12K–32K` prompts (`314–330 tok/s` end-to-end 2K prefill!). 3. Setting `STRATA_METAL_PREFETCH_DRAFT=0` in `stream.cpp` (`ahead()` only prefetches for root token `t=0`) and `AcceptMoE` (`arXiv:2608.02989`) draft routing in `router.metal`/`moe.metal` (`sg > 0` protects `r_prot = 1` and squares `moce_scale`) cuts speculative SSD traffic from `332 -> 177 MB/window` (`94.1%` arena hit rate!) and pushes **32K-context decode to `30.37 tok/s` (`persona_chat`), `33.44 tok/s` (`reasoning`), `34.41 tok/s` (`coding`), and `42.48 tok/s` (`prompt_2048tok_ops`, `3.26 tok/round`, `89.8%` acceptance)**! - - **Rounds 1–22 Optimization Blitz (`bec6ab3`, `4b0f233`, `d74a874`, `89188ac`, `f15273e`, `f11c331`, `059bf9c`, `8bc5f03`, `0548b0c`, `aeb4f92`, `70aef90`, `86255f2`, `c6fccd4`, `ceb1722`, `f73d5df`, `2eb7dc0`, `f3384f6`, `5dad020`, `35f84cb`, `75d719c`, `4b7eddc`, `0d39e8e`, `3c41c24`):** + - **Rounds 1–23 Optimization Blitz (`bec6ab3`, `4b0f233`, `d74a874`, `89188ac`, `f15273e`, `f11c331`, `059bf9c`, `8bc5f03`, `0548b0c`, `aeb4f92`, `70aef90`, `86255f2`, `c6fccd4`, `ceb1722`, `f73d5df`, `2eb7dc0`, `f3384f6`, `5dad020`, `35f84cb`, `75d719c`, `4b7eddc`, `0d39e8e`, `3c41c24`, `e1737f7`):** 1. **MTP `STRATA_METAL_MTP_SINK=64` + `mtp_wrote` Valid-Run KV Fix + Batched 4-Row `fc_hidden` GEMV + `slop4`-Aware `mtp_slop_pen` (`window.cpp`, `qsa.metal`, `mtp_chain.metal`):** Fixed multi-chunk MTP QSA KV zero-reads, preserved the 64-token system-prompt sink in MTP KV, batched `fc_hidden` (`-20.9 MB/step`), and synced `slop4` state into the MTP draft head + wired `SuffixDrafter::propose_after` to extend partial neural drafts. 2. **M5 Multi-Token Decode & Prefill Kernels (`gemv_cat`, `gr_norm_comb`, `gdn_step`, `gdn_scanr_4`, `qsa_prep`, `moe_gu`/`moe_down`, `mm_gu3`, `-1,247` Prefill Barriers, `P_GSLOT` L2 Fix):** Fused `ssm_alpha + ssm_beta` (`gemv_cat`) and `moe_combine_sh + gr_norm` (`gr_norm_comb`, `-83` dispatches & `-48` barriers/window), optimized `gdn_step` (`-12%` at `ntok=4`, `simd_broadcast`) and `gdn_scanr_4` (`-8.4%` at `T=2048`), parallelized `qsa_prep`, fixed the Apple M5 L2 cache-coherency race in `EXPERT_SPLIT=1` via `P_GSLOT` (`100%` bitwise deterministic across all 48 layers), removed `1,247` redundant prefill barriers, and added M5 shape-aware `gemm_f16` tiles + `publish_ready` gated on `prompt`. 3. **Round 4 Prefill + PLE + MTP + MoE Plan Fusions (`gr_norm_h_moe`, `ple_moe_gate_norm`, `ple_query_gate_norm`, `mtp_rms_xprep`, `mtp_combine_gnorm`, `mm_place` 32-subblock + `mm_scan` pad-clear, `mm_gather_h`):** Folded `moe_out` into per-token 4-stream `gr_norm_h_moe` (`-25%`), fused `ple_gnorm` + `ple_gate_norm` (`-42.5%`, freeing **`188.8 MB` of prompt scratch RAM (`2.35 -> 2.16 GB`)**), fused `mtp_rms_xprep` + `mtp_combine_gnorm` (`mtp_ms` down to **`218 ms` / `2.25 ms/draft`**), eliminated `mm_clear` and replaced `mm_place`'s $O(1024^2)$ loop with 32-subblock nibble counting (**`12.9x` faster `mm_place`, `5.89x` faster MoE plan**), and added FP16 `mm_gather_h` (**`1.70x` faster gather, `-40.2 ms/chunk`**). @@ -353,7 +353,8 @@ caveats carried with these: 19. **Round 20 `Window::mtp_load` `707.8 MB` Expert `mlock` (`mtp_ms` `398 -> 122.2 ms` on `11K`, `433 -> 202.4 ms` on `2K`, `40.57 tok/s` Decode) + Decode `qsa_attn` (`-20%..-40.5%`) & `qsa_merge` (`-9.6%..-12.7%`) + `gdn_scanr_4` 32-Bit Base Offsets (`186.15 ms` in `kprof`) (`audits/opus_r20_mtp_qsa.md`, `audits/sol_r20_gather.md`, `audits/argon_r20_gdn.md`):** Uncovered that `Window::mtp_load` (`window.cpp`) was not `mlock`ing the `707.8 MB` of `T_Q2_0` routed experts in `Wm_` (`mtp-q2_0.gguf`), allowing macOS to evict them during `1.12 GB` prompt-scratch prefills and causing `+220..305 ms` of hard SSD page faults during `mtp_draft`; wiring (`mlock`ing) the `707.8 MB` of active MTP experts + `MADV_DONTNEED`ing `181.2 MB` of unused pages cut `mtp_ms` from **`397.88 -> 122.18 ms` (`3.26x` faster!)** on the `10,180`-token Albedo prompt and from **`432.82 -> 202.37 ms` (`2.14x` faster!)** on `prompt_2048tok_ops`, pushing decode to **`40.57 tok/s` (`24.65 ms/tok`, `3,130.15 ms`)**; register-cached `s0, s1` across softmax + unrolled V x4 in decode `qsa_attn` (**`-20.4%..-40.5%` faster** across `pos0 = 512..8192`) and shared `ek[]`/`L_sh` in `qsa_merge` (`-9.6%..-12.7%` faster); and replaced 64-bit per-lane pointers in `gdn_scan_r` with 32-bit base offsets (`s0, k0, v0, ks, rs`) + final-token dead-lookahead elimination, cutting `pg:gdn_scanr_4 x36` in `kprof` at `T=1983` to **`186.15 ms` (`5.17 ms/layer`, `-61.4 ms` / `-24.8%` vs Round 5)**! 20. **Round 21 `gr.metal` `GR_NTOK` Unroll (`-5.4%..-18.0%` across `194` Dispatches/Window) + `qsa2.metal` (`-36.8%..-40.6%` `qsa_attn2` & `-8.3%..-14.6%` `qsa_merge2`) + `gdn_step` 32-Bit `st_off` & `moe_plan` `topk10` `unroll_count(10)` (`-9.5%..-26.6%`) (`audits/opus_r21_gr.md`, `audits/sol_r21_qsa2.md`, `audits/argon_r21_gdn_moe.md`):** Unrolled `GR_NTOK` (`[[function_constant(1)]]`) and `KS`/`UNIT`/`hc` loops in `gr_down_f` (`GR_NTOK <= 6`, **`-5.4%..-7.0%` on `gr_down_q8_8_4`, `-13.4%..-14.2%` on `gr_down_q8_8_8`**) and `gr_up_f` (**`-1.6%` at `ntok=1`, `-6.0%` at `ntok=2`, `-14.7%` at `ntok=4`, `-18.0%` at `ntok=8`**) + moved `gr_q2s`/`gr_p4s` to program-scope `constant` (`67/67` bitwise PASS); ported register-softmax, unrolled 8-cell V blocks, shared `ek[]`/`L_sh`, and barrier-free `qsa_write_prepared` to `qsa_attn2` (`-36.8%..-40.6%`) and `qsa_merge2` (`-8.3%..-14.6%`); and replaced `gdn_step_t`'s 64-bit `st_base` with 32-bit `st_off` + `float4` `conv_w` + `constant` `gdn_q2s`/`gdn_p4s`, and replaced `moe.metal` `plan_body`'s runtime integer division `i / int(a.topk)` with `topk10 ? ((i * 205) >> 11) : (i / int(a.topk))` + `unroll_count(10)` (**`-9.5%..-20.4%` faster `moe_route` and `-11.7%..-26.6%` faster `moe_plan` at `ntok = 2..8`**), setting all-time decode speed records across all 4 workloads! 21. **Round 22 `gr_down_4_4` / `_r2` `NR=2` `xn` Reuse (`-8.1%` `ntok=4`, `-16.8%..-26.1%` `ntok=5..8`) + `moe_down` / `moe_gu` `q2_combine` (`-7.7%..-22.5%` at `NE >= 2`) + `plan_body` `sg==0` SIMD Prefix Sum (`-14.6%..-35.4%`) + Barrier-Free `swiglu_prep` (`audits/opus_r22_gr.md`, `audits/sol_r22_router_glue.md`, `audits/argon_r22_moe.md`):** Added template `NR` + direct `float4` FMA (no `xs` array copy) to `grd_acc` / `gr_down_f`, using 128-thread `gr_down_*_4_4` at `ntok <= 4` (**`-1.9%..-8.1%` faster**) and `gr_down_*_4_4_r2` / `8_8_r2` (`NR = 2` rows/simdgroup sharing `xn` reads) at `ntok >= 5` in `gr_read` and `mtp_gr_read` (**`-16.8%..-26.1%` faster**, `75/75` bitwise PASS); added `q2_combine` (`ushort2 pk[NR]; float d[NR];` = 4 registers for `NR=2`, loading `xs[16]` once per token `t` across all `NR` rows) in `moe.metal`, cutting `moe_down_142_8_2_4` by **`-7.7%..-8.6%` at `NE = 2..4` (`-16.1%` on `T_Q2_0` prep)**, `Q2_0` `moe_gu` by **`-12.2%..-22.5%`**, and `IQ2_XS` `moe_gu` by **`-8.2%..-17.1%`**, plus `sg == 0` `simd_prefix_exclusive_sum` in `plan_body` (**`-8.8%..-22.7%` faster `moe_route` and `-14.6%..-35.4%` faster `moe_plan`**); and converted `swiglu_prep` (`glue.metal`) to program-scope `constant` `glue_q2s`/`glue_p4s` + barrier-free `simd_shuffle_down` (`64 -> 0 B` TG mem, `-48` barriers/window). - 22. **Measured End-to-End Speed (`32K` Context, `6.5 GB` Arena, Unpruned `K=512` `IQ2_XS`, `call_3817687`, `call_2483052`, `call_3102851`, `call_2898988`, `call_4110683` & `call_4178375`):** `prompt_2048tok_ops` prefill **`393.03–405.11 tok/s` (`4,897–5,048 ms` end-to-end; `4,666 ms` / `425.0 tok/s` inside `prompt_chunk` — `-1,416 ms` / `+29.0%` faster end-to-end, `3,301.86 ms` / `600.6 tok/s` pure GPU `kprof`, `2,094` dispatches, `1.12 GB` transient scratch!)**, decode **`41.06–41.7 tok/s`** (`24.0–24.35 ms/tok`, **`3,048.69 ms` decode, `mtp_ms = 193.94 ms` / `1.96 ms/draft`**); `prompt_albedo_11k` (`10,180` tokens) prefill **`432.1–445.79 tok/s` (`22.84 s` vs `28.25 s`, `-5.41 s` faster; Chunk 0 = `4,613 ms`, Chunk 1 = `501.5 tok/s` / `4,084 ms`, `mtp_ms = 122.18–127.54 ms`, and `@10.2K` decode hit `24.71 tok/s` / `1,254.57 ms` vs `22.06 tok/s` / `1,405.37 ms`!)**; `coding` prefill **`120.61–126.78 tok/s` (`2,413–2,537 ms`, `+32.2%`)**, decode **`35.66–37.1 tok/s`** (**`6,141.48 ms` decode, `mtp_ms = 380.29 ms`**); `reasoning` prefill **`120.83–122.92 tok/s` (`2,416–2,458 ms`, `+33.3%`)**, decode **`36.70–36.74 tok/s`** (**`5,961.16 ms` decode, `mtp_ms = 370.78 ms` — under `6,000 ms`!**); `persona_chat` prefill **`119.20–124.62 tok/s` (`2,319–2,424 ms`, `+34.7%`)**, decode **`33.04–33.2 tok/s`** (**`6,628.00 ms` decode, `mtp_ms = 424.19 ms`**). + 22. **Round 23 `ksg` `popcount` (`-2.5%..-5.5%` `IQ2_XS` `moe_gu` & `-3.9%` `IQ3_XXS`) + `gemv_direct_x`/`gemv_q2_raw_combine` (`-3.4%..-5.4% `Q8_0` & `-2.2%..-4.1%` `Q2_0R`) + `mm_gu3` `sign4_iq2xs` & Constant-Ptr Folding (`776.7 ms` `IQ2_XS` & `45.9 ms` `IQ3_S`, `3,293.32 ms` / `602.1 tok/s` GPU `kprof`, `42.01 tok/s` Decode) (`audits/opus_r23_quant.md`, `audits/sol_r23_gemv.md`, `audits/argon_r23_moemm.md`):** Computed `ksg` sign parity (`i | ((popcount(i) & 1u) << 7)`) for `IQ2_XS` (`ctab == 1`, `-2.5%..-5.5%` faster `moe_gu`) and `IQ3_XXS` (`TAB = 256`, `-3.9%`) in `quant.metal`; added direct `float4` FMA `gemv_direct_x` (`Q8_0` `ntok=1` & `LPR=16, NR=1, ntok=2`, `-3.4%..-5.4%` faster) and `gemv_q2_raw_combine` (`Q2_0R` `ntok=1`, `-2.2%..-4.1%` faster) in `gemv.metal`; and added 4-bit register `sign4_iq2xs(j, hh)` (zero `popcount` on `hh==0` lanes) + constant-offset pointer folding (`(k & 255) >> 3` / `>> 2`) + `fma` scale math in `moe_mm.metal`, cutting `W8` by **`-4.19%..-4.35%`** (`pg:mm_gu3_22_16_32_32_4 x34` down to **`776.71 ms` / `22.84 ms/layer`, `-25.5 ms/chunk`**) and `W8` by **`-8.5%` at `T=2048` (`45.93 ms` for `x2`) and `1.82x` at `T=512` (`18.73 -> 10.28 ms` across the VGPR occupancy cliff!)**, pushing full-model `kprof` GPU time at `T=1983` down to **`3,293.32 ms` (`602.1 tok/s` pure GPU prefill)** and `prompt_2048tok_ops` decode to **`42.01 tok/s` (`23.80 ms/tok`, `3,023.11 ms`)**! + 23. **Measured End-to-End Speed (`32K` Context, `6.5 GB` Arena, Unpruned `K=512` `IQ2_XS`, `call_3817687`, `call_2483052`, `call_3102851`, `call_2898988`, `call_4110683`, `call_4178375`, `call_3948986` & `call_4500449`):** `prompt_2048tok_ops` prefill **`390.65–405.11 tok/s` (`4,897–5,079 ms` end-to-end; `4,666 ms` / `425.0 tok/s` inside `prompt_chunk` — `-1,416 ms` / `+29.0%` faster end-to-end, `3,293.32 ms` / `602.1 tok/s` pure GPU `kprof`, `2,094` dispatches, `1.12 GB` transient scratch!)**, decode **`41.65–42.01 tok/s`** (`23.80–24.01 ms/tok`, **`3,023.11 ms` decode, `mtp_ms = 193.94–196.59 ms` / `1.96 ms/draft`**); `prompt_albedo_11k` (`10,180` tokens) prefill **`432.1–445.79 tok/s` (`22.84 s` vs `28.25 s`, `-5.41 s` faster; Chunk 0 = `4,613 ms`, Chunk 1 = `501.5 tok/s` / `4,084 ms`, `mtp_ms = 121.54–127.54 ms`, and `@10.2K` decode hit `24.78 tok/s` / `1,251.04 ms` vs `22.06 tok/s` / `1,405.37 ms`!)**; `coding` prefill **`120.61–126.78 tok/s` (`2,413–2,537 ms`, `+32.2%`)**, decode **`35.30–37.1 tok/s`** (**`6,141.48 ms` decode, `mtp_ms = 380.29 ms`**); `reasoning` prefill **`120.83–122.92 tok/s` (`2,416–2,458 ms`, `+33.3%`)**, decode **`35.93–36.74 tok/s`** (**`5,961.16 ms` decode, `mtp_ms = 370.78 ms` — under `6,000 ms`!**); `persona_chat` prefill **`119.20–124.62 tok/s` (`2,319–2,424 ms`, `+34.7%`)**, decode **`31.78–33.2 tok/s`** (**`6,628.00 ms` decode, `mtp_ms = 424.19 ms`**). ### B. ruled out from the original blueprint -- knotmirror 0.0.1