regolith
strata but optimized for m5 (and maybe more later)
Star
Fork
Watch
regolith README.md
21 kB · 117 lines
Markdown
at main

regolith runs swift-1.5-qwen3.8-flash-next (qwen4_exp, 512 experts, 36 gated deltanet + 12 sparse attention layers + 51.2b n-gram ple table + neural mtp draft head + bf16 vision projector) on apple silicon via a patched strata metal engine.

./scripts/setup_apple_silicon.sh clones the pinned strata and llama.cpp trees into vendor/, applies scripts/strata_metal_complete.diff, downloads apple's metal-cpp headers, compiles strata_window_cli, and streams + repacks the unpruned k=512 iq2_xs model (3.90 gb dense .gguf + 36.23 gb .experts + 28.80 gb ple-iq4_nl.gguf + 889 mb mtp-q2_0.gguf) into models/K512_iq2_xs/.

./scripts/serve_k512_local.sh starts the local server on http://127.0.0.1:8080 (/v1/chat/completions and /v1/messages, 128k context, 6.5 gb wired expert arena, neural mtp + suffix speculative decoding).

research notes, ablations, and benchmark logs live in 00-overview-and-decisions.md .. 04-sources-and-references.md, audits/, bench/, and results/; the persona & anti-slop subproject lives in persona/.

requirements #

macos 26+ on apple silicon (m5 required for the default coop_k4 matmul2d cooperative-tensor path; tested on macos 27 on a 24 gb m5 macbook air), xcode command line tools, cmake, ninja, python3, and uv.

installation #

./scripts/setup_apple_silicon.sh
./scripts/serve_k512_local.sh

if you only want to build the engine without downloading the ~70 gb of model files:

SKIP_MODEL_DOWNLOAD=1 ./scripts/setup_apple_silicon.sh

m5 24 gb performance & memory #

measured on a fanless 13" m5 macbook air (mac17,3, 10-core gpu, 24 gb unified memory, 1 tb nvme ssd) running unpruned k=512 iq2_xs from ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 32k context (--spec 3 --spec-min-p 0.35 --suffix-draft 3, STRATA_METAL_MOCE_BONUS=0.85):

workload prompt prefill decode notes
pure gpu prefill (kprof, 1,983 tok) 633.28 tok/s (3.13 s) — 2,047 metal dispatches/chunk
prompt_2048tok_ops (1,984 prompt + 128 gen) 421.7–432.8 tok/s (4.58–4.70 s) 42.26–42.57 tok/s (23.5–23.7 ms/tok) 88.9% draft accept (3.26 tok/round)
prompt_albedo_11k (10,180 prompt + 32 gen) 434.8–473.8 tok/s (21.49–23.42 s) 34.13–35.47 tok/s (28.2–29.3 ms/tok) 5 chunks of 2,048, 78.3% accept
coding (306 prompt + 219 gen) 120.6–126.8 tok/s (2.41–2.54 s) 35.46–37.10 tok/s (27.0–28.2 ms/tok) c++20 lock-free ring buffer
reasoning (297 prompt + 219 gen) 121.6–122.9 tok/s (2.42–2.46 s) 34.92–37.20 tok/s (26.9–28.6 ms/tok) gdn/qsa derivation
persona_chat (289 prompt + 219 gen) 124.6–128.1 tok/s (2.26–2.32 s) 33.01–33.25 tok/s (30.1–30.3 ms/tok) lowercase + kaomoji
component (k=512 iq2_xs) 32k context 128k context (default) notes
active gpu dense tensors (k512-iq2_xs-metal.gguf) 2.27 gb 2.27 gb 146 coalesced mlock spans (145 page-aligned lazy ranges for 0.34 gb cpu token_embd + 1.28 gb bf16 hc_*)
hyper-connection q8_0 matrices (290 matrices) 0.68 gb 0.68 gb quantized at load from bf16 hc_*
neural mtp draft head (mtp-q2_0.gguf) 0.85 gb 0.85 gb 0.71 gb mlocked q2_0 experts + 0.09 gb q8_0 dense + 0.05 gb 40,525-tok draft vocab (0.18 gb bf16 released)
context state (32,896 / 131,200 cells) 0.70 gb 2.23 gb 36 dual-slot gst_[2] (226 mb) & 3-slot ghist_[3] (0 b/tok growth) + 12 qsa int8 kv & pooled indexers (14.2 kib/cell)
wired moe expert arena (4,129 expert slots) 6.50 gb 6.50 gb mlocked (33.6% of all 24,576 experts; remaining 29.7 gb of 36.23 gb .experts streamed from ssd)
ple n-gram table page cache reserve 0.50 gb 0.50 gb 28.80 gb ple-iq4_nl.gguf (51.2b params) stays on ssd (16 row reads/token)
elastic vision encoder (strata-vision + mmproj bf16) 0.00 gb 0.00 gb 0.92 gb active only during image turns (vram_elastic, 60s idle unload; .sve cached by sha-256)
steady-state decode total 8.91 gb (9.6–9.9 gb rss) 10.42 gb (10.9 gb rss) 9.60 / 11.10 gb planned against 18.80 gb cap (>13 gb left free for macos)
prefill scratch (PromptGemm, MT=2048, STRATA_METAL_PHASE_TRADE=1) +0.00 gb +0.00 gb phase-switched inside the 6.50 gb expert arena (1.12 / 1.32 gb, ~720 slots borrowed during prefill and returned to decode lru after prefill)

optimizations #

model & quantization #

  • unpruned k=512 ukisai gsq-rco iq2_xs metal bundle (3.90 gb 16 kib-aligned dense .gguf + 36.23 gb contiguous [gate | up | down] .experts, blk.8/13/37 iq1_m replaced with iq3_s/iq2_xxs, duplicate 28.80 gb ple table omitted)
  • native 49th-layer neural mtp draft pack (mtp-q2_0.gguf, 889 mb, runtime q8_0 dense quantization + madvise(MADV_DONTNEED) on 181 mb bf16 source matrices + 707.8 mb mlocked q2_0 routed experts + 40,525-token draft_vocab_en.bin)
  • reasoning_effort|default('low') chat template default

ssd expert streaming, unified memory arena & moce routing #

  • zero-copy file-backed mmap of <model>-metal.gguf with coalesced inward page-aligned MADV_DONTNEED ranges (145 lazy spans for cpu-only token_embd and skipped bf16 hc_*) and 146 coalesced mlock complement spans on active gpu dense tensors (never dropping shared 16 kib boundary pages)
  • wired metal expert arena (STRATA_METAL_WIRE_ARENA=1, STRATA_METAL_MLOCK_DENSE=1, fixed make_arena class-sizing loop) + phase-switched PromptGemm prefill scratch (STRATA_METAL_PHASE_TRADE=1, borrowing 1.12–1.32 gb of tail slots from the 6.50 gb arena during prefill and returning them to the decode lru after prefill at +0.00 gb extra memory)
  • dynamic n_expert (k = 224 / 256 / 320 / 512) across host and metal router/moe shaders
  • closed-loop moce cache-prior routing (STRATA_METAL_MOCE_BONUS=0.85, top-3 unperturbed protection, race-free single-read smap bitmask in moe_route / router_route / route_topk, deterministic smap_ publication at layer/cut boundaries, and opt-in rank-1 bonus unscaling fix)
  • acceptmoe speculative draft routing (STRATA_METAL_DRAFT_MOCE=1, r_prot = 1 and squared moce_scale on draft rows sg > 0) + root-token-only decode prefetch (STRATA_METAL_PREFETCH_DRAFT=0, STRATA_METAL_PREFETCH_UNPERT=1)
  • q8_0 cross-layer lookahead router (STRATA_METAL_PRED_Q8=1) and lossless bf16 main router (STRATA_METAL_ROUTER_BF16=1)
  • token-0-only router-guess gemv fast path (tile_ntok in Window::gemv) across all 93 predL_ / predL2_ dispatches when STRATA_METAL_PREFETCH_DRAFT <= 0
  • resident-first 2-phase routed moe execution (STRATA_METAL_EXPERT_SPLIT=1, STRATA_METAL_SHEXP_OVERLAP=1, STRATA_METAL_SPLIT_TURN=1) with MTLSharedEvent ssd handshake and P_GSLOT plan slot pinning
  • prefill next-layer (l + 1) router lookahead before layer l's ssd cut, most-tokens-first demand queueing, layer_rank_ hottest-first background prefetch, in_prompt_ single-piece pread, STRATA_METAL_PROMPT_CHUNK=2048, and tail-protected STRATA_METAL_PROMPT_MOCE
  • multi-chunk (n > C) hot-expert lru and !hot scratch-pool preservation across prompt chunks + allow_hot = false on initial layer-0 hold_region prefetch + separate prompt/decode ssd read counters
  • asynchronous background ple ssd gather_batch thread during prefill + direct-mode ple prefetch_rows duplicate-slot reuse and error-drain in PleTable::gather_batch

speculative decoding, kv sessions & suffix cache reuse (scr) #

  • single-command-buffer gpu neural mtp draft chain (mtp_chain.metal, STRATA_METAL_MTP_CHAIN=1) with combined (a + 1)-row catch-up + draft step-0 batch (STRATA_METAL_MTP_DEFER_CATCHUP=1, QSA_LAST_Q function constant 29, -20% to -26.8% per draft round) and SuffixDrafter::propose_after hybrid continuation
  • 64-cell attention sink (STRATA_METAL_MTP_SINK=64) + 256-cell local sliding window (STRATA_METAL_MTP_WIN=256, synced with mtp_prefill tail_cap) + mtp_wrote valid-run kv tracking + dead mtp indexer elimination (QSA_ROW0=24 k/v-only qsa_prep)
  • zero-copy dual-slot gdn recurrent state (gst_[2], ghist_[3], STRATA_METAL_GDN_RING=1, a.commit == 2u dual2 / 3u spec1 in gdn.metal, eliminating Window::fold() 100% of the time on speculative verification) + 24 kib tail_bak_ qsa raw-key snapshot/restore in Window::run / Window::commit
  • opt-in online cost-model SpecGate (STRATA_SPEC_GATE=suffix|1), step-aware STRATA_METAL_SPEC_MIN_P_SCHEDULE / STRATA_METAL_SPEC_MIN_CUM_P, and exact/typical temp > 0 speculative verification (STRATA_SPEC_SAMPLE=exact|typical)
  • elastic metal vision (strata-vision + mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf + 2d (t, h, w) m-rope qsa_prep#mrope + GENI .sve embedding injection + multi-turn image kv reuse)
  • 4-thread F_NOCACHE nvme session save/load (--save-session / --load-session, Window::session_save / session_load) serializing all 36 gdn + 12 qsa + ple + mtp sliding-window states (mtp_cell0_, mtp_cell1_, mtp_sink_) directly from/into MTLBuffers at 5.2 gb/s
  • bonsai-style automatic nvme prompt & conversation .kvc cache (--cache-disk / --cache-disk-size 16384 in --serve): startup .kvc header indexing (< 1 ms), post-DONE zero-latency auto-save with obsolete-prefix pruning, 2-prompt shared-prefix learning & .pin checkpointing (prompts-seen.bin), zero-copy prefix-share delta .kvc storage against pinned prefixes (.parent + SessionHeader base_cells, storing only [c0, n_cells) positional cells while reading [0, c0) directly from the parent .kvc), and automatic cross-conversation / post-restart nvme restore + scr splice over disk
  • suffix cache reuse (scr, Window::scr_relocate + strata::core::scr_plan with max_runs = 64 + 2-pass non-overlapping left/right-shift relocation before gap prefill + scr_carry 0..3 newline pad insertion for Δ ≡ 0 (mod 4) + <path>.pads + in-place active_path .kvc replacement across clm edits + 2-slot mru ck_ring in --serve) relocating qsa group-0 neox rope keys/pooled keys and mtp cell bounds in place while retaining post-edit 36-layer gdn state (lcp >= 0 multi-span clm memory edits & stripped-<think> turns)

apple silicon metal decode kernels (ntok = 1..8) #

  • gemv.metal: m5 per-format/per-shape/per-ntok tile selector in Window::gemv_tile & mtp_encode::tile ({32,2,4,1} for 2560->512, {8,2,4,1} for q2_0 640->2560 at ntok >= 4), #pragma clang loop unroll(full) across NTOK/NR/KS loops, fmt<F>::TAB > 1 barrier guards, gemv_cat (ssm_alpha + ssm_beta, mtp k + v / gate + up) & q8_0 gemv_catk, gemv_direct_x (q8_0), gemv_q2_raw_combine with ieee-754 fp16 0x64006400u mantissa-or decode (q2_0r), and 5-register packed_uint4 gemv_iq4_combine (iq4_xs)
  • moe.metal & quant.metal: single-buffer q[UNIT] gate/up register reuse (NE > 1), NE == 1 pointer/activation hoisting in gu_body/dn_body, UNIT == 32 fused qg/qu decode (q6_k), q2_combine<F> (q2_0/q2_0r/iq2_xs), 6-vgpr deq_iq_packed + iq_combine, ieee-754 fp16 0x64006400u mantissa-or decode in q2_combine<T_Q2_0R>, 64-bit uint2 grid loads in deq<T_IQ2_XXS/IQ2_XS/IQ2_S>, ushort q2/s2 + pre-shifted qh in deq<T_IQ2_S>, sign4_iq2xs + dead sign-table (ksg/load_signs) removal + shrunk TAB (1024/512/2048), QUANT_CTAB constant-memory tables (iq3_s pinned to threadgroup tab), m5 gl=16 / dl=8,dr=2 tiles, plan_body topk10 ((i * 205) >> 11) fast division + unroll_count(10) + sg == 0 simd_prefix_exclusive_sum, and moe_route e512 + preloaded 16-bit res_mask
  • window.cpp decode graph: wave-zipped resident routed experts (phase = 1) with shared-expert gemvs and swiglu_prep, deferred moe combine (comb_defer) folded into 4-tg-per-token gr_norm_comb (gr2.metal) and layer-0 ple_comb_query_gate_norm, and load-time window_init pso prewarming
  • gr.metal, glue.metal, & units.metal: 128-thread gr_down_*_4_4 (ntok <= 4) and 2-row-per-simdgroup gr_down_*_4_4_r2 (ntok > 4) with direct float4 fma, packed char4 GrdUnit<T_Q8_0, 2> with ROLL = true at KS = 4, GR_NTOK loop unrolling, program-scope constant gr_q2s/gr_p4s & glue_q2s/glue_p4s, barrier-free simd_shuffle_down in gr_up_f and swiglu_prep, n == 2560 unroll + apply hoist + pre-offset pointers in gr_norm, float4 vectorized xprep, and topk == 10 unrolled + pre-offset moe_combine_sh (STRATA_HC_Q8=1 default, opt-in STRATA_HC_Q4/STRATA_HC_Q5)
  • gdn.metal: gdn_step_t simd_broadcast, ib == 0 readout pruning, 32-bit st_off state offsets, float4 conv_w, program-scope constant gdn_q2s/gdn_p4s, 5 -> 4 barriers/token with token-invariant dt/ssm_a hoist + shrunk part[8]/red2[IB-1][S], and critical-path kl split across barrier 2 + pre-barrier-1 hist_out writeback + pre-barrier-3 global state store overlap
  • qsa.metal & qsa2.metal: parallelized qsa_prep (320 b threadgroup memory, 4 -> 2 barriers), hoisted/unrolled qsa_score, register-cached keys[4] + 20 -> 13 radix barriers in qsa_topk2, char4 K loads + register-cached softmax s0, s1 + unrolled V x4 in qsa_attn, and barrier-free simd_shuffle_down + shared ek[]/L_sh + float2 (m, l) loads + simd_max in qsa_merge
  • ple.metal, mtp.metal, mtp_chain.metal, & sample.metal: fused ple_comb_query_gate_norm and ple_query_gate_norm, register-cached ple_gnorm, ple_conv_res (eliminating gated buffer), ping-pong part[16] barrier halving across all ple.metal norms, [t][c] multi-row stream batching for mtp.fc_hidden, fused mtp_rms_xprep & mtp_combine_gnorm, float4/unrolled mtp_xprep/mtp_rms_xprep/mtp_combine_gnorm/mtp_embed/mtp_pick/mtp_slop_pen, and unrolled tg_argmax in sample.metal

apple silicon metal prefill (prompt_chunk) kernels #

  • window.cpp prefill graph & scratch: -1,247 redundant barriers removed + router/plan/shared-expert wave zipping (-190 barriers) + sigs right after route_topk so moe plan and shared-expert gemms overlap host/ssd at_cut(l) + Wr/Wr2 parity ping-pong (-47 router dq_f16 dispatches/chunk) + fused gdn ssm_alpha + ssm_beta (N = 96) gemm (-36 dispatches) + fused qsa Wk + Wv + Wik (+ Wiq) (N = 1152 / 1664) gemm (-24..36 dispatches) + qsa_score_mm/qsa_topk pso prewarming + multi-stage PromptGemm scratch aliasing (2.35 gb -> 1.12 gb at MT = 2048)
  • dense_mm.metal & mma.metal: m5 shape-aware gemm_f16 tile selector (M > 1536 threshold, 16_64_2, 32_64_2, 32_128_2, 64_32_2, 16_96_2 for Walbe, and 64_128_4 when M < 1536 && M % 128 in 1..64), hardware 2d-grid dq_f16 with half4 stores + TAB == 1 barrier elimination + T_Q5_1 + register table init & perm == 4 contiguous writes in dq_f16<T_IQ4_XS/IQ4_NL>, unrolled DW<T_IQ4_XS/IQ4_NL/Q6_K>, and half4 act_f16
  • prompt.metal: folded mm_combine + moe_out into per-token 4-stream gr_norm_h_moec and gr_apply_moec (plus gr_norm_h4) with register-cached rv, stream-by-stream reduction, compacted moe_comb_slots roff, and k-outer loop inversion; cooperative-tensor register epilogues in gemm_hc_up_reg_32_128_4 (unbound dead p.mixed, unrolled epilogue, hc_up_tile<16/32> row-tail trimming), gemm_hc_down (fusing gr_epi_down, * 0.25f, TN = 96 tail tile, guarded TM > 32 row-tail trimming), and gemm_swiglu_up (fusing swiglu_h, guarded TM > 32 row-tail trimming); n_exp == 512 unrolled route_topk
  • moe_mm.metal & ple.metal: mm_clear elimination (pad-clear folded into mm_scan), 32-subblock nibble-counted s_id-free mm_place, 128-bit uint4 fp16 mm_gather_h, single-rw mm_gu3 / mm_down3 with TAB > 1 barrier guards, hoisted row pointers (g0_l, u0_l, w0_l) and boff/sboff, hoisted tab + hh, branchless s >> (4 * hh) in W8<T_IQ3_S/IQ2_S> + W8<T_IQ3_S>::block, 256-wide super-block scale caching dg[]/du[] (T_IQ2_S/IQ2_XXS/IQ2_XS), W8<T_IQ2_XXS> direct byte loads, sign4_iq2xs + constant-offset pointer folding, scalar uint or in W8<T_Q6_K> / W4<T_Q6_K>, half4 coalesced HS store epilogue (T_IQ2_XS/IQ2_XXS/IQ3_XXS/Q2_0/IQ2_S/IQ3_S), 64-element block stepping (k0 += 64, w0_l += 18) + ieee-754 fp16 0x64006400u mantissa-or decode in W4<T_Q2_0>::run_q (mm_down3), and 1-tg-per-token ple_moe_gate_norm_c with simd slot compaction + ple_conv_res (half4 weights, branchless t >= 9) + ple_hist_adv
  • qsa_prompt.metal & gdn_prompt.metal: 0-barrier 1-simdgroup-per-row qsa_prep_sg (tg = 32 for ntok >= 3, -8% to -33% faster, 100% bitwise) + qsa_prep#qf16 fused fp16 query output, qsa_score_mm causal-future tile skip (b0 >= max_bid) + dead[0..3] hoist, qsa_topk cached keys[4] + simdgroup bucket scan + 2-barrier fused (equal, greater) emit (and skipped when p.flash && pos0 + T <= 2051), qsa_flash 4-simdgroup online softmax with sp[4][HPG*QCH] (13,760 b threadgroup memory, 1 barrier/tile) + unconditional hoisted rv/rk fills + byte-offset/zero-scale hoist + 32-bit ko[8]/vo[8] offsets with pre-added d0 + kl + c0 < nc pv bound + vec4 epilogue, 1-simdgroup-per-head barrier-free gdn_pre (float4 value-head conv, vector simd_sum(float4)) & gdn_post (hoisted pointers, preloaded gate/gamma, vector simd_sum(float4)), chunked WY matmul2d gdn_wy_prep + gdn_wy_scan (16-token chunks, (I + A)^-1 via 6 fp32 16x16 matmul2ds, 3-mma fp16 (hi, lo) split, 46 mb scratch aliased inside p.qkv, -26.6% to -28.3% scan time / -53.6 ms per 2k chunk), and logit-domain delayed-softmax top-10 routing in route_topk (-4.5% to -8.0%) & moe_route

anti-slop, persona data & serving #

  • slop4 (slop4.h) per-position stateful shaper (12 b/pos in window.cpp + gpu mtp_slop_pen in mtp_chain.metal with STRATA_METAL_MTP_SLOP=1), gated strictly on PROSE && !TOOL && !FENCE, with clause-entry branch-point penalties, slop4::hold() <|im_end|> mass preservation, </think>\n\n step0 empty-turn guard, slop_scan, and scr_relocate support
  • monroe prior log-odds llm-ism miner (mine_llm_isms_from_transcripts.py, pure per-order unigram+uniform prior with zero self-pooled weight) partitioning structural speech acts (phase3_sentence_surgery in prepare_phase3_sft_data.py, in-place span deletion preserving markdown/fences/kaomoji + r-tuning word-boundary abstention) vs surface lexical cliches (phase4_lexical_ftpo)
  • strata v0.1.45 serve/ & tools/strata_tokenizer.py ports + local hardening: incremental PromptEncoder prefix/plain-span reuse capped at PROMPT_ENCODER_KEEP_MAX = 16 with rejected-prompt forgetting, bounded tool-call parser scan cursor, content-keyed json lru logit-bias cache (stale _last identity shortcut removed), macos iokit/metal telemetry (MTLCreateSystemDefaultDevice cached once and objc_released), STRATA_OMIT_IMAGES_WHEN_NO_VISION=1 text-only image omission, and anthropic hard thinking budgets + answer_room answer reservation
  • standalone llama.cpp -cmoe apple silicon apfs / mmap patch (scripts/patch_m5_lazy_moe.py)

acknowledgements #

models used while building and auditing this: gemini 4 argon, claude opus 5.5, gemini 3.8 flash, gpt 6.1 sol, gpt 6 astra, and claude sonnet 5.5.

upstream & models #

papers #

  • on the design of qwen3.8-next architecture (arXiv:2608.30320)
  • mixture of cache-conditional experts (moce, arXiv:2412.00099) & acceptmoe (arXiv:2608.02989)
  • context language models (suffix cache reuse, arXiv:2609.37725)
  • riemannian constrained optimization (rco, arXiv:2605.00649) & gumbel-softmax quantization (gsq, arXiv:2604.18556)
  • antislop / ftpo (arXiv:2510.15061), verbal tics (arXiv:2604.19139), & r-tuning (arXiv:2311.09677)