From fadc5c26bd5335162bb7a16fa53050815ff84dbc Mon Sep 17 00:00:00 2001 From: dawn <90008@klbr.net> Date: Sun, 18 Oct 2026 12:22:21 +0300 Subject: [PATCH] document paired-run review limits and baseline provenance --- audits/r39_integration.md | 4 ++-- audits/r39_opus_review.md | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/audits/r39_integration.md b/audits/r39_integration.md index 1484b13..083b4e9 100644 --- a/audits/r39_integration.md +++ b/audits/r39_integration.md @@ -51,9 +51,9 @@ non-empty request logit biases remain unsupported by the Metal engine (no logit_ ## quiet paired CLI followup -same-day baseline reconstruction independently checked by opus: only the PLE object differs in the host archive; all other linked objects match. runtime shader directories restore the exact prior versions. both arms run the same inputs/env, with service stopped and other agent suites complete. source/build provenance, frozen shaders, commands, full logs and summary: `results/round39/model-ab/`. +same-day baseline reconstruction independently checked by opus: only the PLE object and regenerated archive symbol index differ in the host archive; all other linked objects match. runtime shader directories restore the exact prior versions. both arms run the same inputs/env, with service stopped and other agent suites complete. source/build provenance, frozen shaders, commands, full logs and summary: `results/round39/model-ab/`. -excluded one warmup per arm, then 10k order A/B/B/A/A/B and 2k A/B/B/A. every run exits 0 and **all five pairs produce identical generated token IDs**. +excluded one warmup per arm before the 10k case, then 10k order A/B/B/A/A/B and 2k A/B/B/A (no separate 2k warmup). every run exits 0 and **all five pairs produce identical generated token IDs**. | case | candidate/baseline decode time, adjacent pairs | candidate/baseline prefill time | |---|---|---| diff --git a/audits/r39_opus_review.md b/audits/r39_opus_review.md index 6bf1801..3a5146c 100644 --- a/audits/r39_opus_review.md +++ b/audits/r39_opus_review.md @@ -188,3 +188,33 @@ cost on darwin (S1), answer_room default (S2), prompt-cache cap (S3). compare the generated ids per pair as a correctness gate. (2) the 10K case decodes ~31 tokens in ~0.9 s, which is too short a window to resolve a 2% decode delta with 3 pairs; call that case's decode inconclusive unless the spread is clearly under 2%, and read prefill separately. + +## followup v5 — reading the paired CLI A/B (logs only: `results/round39/model-ab/runs.json`, `summary.json`) + +amended at the parent's request: everything below is descriptive. nothing here is timed beyond what runs.json +records, and no statistical bound is claimed. + +- output token IDs equal in all 5 measured pairs (and the warm pair). aggregate `STRATA_SPEC` rounds/drafted/accepted are equal + within each case for every run of both arms (10K: 13/23/18; 2K: 39/99/88). equal aggregates support comparable work; + they do not establish an identical acceptance trajectory, which is not logged and is not inferred from the output IDs. +- drift: decode ms/token rises overall through the session, 28.65 → 33.05 over the 10K series, but not monotonically + (10K decode 920.70 ms, then 915.71 ms in consecutive runs; 2K 4213.72 then 4143.13). it stays at 31.5–33.2 in the + 2K series. 10K prefill goes 21.6 s → 30.4 s. this affects both arms. sustained-load throttling on a fanless Air is + plausible but unmeasured (no thermal readings), so no cause is claimed. +- descriptive drift-adjusted fit (OLS: value ~ run index + arm, computed from runs.json; n = 6 and n = 4, so no + confidence intervals are claimed): + - 10K decode: −5.9 ms (−0.6%), residuals up to ±22 ms. + - 10K prefill: −431 ms (−1.7%), residuals up to ±1.0 s. + - 2K decode: +64.6 ms (+1.6%), residuals ±49 ms, 1 dof. + - 2K prefill: +210 ms (+3.2%), residuals ±232 ms, 1 dof. +- prefill as a same-path comparison: prompt chunks call `gather_batch` with T = 2048 (one 1984-token chunk for 2K; + 4×2048 + 1988 for 10K). old and new code take the same slow path for T > 16, and the shader changes are not + executed on the M5 default path. the prefill arm differences nonetheless have opposite signs (−1.7%, +3.2%). this + shows the run-to-run instability of this setup; it is not a statistically established noise floor. +- code-path note (an operation count, kept separate from the measurements; it does not enter the verdict): in decode, the verify windows call `gather_batch` with T ≤ 4. the new + matching adds at most 4×16 key comparisons of 64 B each and issues no more SSD read tickets than the old code (its + fast path covers every case the old fast path did; the non-fast path is unchanged). its latency was not timed. +- verdict: inconclusive for small performance changes. no consistent slowdown is visible in 10K; the 2K decode + differences (+1.5–1.7% per pair) cannot be separated from the drift and instability in this run. a cooled, stationary + experiment would be needed to resolve differences at this scale. the parent's wording in `audits/r39_integration.md` + matches this. -- knotmirror 0.0.1