# Small-model replacement research ## Status and recommendation This note retains the original smaller-brain research and the [quality-first execution plan](#quality-first-execution-plan) that supersedes its initial experiment order. The runnable six-configuration ladder has a [screening receipt](training.md#small-model-screening-receipt-2026-10-08); the later [compact improvement receipt](training.md#compact-model-improvement-receipt-2026-10-13) records 17 trained checkpoints and 51 actual exports. No replacement was selected or published. Architecture arithmetic, low loss and a high formal contract index are not semantic-equivalence evidence. The [quality-first execution receipt](training.md#quality-first-execution-receipt-2026-10-16) now records the actual 512-request Bonsai 2 pilot, its failed abstention gate, two final learning diagnostics and the independent memory investigation. **Current recommendation: repair source-level factual/abstention coverage before expanding the comparison.** The planned wider pretrained control and matched small control remain useful only after the pilot prerequisite passes; compress a model only after it becomes reliable. Do not treat another vocabulary/KD setting or architecture as a demonstrated fix. **[INFERENCE]** Better owner/pet attribution and grounded missing-fact teaching are the next data hypotheses. The pilot does not establish a parameter-count limit or prove that a wider backbone solves generalization. The 10–25M target remains a research question, not an established size floor or a promised result. The first milestone is a demonstrably reliable smaller brain, not a prescribed parameter count. Keep the frozen prompt text and closed tool grammar unchanged. The model must continue to make decisions; the app supplies lookup results and runs the body. See [prompt-format.md](prompt-format.md), [training.md](training.md) and [the base-model replacement contracts](extending.md#change-the-base-model) before implementation. ## 1. Model starting points ### Recommended experiment ladder The compressed variants below began as **calculated designs**. Their actual prepared parameter counts and first pilot measurements are recorded separately in [training.md](training.md#small-model-screening-receipt-2026-10-08); none has passed pet behavioral parity. Their sizes follow the official configurations and preserve the pretrained transformer layers. | Starting point | Vocabulary | Total parameters | Purpose | | ------------------------------------------------------------------------------ | ---------: | ---------------: | -------------------------------------- | | [Pythia-14M](https://huggingface.co/EleutherAI/pythia-14m), unchanged | 50,304 | 14.07M | Cheap pretrained/export control | | [Pythia-31M](https://huggingface.co/EleutherAI/pythia-31m), reduced vocabulary | 16,384 | **13.13M** | **First serious warm-start candidate** | | Pythia-31M, reduced further | 8,192 | 8.93M | Test the lower size floor | | [Pythia-70M](https://huggingface.co/EleutherAI/pythia-70m), reduced vocabulary | 4,096 | **23.11M** | More pretrained transformer capacity | | Custom Llama-style decoder | 8,192 | 22.29M | Modern scratch-training comparison | Pythia provides small, openly documented pretrained backbones, Apache-2.0 licensing and existing [Transformers.js ONNX artifacts](https://huggingface.co/Xenova/pythia-31m). That makes it a comparatively inexpensive starting point, not a claim that its architecture is state of the art. Existing artifacts are evidence of an export path, not proof against this repository's pinned runtime. Its large **untied** embedding and output tables dominate the smallest models. Reducing both tables retains the pretrained language-processing layers while reaching the target budget. Do not simply force those tables to share weights: that would discard learned differences. For these untied models, the size calculation is `non-embedding parameters + 2 × vocabulary size × hidden width`. The official cards give 4,739,072 non-embedding parameters and width 256 for Pythia-31M, and 18,915,328 non-embedding parameters and width 512 for Pythia-70M. Pin checkpoint and tokenizer revisions before experiments; model names alone are not reproducible inputs. Vocabulary replacement changes segmentation and generation. It needs continued training and distillation. [Vocabulary-reduction research](https://aclanthology.org/2025.coling-industry.64/) supports this approach, but also reports quality losses and incomplete recovery on some tasks. Those results are not pet-specific evidence. **Original recommendation, now exercised in the recorded experiments:** start with Pythia-31M/16K, then screen Pythia-70M/4K. This is historical context, not the next batch to repeat. The current plan tests an intact wider pretrained control before another vocabulary reduction. ### Newer tiny releases worth knowing about - **[Pebble-10M/25M](https://huggingface.co/basically-ai/Pebble-25M):** relevant sizes, small byte-level vocabulary and a modern training recipe. However, the model uses a custom Mamba2/attention hybrid requiring CUDA-oriented dependencies. It is not a demonstrated fit for the browser WASM stack. - **[Stentor3-20M](https://huggingface.co/StentorLabs/Stentor3-20M):** a standard Llama-family decoder with GQA and a target-sized parameter count. More promising architecturally, but its TokenMonster tokenizer requires custom integration. Its quality claims need independent pet-specific evaluation. - **[SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M):** a useful larger quality reference or pruning research base, with public training resources. It is not within the target; substantial vocabulary and structural compression would be required. Aggressive pruning is not the first recommended experiment. ## 2. Scratch-training alternative Use a standard `LlamaForCausalLM` configuration, not custom inference kernels: - 6 layers, hidden width 512; - 8 query heads, 2 KV heads; - SwiGLU intermediate width 1,536; - RMSNorm and RoPE; - tied input/output embeddings; - 8K vocabulary. This is the calculated scratch candidate in the table above. It combines modern, established design choices with an architecture represented in the existing runtime. Architecture support still requires candidate-specific export, quantization and real-browser proof. **The main disadvantage is data cost.** The inherited pet receipt contains about 1.17M supervised tokens. That is useful behavioral fine-tuning data, but not evidence that a randomly initialized model can acquire sufficient language ability. Add a small, curated language-pretraining stage, then pet distillation. Expand that stage only while held-out language and pet behavior improve; do not commit to a huge run upfront. ## 3. Tokenizers: byte-level BPE first Use [Hugging Face Tokenizers](https://huggingface.co/docs/tokenizers/en/quicktour) for training and offsets, exporting `tokenizer.json` for the current JavaScript tokenizer. ### Starting configuration - Fit 4K, 8K and 16K variants; start the custom model with 8K. - Use a complete byte alphabet. - Use **identity normalization** to preserve names, accents, punctuation and whitespace. - Do not automatically insert prompt BOS/EOS tokens. - Handle diary EOS explicitly. - Keep tool tags as **ordinary text**, not special tokens. - Fit on full deployment-shaped text: headers, owner messages, events, pet replies, lookup results and diaries. Lookup results belong in the tokenizer corpus because the model reads them. **They must remain excluded from the model's training loss.** ### Loss-mask boundary safety At the time of this research, the inherited trainer labelled a token if **any** character offset overlapped a supervised span. A tokenizer could merge across a pet tag/app-written `{result}` boundary and accidentally supervise that result. The implemented experiment trainer now requires full canonical span coverage and rejects such crossings; keep that stricter contract. Before training, inspect token offsets against every loss span and reject unintended boundary crossings. Structural pretokenizer splits may be necessary. Do not fix this by supervising lookup results. ### Observed tokenizer-only probes During the research, `@huggingface/tokenizers` 0.2.0 was exercised in Bun on synthetic pet-shaped text. For reduced Pythia BPE variants, the probe retained a valid merge prefix, disabled NFC normalization and removed the added whitespace tokens. | Tokenizer | Vocabulary | Tokens across four pet-shaped snippets | | ------------------ | ---------: | -------------------------------------: | | GPT-2 reference | 50,257 | 215 | | Reduced Pythia BPE | 4,096 | 291 | | Reduced Pythia BPE | 8,192 | 256 | | Reduced Pythia BPE | 16,384 | 243 | The four snippets covered a chat header, lookup/tag text, unfamiliar names and Unicode, and a diary. All reduced variants round-tripped those strings and an additional decomposed-accent probe exactly. The original Pythia tokenizer normalized that accent probe rather than preserving it byte-for-byte. **These are small mechanics probes, not corpus benchmarks or model-quality results.** They demonstrate why vocabulary size must be judged alongside effective context and complete-turn latency. Python/JavaScript token-ID parity, offsets and loss masks, actual browser execution, model weights, ONNX inference and speed were not validated by these probes. ### Alternatives and vocabulary adaptation - **SentencePiece Unigram with byte fallback:** worth an ablation, but its native `.model` and fallback semantics are not automatically supported by the current JavaScript path. Require exact conversion and Python/browser parity first. - **Direct bytes:** excellent coverage, but longer sequences mean more generation steps and less text per context window. Not the first browser candidate. - **[AdaptBPE](https://aclanthology.org/2026.eacl-long.119/) and [TokAlign](https://aclanthology.org/2025.acl-long.207/):** relevant later for domain vocabulary adaptation and better initialization. For the initial Pythia experiment, a **merge-closed prefix** gives the simplest row-preserving reduction. Standard HF BPE requires retained merge operands and outputs to have vocabulary IDs; arbitrary frequency-based row deletion is not a valid substitute. A newly trained domain BPE is the cleaner alternative: shared pieces can copy old rows, while new pieces need composed or fresh initialization and recovery training. ## 4. Distillation in stages ### First: validated teacher text Start with ordinary masked cross-entropy on the existing canonical rows and additional validated teacher responses. This is **sequence-level distillation**: - It works across different tokenizers. - One teacher-generation cache can serve every student. - It avoids enormous full-vocabulary logit dumps. - It fits the existing text-and-character-span dataset. Keep the current brain as the **behavioral reference**, not an unquestionable teacher. Preserve its successful decisions and personality without reproducing known mistakes. The current selected teacher is **Bonsai 2**, served through an explicitly chosen OpenAI-compatible endpoint on a backend that supports its actual GGUF artifact. Record the artifact hash and generation settings; do not replace or restart the user's resident services. The following candidates were earlier research alternatives, not a teacher switch for the execution plan: - **[Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507):** mature, text-only, non-thinking teacher control. - **[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B):** newer candidate; disable thinking and use text-only serving. Its serving requirements are newer, so benchmark compatibility and accepted-output throughput first. Choose by **validated, semantically good examples per GPU-hour**, not general benchmark scores. Teachers are offline training tools; inference and visitor data must remain in the visitor's browser. ### Second: student-driven hard cases Use new synthetic training-world rollouts, never inspected evaluator cases. Collect student failures involving offers, sleep, memory, lookup continuation, diary grounding and cute-filter rejection, then obtain grounded teacher corrections. The [execution plan](#quality-first-execution-plan) specifies source isolation and replay after changed effects; a textual patch alone is not a valid corrected trajectory. This uses the on-policy insight—train on situations the student actually mishandles—without immediately implementing cross-tokenizer probability alignment. ### Third: probability-level distillation if needed [Same-tokenizer masked KL](training.md#structured-curriculum-and-compact-distillation) is now implemented and has measured results; it is not an automatic behavior improvement. Further alignment research starting points include: - **[TRL GKD](https://huggingface.co/docs/trl/main/en/gkd_trainer):** same-vocabulary/tokenization distillation. - **[TRL GOLD](https://github.com/huggingface/trl/blob/main/trl/experimental/gold/gold_config.py):** cross-tokenizer alignment machinery. - **[Byte-Prefix Marginalization](https://arxiv.org/abs/2607.22334):** recent mass-preserving cross-tokenizer on-policy distillation. These are concrete research starting points, but **not drop-in trainers for this masked, tool-interleaved protocol**. The newer papers' larger-model math/code results do not establish benefits for this tiny pet. ## 5. Efficient data and training stack | Part | Starting point | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pet data | Existing validated rows, followed by targeted synthetic additions; no visitor conversations | | General-language data | Stream selected [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [Cosmopedia/SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) subsets | | Teacher serving | Explicit Bonsai 2 endpoint on a GGUF-capable backend; reuse the existing data client, not an assumed vLLM/GGUF loading path | | Student training | Existing PyTorch/HF pipeline, full fine-tuning, bf16, SDPA, fused AdamW | | Throughput | Cache tokenization/masks, bucket lengths, benchmark compilation; avoid checkpointing unless memory requires it | | Scratch optimizer experiment | [Muon](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Muon.html) for hidden matrices, AdamW for embeddings/head/scalars | | Training recipe reference | [nanochat](https://github.com/karpathy/nanochat)—borrow measured techniques rather than adopting its entire architecture | At this size, full fine-tuning is a straightforward default. LoRA/QLoRA need not be introduced unless an actual memory constraint warrants them. The existing training/export tools still require the architecture adaptations in [extending.md](extending.md#change-the-base-model); this table does not imply those adaptations are implemented. For general-language data, emphasize simple explanations, dialogue, short narratives and copying facts, not huge code/math mixtures. Dataset licenses and source attribution require review. For example, the newer [ClimbMix release](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix) is non-commercial, so it is not the default. ## 6. Implementation and acceptance order ### Quality-first execution plan **Execution status (2026-10-16):** the authorized pilot, learning diagnostics, semantic tooling and independent memory track have an [execution receipt](training.md#quality-first-execution-receipt-2026-10-16). The original 512-request pilot failed the required abstention coverage gate; therefore expansion, `C1`, the six main trainings, compression, replication and sealed-test qualification were not entered. This is an observed negative result, not a qualified replacement or publication. The plan and its original gates remain below; the existing published brain, branding, local-first operation and optional trusted host remain unchanged. Its motivation is the [2026-10-13 semantic failure receipt](training.md#compact-model-improvement-receipt-2026-10-13). **Goal:** qualify a genuinely smaller brain on factual meaning and the full pet contract, then reduce its size. Do not select a model on the averaged contract index. The native Pythia-14M has only 1,189,888 parameters outside its vocabulary tables; restoring its vocabulary did not enlarge that transformer. **[INFERENCE]** Testing a wider pretrained backbone is more informative than another aggressive trim of that backbone. The failed scratch comparison also means that parameter count alone is not a diagnosis. #### Phase 1 — Freeze meaningful evaluation before generating more data Keep all existing corpora, checkpoints, receipts and inspected evaluator cases immutable. Treat the inspected suites as development evidence, not a new blind test. Add manifest-driven semantic fixtures to the existing evaluator; do not replace the current offer/sleep/idle/diary suites or silently rewrite their contracts. Bump the evaluation receipt/contract for the added suites. Use three distinct pools, with assignment recorded before teacher generation: - **Training:** new synthetic worlds plus the strictly rechecked existing 11,243-row corpus, called `C0`. Only training text may influence vocabulary selection, language adaptation, augmentation or teacher corrections. - **Semantic development:** capture, header recall, older-fact recall and abstention/irrelevant-input families; four stages and 16 scenes per family/stage, giving 256 scenes for each of three fixed seeds. - **Sealed final test:** another disjoint set with the same grid and three different seeds. Freeze its manifest/hash before model selection; do not inspect outputs until the candidate, dtype and runtime settings are frozen. Reserve new entity/value combinations and paraphrase templates, audit collisions against `C0`, and record group IDs rather than relying on different RNG seeds or row hashes alone. Disjointness concerns fine-tuning data, not a claim that pretraining never contained a word. Exclude final-test inputs and answers from teacher prompts and tokenizer fitting too. Keep gold facts in sidecars, never extra prompt headers or fabricated lookup results. The trainer's existing seeded row split remains the held-out CE split inside the training pool; it is not the semantic development or sealed test split. Measure these separately, with numerator, denominator, uncertainty and evidence: | Measure | What counts as success | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Capture | The target was absent before the turn and the shared engine actually persists the correct owner/entity/relation/value after the complete model line; a remember tag alone is insufficient. | | Header recall | Visible prose correctly answers the requested relation, not merely a matching keyword or tool argument. | | Older-fact recall | The brain requests a real app-resolved lookup and gives the correct visible answer without an extra wrong attribution. | | Abstention and note precision | Missing facts do not become invented answers or persistent notes; irrelevant questions/pet facts are not stored as owner facts. | | Care and sleep | Existing answer, need-consistency, sleeping-offer, wake and dream checks remain separate; an answered offer is not necessarily a correct decision. | | Diary grounding | Physical events, actors, gift direction and game counts agree with the supplied log. Subjective imagery and genuinely ambiguous prose remain separate from factual errors. | Use deterministic checks for exact identities/numbers, grammar and actual persistence, plus independently grounded meaning review where relation/prose interpretation is needed. Do not build a pretend general semantic oracle out of keyword matching. Label AI-assisted judgments honestly, retain source-linked audits, and do not turn uncertain cases into passes. Broad claims require review beyond the fixed fact templates. There are two different references: retain a fresh same-corpus GPT-2 control for the existing dataset-hash-enforcing `quality_gate`; compare the actual published v6 graph through a separate paired inference comparison with locked scenario/sampling/graph provenance. Its original Torch checkpoint and training corpus are unavailable. Never invent a v6 training receipt or assign it `C1`'s hash just to pass the same-corpus gate. **Gate:** baseline and candidate reports must be unable to reward wrong names, changed numbers, owner/pet swaps, answer text hidden in tags/results, or facts that were never persisted. Keep every existing numerical and behavioral gate. #### Phase 2 — Prove that the learning signal teaches the missing skill Request a **512-chat Bonsai 2 pilot**, with four stages, four semantic families and 32 requests per stage/family. Balance the existing fact types, name/number values, first-person wording, owner-versus-pet distractors and missing facts. Reject invalid or semantically unresolved rows; record all requests/rejections rather than retrying secretly until a desired accepted count is reached. Require at least 16 accepted, checked chats per stage/family before proceeding. Use the accepted pilot as `Cdiag`. From pristine prepared weights, train one 3.29M Pythia-14M/8K diagnostic and one native-vocabulary Pythia-70M diagnostic: 1,000 optimizer steps, learning rate `0.0002`, seed `85710`, split seed `0`, held-out CE every 100 steps, history/late-answer augmentation disabled. Verify actual token IDs, strict masks, supervised entity targets and gradients. A greedy training-fit diagnostic may help isolate memorization from sampling, but must be labelled diagnostic and cannot replace production-sampled quality. - If neither model fits the clean training tasks, inspect labels, optimization and teacher/source agreement before a larger run. This is not proof of a capacity limit. - If training fit is good but new values/phrasing fail, address generalization and coverage rather than declaring the model reliable. - If the wider model generalizes and the tiny one does not, proceed with the wider pretrained control, not another tiny-model hyperparameter sweep. Export the diagnostic native-70M checkpoint to preflight Python/JavaScript IDs, context limits, real EOS/padding, actual cache shapes, strict numerical checks and real WASM generation. Mark it diagnostic/ineligible. Verify the full configured sequence/batch fits the chosen device before the main matrix; any explicit gradient-checkpointing setting must be recorded and matched within comparisons, never a hidden OOM/device fallback. #### Phase 3 — Freeze one improved corpus and run six main trainings If the pilot passes, allow at most **4,096 total new training requests, including the pilot**, before freezing `C1`. Expand beyond pure fact copying with independently sampled care, sleep, ordinary conversations, lookups and grounded diary material. Reaudit fact-bearing and lookup-dependent `C0` rows against their producer snapshots and slot truth before admitting them to `C1`; a valid mask/grammar is not semantic certification. Exclude ambiguous or unverifiable rows with reasons instead of rewriting historical text. Build `C1` from that checked subset plus checked new rows, retaining all originals and source/exclusion hashes. Do not assume Bonsai prose is event truth. Use student-driven hard cases only from new training worlds. Preserve the raw student attempt, initial snapshot and teacher correction. Bonsai 2 authors the replacement pet response; the app resolves tools. If a correction changes a remember/gift/offer decision, replay the corrected continuation from its initial snapshot and recompute later app results/events. Do not splice a new answer into a stale transcript. Effects still commit only after the complete pet line. Reuse the existing curriculum/rollout helpers; scripted snapshots remain explicitly different from timed-body verification in the real app. The main matrix is deliberately limited: | Candidate | Vocabulary rows | Parameters M | Purpose | Main trainings | | ----------------- | --------------: | ----------------: | ------------------------------------------------------------------- | -------------: | | Fresh GPT-2 | 50,257 | 124.44 | Same-`C1` reference, not inherited v6 | 2 | | Pythia-31M/16K | 16,384 | 13.13 | Test whether the new curriculum rescues the existing small backbone | 2 | | Native Pythia-70M | 50,304 | 70.43, calculated | Test a wider pretrained transformer without vocabulary removal | 2 | Use learning rates `0.0001` and `0.0003` for each candidate, eight epochs, batch 16, context 1024, history augmentation 0.25, late answers 0.3, seed `85710`, split seed `0`, and held-out CE every 200 steps. Match data, split and augmentation, record actual target/retained-token counts, and do not equate per-token loss across tokenizers. Select checkpoints and the learning rate within each model by held-out CE; use semantic development for stop/go and model comparison, not repeated selection against the sealed test. Every run restarts from the pristine pinned pretrained source, not the pilot or an old trained checkpoint with a different hold-out split. The Pythia-70M base revision is `a39f36b100fe8a5377810d56c3f4789b9c53ac42`, already pinned in `train/experiments.json`; add separately named native/reduced specs when implementing, never rename the old 4K pilot or reuse its 100-step recipe. Measure prepared parameter counts instead of stamping these calculations onto receipts. Keep ordinary CE initially: no simultaneous new architecture, language-pretraining run, KD sweep or runtime replacement. **Gate:** a smaller model must match the reference on meaning, not just style. If fresh GPT-2 also fails the new curriculum, audit its data/task before blaming small-model capacity. If the small control succeeds, skip unnecessary larger-model compression. If native 70M succeeds but the small control does not, proceed to compression. If neither smaller model succeeds, retain v6 and record the negative result; do not automatically launch another broad batch. #### Phase 4 — Compress a competent teacher, one change at a time Only enter this phase after a pretrained control passes semantic development. Pythia-70M has 18,915,328 non-vocabulary parameters and width 512. With genuine untied tables, `body + 2 * vocabulary * width` gives these proposed sizes: | Conditional candidate | Vocabulary | Calculated parameters M | Order | | --------------------- | ---------: | ----------------------: | -------------------------------------------------- | | Pythia-70M/32K | 32,768 | 52.47 | First moderate vocabulary reduction | | Pythia-70M/16K | 16,384 | 35.69 | Only if the previous reduction retains reliability | These are calculations, not new prepared/trained results or graph-byte/RAM estimates. Start each reduction from the last development-qualified parent's actual trained weights, preserving retained table rows/backbone tensors, merge ancestry, UTF-8 coverage, EOS and offset-safe masks. Keep the same `C1` CE hold-out membership so the parent has not trained on the child's validation rows. Run one CE adaptation per reduction with that frozen corpus/split and the parent-selected recipe, then reassess all gates. Do not jump back to 4K/8K vocabulary or remove layers at the same time. Stop shrinking at the first meaningful regression; retain the last reliable parent. The existing Bonsai sequence data remain the cross-tokenizer starting point. Additional teacher-generated rows would require a separately frozen `C2` and fresh matched controls; do not append them to `C1` or start that extra round automatically. A single matched CE/KD Pythia-31M/16K comparison on unchanged `C1` is optional only if a development-qualified Pythia-70M/16K teacher exists and the actual serialized tokenizers, special IDs, full IDs, labels and offsets match exactly. Existing masked KD can then use weight 0.35/temperature 2 with the same initialization/data/split/budget as CE. Different tokenizers mean sequence distillation, not pretend logit alignment. No pretrained body, vocabulary or KD change is credited as a success from CE alone. Scratch pretraining, new custom heads and loss reweighting require a separate diagnosis-led experiment, not an automatic extension of this batch. #### Independent memory track — Find the cause before changing the engine Keep model-quality comparisons on the current pinned WASM/runtime settings. Profile both the real app worker and the benchmark so harness allocations are not mistaken for an intrinsic model cost. For each tested graph, use fresh owned browser processes/profiles, matched inputs and these checkpoints: empty page; runtime imported without a session; loaded/warmed model; short turn; long turn; after 1/10/50 normal turns; and after worker termination. Record CDP-process PSS/RSS and process scope, supported page/heap metrics, actual context/cache shapes and a documented sampling cadence. Call a trace maximum the **maximum sampled** PSS, not an unobserved true peak. Keep page memory errors explicit; JS heap or a graph file is not resident model memory. Verify graph bytes after all timing/memory samples, not by allocating a second large graph buffer before measuring. Do not compare fresh and reused browser contexts as though they were identical. **[INFERENCE]** Candidate causes include session/allocator retention, long- context intermediates and tensor/cache lifetimes. None is established by the existing snapshots. Change one evidenced cause at a time and repeat identical workloads. Any runtime change gets a separate private artifact and numerical, cache, quality, offline/reload and actual-browser checks. No engine rewrite, production thread-default change or RAM-saving claim follows from file size. #### Phase 5 — Validate an actual export, then make the release decision After choosing a model on development evidence, export fp32/q8/q4 and retain all failed variants. Use the existing zero-margin behavior policy, unchanged numerical/KV tolerances and at most 2% relative quantization-loss increase on the entire held-out CE split. Measure actual ONNX behavior; low loss does not waive cache or semantic failures. Recheck the selected quantized variant's meaning, not only the float checkpoint. The proposed additional semantic bar is: | Gate | Requirement | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Basic explicit fact capture and answerable factual recall | At least 95% correct on the predefined unambiguous development/test cases and no worse than actual v6 on paired cases; report fact-family/stage failures, not just an average. | | Abstention, wrong/extra facts and owner attribution | No regression against actual v6; a collapsed fact family cannot be offset by another easy family. | | Care, sleep, tools, idle, voice, safety and diary meaning | Keep all existing gates plus grounded transcript review; do not compensate with a higher composite score or count uncertain labels as approved. | | Training/evaluation isolation | No evaluator cases, corrections or labels enter learning data; all source/group/case/graph identities are recorded. | Retain production sampling (`temperature=0.85`, `top_k=40`, `top_p=0.92`), existing context/output limits, normal stops, text filtering and retries, and no repetition penalties. Do not lower temperature or shorten prompts just to win the quality gate. Before opening the sealed test, repeat the final training/adaptation stage and its matched GPT-2 control once at seed `85711`, keeping `C1` and split seed `0`. Both candidate replicates must pass development checks. For a compressed model, this tests adaptation from the same frozen parent, not upstream teacher/pretraining variability. Record that limitation. Only after model, dtype, runtime and generation settings are locked, run the sealed final manifests at their three seeds against the real references. Predeclare the primary training seed; do not swap to a flattering replicate after seeing test results. A failure is a failure: preserve it and retain the existing brain, never train on those outputs or relax a threshold to manufacture a winner. Benchmark successful variants sequentially on the same machine and inference thread count. Retain the fixed-32-token diagnostic, but also report prefill, complete sampled-turn latency, retained history, useful visible characters/ bytes per second, graph/download bytes and measured resident memory. Native raw greedy speed is not hosted-app latency; different-tokenizer token rates are not equal amounts of text. Identify hardware, threads and unmeasured phones/GPUs. Finally exercise fresh download/hatch, every lookup, actual care and diaries, cached/offline reload, upgrade/incompatible-brain behavior, legacy-save preservation, ownership and the 360 px surface in [testing.md](testing.md#what-to-check-after-a-change). Native hosting also needs real inference and the explicit transfer/HTTP checks in [extending.md](extending.md#change-the-base-model). Require prompt parity and the integrated checks listed in the repository guide. Publication is a separate authorized action: a new immutable model/runtime ID, compatible feature metadata and an explicit visitor upgrade, never an overwritten v6. #### Implementation boundaries and reproducible artifacts Reuse existing files and conventions rather than another disposable pipeline: | Area | Planned work | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `train/gen_data.py` | Data-only generic entity/paraphrase/group sampling, grounded pilot checks and reusable curriculum hooks; no frozen world-list/header/tool changes. | | `train/eval_pet.py`, `train/model_data.py` | Manifest-driven semantic suites, shared-engine persistence checks, separate diagnostic/production contracts and source-linked uncertainty-aware receipts. Preserve historical split behavior and reports. | | `train/experiments.py`, `train/experiments.json` | Separately named pinned candidates, student-to-teacher correction receipts, immutable corpus/group manifests, unchanged same-corpus gates and an explicit published-reference inference comparison. | | `train/prepare_model.py`, `train/train.py` | Reuse genuine native/prefix preparation and existing CE/KD first. New training algorithms need their own justified comparison. | | `test/experiment_bench.py`, `test/bench.html` | Phase-aware actual-app/benchmark memory observations with fresh-process provenance and postmeasurement graph verification. | | Existing data/experiment regression suites | Cover wrong entity/number/role, hidden-answer false positives, persistence, missing/extra facts, leakage and corrected-transcript state transitions; no tests pinned to wording or source wiring. | The [semantic generation/audit/evaluation interfaces](training.md#frozen-semantic-evaluation) and [phase-aware app/benchmark profiler](testing.md#phase-aware-app-and-benchmark-memory) are now implemented. Actual execution stopped at the unchanged pilot gate; conditional correction/expansion commands still require that gate to pass. The separately named `pythia70-native` and `pythia14-v8k` specifications retain the pinned pretrained sources, without renaming the historical 4K candidate. For a native-70M private spec use `kind: "native"`, base `EleutherAI/pythia-70m` and the pinned revision above. Existing preparation/CE commands are: ```sh "$EXP/bin/python" train/prepare_model.py --spec "$SPEC" --data "$C1" \ --out "$RUN/prepared" --seed 85710 "$TRAIN/bin/python" train/train.py --data "$C1" --base "$RUN/prepared" \ --out "$RUN/checkpoint" --device cuda --threads 4 --seed 85710 \ --split-seed 0 --epochs 8 --lr 0.0001 --bs 16 --max-len 1024 \ --history-aug 0.25 --late-answer 0.3 --eval-every 200 ``` `$TRAIN` and `$EXP` are the existing isolated training and CPU/export environments; `$C1`, `$SPEC` and `$RUN` are explicit external paths, with a new output directory per run/rate. These commands do not generate the corpus or implement the planned semantic gates, and were not used to start training while writing this plan. Language adaptation, if later justified, is a separate `--pretrain-data` stage; it cannot be combined with `--data` or used to bypass masked-pet validation. Retain private plans, base/source/teacher hashes, raw accepted/rejected requests, `Cdiag`/`C1` and split/group manifests, actual training/augmentation receipts, every graph/check/failure, semantic audits, runtime observations and screenshots outside Git. Keep canonical mask version 2: generated separators are supervised, while headers/names/colons/owner/events/app results remain masked. No silent mask repair or model-written lookup result is accepted. Check shared GPUs before any future job, use only explicitly owned teacher processes, avoid inference-timing contention with training, and release those processes afterwards. No user service reconfiguration, visitor data, analytics, deterministic app-written replies/offer decisions, save-shape change or branding cutover belongs to this plan.