Working on subtokenling #

subtokenling is a virtual pet with a fine-tuned language-model brain. Local mode keeps state and inference in the browser; optional remote mode explicitly transfers the pet to a trusted Rust host. Both use the shared rules in core/.

Read the relevant docs first #

Area	Documentation
Architecture and source layout	docs/architecture.md, core/README.md
Optional private hosting and native models	server/README.md
Prompts, tags and lookups	docs/prompt-format.md
Simulation, saves and UI	docs/pet-simulation.md
Host page and embedding	docs/host-page.md
Data generation, training and evals	docs/training.md
Export, quantization and brain loading	docs/export-and-runtime.md
Publishing, server requirements and shared GPUs	docs/deployment.md
Tests and browser checks	docs/testing.md
Adding features or replacing the brain	docs/extending.md
Known pitfalls	docs/gotchas.md

Non-negotiable safeguards #

• Prompts are a contract with the weights. Keep train/world.py and core/src/world.rs in parity across native and WebAssembly builds. Changing headers, word lists, thresholds, event lines or tool names requires coordinated data changes, retraining and a new model id. The ## tokenling chat / ## tokenling diary markers are frozen wire text, not branding. Tags and values form a closed grammar; fix invalid training data rather than accepting invented tags.
• The brain decides; the app runs the body. No canned replies, random wandering or deterministic acceptance of offers. Do not hide care offers based on needs to force the expected answer. Without a loaded brain, eggs cannot hatch and pets doze with care disabled. Preserve the trained offer, sleep and silent-line rules in docs/prompt-format.md.
• Lookups are app-written. Stop at [clock], [look], [recall:…] or [yesterday], append the actual result, then resume generation. Mask results in training; keep recall's deterministic word-overlap search in parity.
• Preserve generation safeguards. Prompts end at Name: with no trailing space. Do not enable repetition penalties. Keep the NOT_CUTE text filter and retry in core/src/world.rs / core/src/turn.rs.
• Validate training data. Keep canonical text and correct loss coverage; exclude owner/event lines, headers and lookup results from supervision. Never bypass gen_data.recheck to get a run through. When behavior changes, update check_body in train/gen_data.py to catch stale rows; repair corrupt masks only through explicit migration.
• Protect saves and tab ownership. Unreadable or newer saves must show the bad egg without writes. Stored-shape changes require a version bump and migration tested against a real older save. Only the Web Lock owner may run or save the pet. Preserve the byte-for-byte legacy storage bridge and current-key precedence in docs/pet-simulation.md; retire legacy bytes only after a successful copy/write. The lock deliberately stays tokenling.v1 to coordinate with old tabs; current save keys use subtokenling.*.
• Published assets are immutable. Never overwrite model or runtime files in a published directory, or rename an existing artifact for branding. Ship a new id/directory. Visitors retain compatible older brains until they accept an upgrade; gate new trained capabilities by named features. Empty older after a prompt-format change.
• Replacement brains need evidence. The current WebAssembly pipeline is baseline-specific, not model-agnostic. Follow docs/extending.md to adapt training, tokenization, generation, export and quantization. Measure quality, size, speed, memory and the real-browser flow before publishing. Preserve inherited receipts as historical evidence, never as new-model results.
• Keep data private. Local pet/owner data stay in the browser. Only explicit remote setup/transfer may send the save to the owner's chosen trusted host; never log saves, prompts or access keys. No analytics or tracking beacons. Use generic names in data, fixtures and docs.

Implementation and documentation #

• Use strict TypeScript and Preact components for interactive markup and handlers, refs for canvas/input integration, and typed state/worker protocols. Do not reintroduce DOM-building controllers. Animation frames update the canvas, not the component tree. Preserve window.subtokenling debug controls; do not add an old-name alias.
• Keep explanations and build details readable without JavaScript. Check UI changes in a real browser, including horizontal overflow at 360 px; capture #pet-lcd when changing art or animation (docs/testing.md).
• Update the relevant area docs with behavior, command, number or layout changes. New files also need the layout tables in README.md and docs/architecture.md updated; durable pitfalls belong in docs/gotchas.md.
• Keep docs host-neutral and code portable: no deployment-specific paths, hosts, services or defaults. Use placeholders in docs and arguments/env for deployment inputs; document variables in script headers and area docs. Test new scripts against stand-ins. Keep datasets, checkpoints, exports and logs out of Git. Check shared GPUs before jobs and release them promptly.
• Report only observed results. Label estimates and untested behavior; speed receipts need the machine and thread count. Record each new model's own data/training/eval/quantization receipt in docs/training.md.

Before committing #

Use Node.js 24+ and npm ci for pinned frontend dependencies. The shared rules build requires Rust 1.88+, the WASM target and pinned wasm-bindgen (see core/README.md). Run:

npm run format:check
npm test
cargo test --locked --manifest-path core/Cargo.toml
cargo test --locked --manifest-path server/Cargo.toml

npm test covers strict types, prompt parity (must be 0 mismatches), the page contract, body/save tests, data rules and script lifecycle checks. Exercise changed behavior in the real app as described in docs/testing.md. Re-read updated docs against the implementation and check relative links and referenced repository paths. Build/deployment and training commands live in their area docs.