Why the decoder talks the way it does #
Offline analysis, 2026-10-24. No model calls; nothing was deployed. Main is
dfde1dc; heavypad runs 5e500f8, which is the joint categorical decoder
from 3014e71, not the factored decoder described in decoder.md.
The measured decoder on main has never been deployed.
The data are the ignored scratch/*/traces written by the 5e500f8
evaluations: 51,156 recorded 250-word heats, 1,711 recorded runoff decisions
and 93 recorded stopping trajectories. Those runs share the proposal heats and
finalist quota of both decoders, and the runoff Choice and T=0.4 sampler of
main's factored decoder. Reproduce with
uv run python -m experiments.decoder_math scratch/*/traces
0. What is actually live #
The four heavypad traces from 2026-10-24 (the story, unknown, then sure
and naha. to two more asks) show the deployed 5e500f8 decoder sampling at
T=1 from one joint Choice of about 179 options: 79 word pieces, 99 single
characters and END. In the story generation the raw favourite held a median
0.26, characters held 0.19 of the mass, END held a median 0.13 per step, and
the sampled piece was the favourite in 5 of 20 steps with a median probability
of 0.04. gonna, billy, happens, alright, listen, bob and alive
were ranks 10–22 while story, boy, called, lived and somewhere led.
naha. is a word piece, then a character, then a period, each at 0.01–0.02.
That is the salad: uniform-ish draws from a two-decimal tail. unknown ended
because END, competing inside the same Choice, was 0.33 after one word.
So the live problem and the main-branch problem are opposite ends of the same knob. Live samples the whole quantized tail at T=1; main's T=0.4 sampler collapses onto the favourite (section 2). Neither has a sequence prior (section 4). The cost is the same, about 48,000 input tokens and 0.7 seconds per word, because both run the same 21 heats.
1. The proposal stage is a quota, not a search #
Each step scores the 5,000 most frequent words in 20 frequency bands of 250, keeps the four highest per band, and adds words from the thread. That yields about 84 runoff options per step. Inside a heat the model puts nonzero mass on a median of 26 of 250 options; the top option has a median 0.23 and the top four hold 0.51. Every reported probability is a two-decimal value, so the fourth-versus-fifth finalist boundary sits at 0.02–0.04, where rounding and option order decide. Heat isolation already measured the order effect: reversing the options swapped two of the four finalists in every tested heat (median total variation 0.32). Finalist positions inside a heat are otherwise uniform (mean relative position 0.495), so the issue is noise at the boundary rather than a pile-up on the first option.
The quota then decides who competes. Per decision, the 52 finalists from bands 7–19 (ranks 1,750–5,000, where the content words live) receive 0.27 of the runoff mass combined; the eight finalists from the 500 most common words receive 0.31; thread words receive 0.16. Of the sampled dictionary words, 43% come from the 500 most common words and the median frequency rank is 364. Another 22% of all sampled words are thread echoes. Content words rarely reach the runoff, and when they do they face filler that always has a seat.
2. The runoff is nearly greedy, and greedy means filler #
The runoff distribution over 84 pieces has a median top probability of 0.28
and a perplexity of 13. Sharpening at T=0.4 turns that into a top probability
of 0.72 and a perplexity of 2.5; the sampled piece was the raw favourite in 69%
of decisions. The question is a fluency question (which piece most naturally
comes next), and the most natural single next word given a bag of unrelated
candidates is a discourse marker. That is sure, alright, listen,
anyway, basically, tho, lol.
Sharpening also amplifies rounding. At T=0.4 a 0.01 option versus a 0.02 option is an odds ratio of 5.7, and the 0.01 bin holds a median 0.16 of raw mass. The claim elsewhere that every option has a 0.01 floor is not what we observe (zeros are common); two-decimal rounding is.
| Runoff temperature | Median P(raw favourite) | Effective candidates |
|---|---|---|
| 0.4 (live) | 0.72 | 2.5 |
| 0.7 | 0.44 | 6.8 |
| 1.0 | 0.28 | 13.3 |
3. Stopping asks the wrong question for a rambler #
The hazard is s = f(1-u), with f the probability that the answer has already
addressed the request. A terse non-answer satisfies f immediately: hi scores
0.92–1.00 after one word, and unknown has the same shape. Along recorded
paths the mean hazard is 0.06–0.09 per word, which is a geometric length with
mean about 13 words; the observed median reply is 12 words. Rambling is what
happens after the request is answered, and under this rule that is exactly
when the reply ends.
Scaling the recorded hazards by a constant λ (the willingness to stop once done) and capping at the post length gives these expected lengths:
| λ | Median expected words |
|---|---|
| 1.0 (live) | 13 |
| 0.5 | 20 |
| 0.25 | 30 |
| 0.1 | 42 |
4. There is no sequence prior anywhere #
Every decision is a one-step judgment over an unordered bag of words. Jev
returns a calibrated judgment about options it is shown; it is not a language
model, and sharpening a judge does not make one. The vendor's jaggedness page
says chaining choices into text works poorly. This repository has about twenty
experiments, all with Jev-only proposers; none sustained coherence past
roughly twenty words. Cee's bot, whose output read as Kyle Mooney, is also a
dictionary tournament (top-50 sampling, five pairs of history, by Cee's own
account) and also produces salad. What made it funny was semantic circling
(led, lead, leading, stone, rock around a band it could not name),
which needs content words near the request in the candidate set and wide
sampling. Our quota and T=0.4 remove both.
Cost today is a median 48,535 input tokens and 0.69 seconds per word, because all 21 heats carry the full state in one request. At $0.042 per million input tokens that is about $0.002 per word.
What to change #
Two paths. They can be compared on the same prompts and seeds.
A. Jev-only, keeps "Jev picks every word".
- Shuffle heat option order per step. Free; turns the systematic boundary effect into noise. Averaging an original and a reversed order doubles heat cost and removes it.
- Replace the per-band quota with a threshold: keep every option at or above 0.03 (clear of rounding), up to the 254-option runoff bound. Filler loses its guaranteed seats; content words that score reach the runoff.
- Runoff temperature 0.7 (about seven effective candidates), not 0.4.
- END probability λ·f(1-u) with λ≈0.25, rising toward 1 as the post nears 300 graphemes so the cap is rarely hit mid-phrase. Expected length about 30 words on Bluesky.
- Optionally a 10,000-word list so words such as
frog(rank 8,410) andwizard(7,986) exist; this doubles heat cost.
Expected result: longer, more topical, still salad. Loops remain possible because nothing remembers what was said.
B. A small language model proposes, Jev chooses. This is the vendor's recommended pattern and what another builder in Cee's thread reported works best. A small base model (heavypad has an RTX 4070 Laptop with 8 GiB and a working llama.cpp CUDA build) returns its top 50 next words with probabilities p_LM; one Jev Choice over those 50 pieces returns p_Jev; sample
π(w) ∝ p_LM(w)^α · p_Jev(w), END ∝ p_LM(end)^α · λ f(1-u)
at T=1. α is the one knob: at α=0 Jev picks freely among grammatical moves and the voice is Jev's; at α=1 it is the product policy. This is the only path where coherence is structural rather than lucky, and it is about five times cheaper per word (one 50-option Choice instead of 21 heats). It changes the project's premise that Jev supplies all content, so it needs an explicit yes.
Before either path: deploy main. dfde1dc carries the factored decoder
whose measured comparison produced
here's a story: once upon time there lived a man named john. he was a spy. he suddenly ran away against the same prompts. It removes characters from the
word runoff and keeps END out of the word Choice. It is still short and
filler-prone for the reasons above, but it is not naha.. Deployment needs
Nate's authorization.
Outcome, 2026-10-24. Main was deployed, Nate added $5 to the ledger, and path A ran in two rounds; see the comparison. Round two (shuffled heats, finalists at or above 0.05, T=0.7, stop factor 0.25, a syntactic repetition filter) became the shared default and was deployed. Path B remains unstarted pending Nate's decision.
Validation needs a research allowance. The ledger holds 259,263 tokens, below the 262,144 per-request reservation, so no paid comparison can start. Eight prompts (the two live threads, a story, a greeting, an image, a list, a why question, a follow-up) times three seeds: path A at roughly 2M input tokens per 40-word reply is about $2.00 for 24 replies, twice that with two heat orders; path B at roughly 0.4M per reply is about $0.40. A $3–5 allowance covers both with the usual headroom for failures.