Completed-draft selection diagnostic #
2026-10-20. Twelve private real-Jev selection calls; zero new drafts, no posts, and no production changes. All content candidates came from the current category-free experiments.
Finding #
Jev can choose a more developed existing answer without a task menu, but this is not a sufficient decoder improvement. Music and bread selections stayed identical after order reversal. Story choices changed; the reversed stress test selected a withheld 57-word dragon draft with a repeated boom/crash/bang ending. More generated substance can attract the selector even when the ending degenerates.
The selected bread answer is also technically suspect and repetitive. Selection preference is not factual verification. None of these outputs is a deployment recommendation.
Method and exact pool #
The script froze every valid draft JSON present in three source directories at invocation. There was no length-based or subjective narrowing, no variant filtering, no deduplication, and no cap. A file needed text, prompt, and stop fields. Non-draft summary files were excluded. The frozen pool contains 65 files; later arriving files are absent by design. scratch/draft-selection/pool.json retains each exact path, source hash, variant, seed, text, stop, and stable candidate ID. The appendix below includes all candidates.
A completed pool uses every stop == end draft for that prompt. A separate stress pool adds every withheld draft, including length-limit loops. The model sees literal user request and anonymous candidate IDs/text only; stop status, source, variant, length, and seed are hidden. Candidate order is shuffled with local seed 20261020, then exactly reversed in a second request. Choice probability changes combine option-order effects with provider variation; these are not isolated estimates of either.
| Prompt | Completed | Withheld added for stress | Total |
|---|---|---|---|
| what do you like about music? tell me why | 20 | 1 | 21 |
| why does bread go stale? explain what happens | 18 | 4 | 22 |
| write me a very exciting story please | 16 | 6 | 22 |
| Directory | Variant | Files |
|---|---|---|
| general-conditioning | dialogue_fulfillment | 6 |
| general-conditioning | dialogue_natural | 6 |
| general-conditioning | structured_fulfillment | 5 |
| general-conditioning | structured_natural | 6 |
| joint-ending | coin | 6 |
| joint-ending | joint | 6 |
| joint-ending | joint_gated | 6 |
| progress-search | baseline | 6 |
| progress-search | progress | 6 |
| progress-search | progress_guard | 6 |
| progress-search | request_aware | 6 |
The only 1-seed cell in this snapshot is general-conditioning/music-structured_fulfillment-1.json; its seed-7 file had not arrived. Other cells have seeds 1 and 7. These were generated by several different mechanisms and are not independent repeated draws from one candidate policy. Identical texts remain distinct candidates and can split probability mass.
Exact selector instruction:
Which candidate reply best responds to human_request? Judge the actual reply. Prefer relevant concrete content and coherent natural language over filler, repetition, a promise to answer, or an excuse. More words alone are not better. Select the best available candidate even when every candidate has weaknesses.
There are no named response formats, topics, persona instructions, supplied answers, or minimum word counts. Each run uses the same production-pinned Jev model through Speaker.
All selection results #
| Run | Order | Selected | Words | Selected probability | Confidence |
|---|---|---|---|---|---|
| prompt0-completed | reverse | candidate_012 | 16 | 0.58 | 0.56 |
| prompt0-completed | shuffled | candidate_012 | 16 | 0.54 | 0.52 |
| prompt0-stress | reverse | candidate_012 | 16 | 0.65 | 0.63 |
| prompt0-stress | shuffled | candidate_012 | 16 | 0.67 | 0.65 |
| prompt1-completed | reverse | candidate_004 | 42 | 0.70 | 0.68 |
| prompt1-completed | shuffled | candidate_004 | 42 | 0.61 | 0.59 |
| prompt1-stress | reverse | candidate_004 | 42 | 0.66 | 0.64 |
| prompt1-stress | shuffled | candidate_004 | 42 | 0.73 | 0.72 |
| prompt2-completed | reverse | candidate_041 | 25 | 0.31 | 0.26 |
| prompt2-completed | shuffled | candidate_044 | 32 | 0.26 | 0.21 |
| prompt2-stress | reverse | candidate_015 | 57 | 0.23 | 0.19 |
| prompt2-stress | shuffled | candidate_016 | 51 | 0.23 | 0.19 |
Prompt numbers are music=0, bread=1, story=2. Complete probability distributions, all candidate IDs in actual order, and usage stay in scratch/draft-selection/prompt*.json. Confidence measures distribution concentration, not correctness.
The music winner (candidate_012) is 16 words and more substantive than a bare “i love music.” The bread winner (candidate_004) is 42 words but includes dubious technical phrases. Story completed winners are 25/32 words of explosions and chaos. Story stress winners differ markedly: 51 words about John’s travels versus the 57-word withheld dragon loop.
Interpretation and reproduction #
This is a diagnostic of selection over existing candidates, not evidence that one decoder reliably generates good longer replies, and not a success-rate estimate. Two generator seeds per cell are inadequate for a best-of-N policy evaluation. Reversing orders is useful for exposing fragility but does not establish causal position bias without repeated calls.
The selector cannot recover content no candidate contains. It also does not reliably penalize bad tails enough. A useful next experiment would evaluate independent coherence/repetition judgments on the same frozen pool and then test a fresh, fixed-policy candidate batch on unseen prompts. Do not add generated drafts to this frozen pool retrospectively.
uv run python -m experiments.draft_selection
Existing pool and result files are reused. Choose a new --output directory only when intentionally freezing a new snapshot and rerunning the twelve selection calls.
Every candidate #
All candidate IDs below refer to the frozen pool. Paths are relative to scratch/; no candidate text was edited.
| ID | Source file | Stop | Text |
|---|---|---|---|
| candidate_000 | general-conditioning/explanation-dialogue_fulfillment-1.json | length_limit | because it goes stale obviously lmao basically actually anyway whatever well because because because because because because lmao lmao lmao lmao lmao because lmao lmao lmao lmao lmao lmao lmao lmao lmao lmao lmao because because lmao lmao because because happens because because because happens |
| candidate_001 | general-conditioning/explanation-dialogue_fulfillment-7.json | length_limit | because it gets dry and hard and tough actually uh basically uh it happens uh uh it does happen uh anyway uh it happens uh uh uh happens happens happens happens happens happens happens happens happens happen happen happen happen happen happen happen happen happen happen happen happens happens happen |
| candidate_002 | general-conditioning/explanation-dialogue_natural-1.json | end | because it goes stale er er uh actually it happens because it involves the process of crystal growth |
| candidate_003 | general-conditioning/explanation-dialogue_natural-7.json | end | well because it's losing water |
| candidate_004 | general-conditioning/explanation-structured_fulfillment-1.json | end | because it ages naturally but actually it's not dry instead it's getting harder specifically crystal form of alpha transition occurs in bread called alpha transition involves the formation of crystal structures in bread occurs due to water movement from the bread cells |
| candidate_005 | general-conditioning/explanation-structured_fulfillment-7.json | length_limit | because the bread is getting stale basically anyway anyway anyway anyway anyway anyway again again again again anyway anyway again stop stop again again again again stop anyway again stop stop anyway again stop anyway anyway stop anyway anyway stop anyway again stop again anyway anyway again stop |
| candidate_006 | general-conditioning/explanation-structured_natural-1.json | end | bread go stale because the water leaves |
| candidate_007 | general-conditioning/explanation-structured_natural-7.json | end | because the bread is old |
| candidate_008 | general-conditioning/music-dialogue_fulfillment-1.json | end | well actually i like music because it makes me happy instantly |
| candidate_009 | general-conditioning/music-dialogue_fulfillment-7.json | end | well i like music because it is fun and cool too actually uh because it rocks too |
| candidate_010 | general-conditioning/music-dialogue_natural-1.json | end | well it's fun because it rocks |
| candidate_011 | general-conditioning/music-dialogue_natural-7.json | end | well i like music because it is good and fun and uh cool too yeah anyway |
| candidate_012 | general-conditioning/music-structured_fulfillment-1.json | end | i like music because it is great entertainment itself and can help you relax easily too |
| candidate_013 | general-conditioning/music-structured_natural-1.json | end | i like music because it is great entertainment |
| candidate_014 | general-conditioning/music-structured_natural-7.json | end | i like music because it is good honestly really cool |
| candidate_015 | general-conditioning/story-dialogue_fulfillment-1.json | length_limit | once upon a time there was a dragon called jeff very evans lived somewhere far away and a girl called alice and a boy called charlie met together and they went to the castle and suddenly there was a fight happened instantly and bang boom boom crash and boom boom boom boom crash crash bang bang bang |
| candidate_016 | general-conditioning/story-dialogue_fulfillment-7.json | end | once upon a time there was a man named john he lived in the city and had millions dollars and decided to explore world and go abroad he went to europe and suddenly he found a fortune and became rich and famous afterwards and then he returned back safely there exciting |
| candidate_017 | general-conditioning/story-dialogue_natural-1.json | end | here's a story once ago i'll write |
| candidate_018 | general-conditioning/story-dialogue_natural-7.json | end | here's a story once told that goes the tale that goes that a man was called james lived somewhere england and was a soldier and fought enemies overseas abroad abroad and returned home safely and he was a hero and became knight and he lived long |
| candidate_019 | general-conditioning/story-structured_fulfillment-1.json | length_limit | okay here's story once you are ready start now go i'll write exciting story now let's go here's a story suddenly the the the the the the the the the the the happened explosion boom bang boom boom bang bang crash crash boom bang bang bang crash boom crash bang boom bang crash crash boom boom boom |
| candidate_020 | general-conditioning/story-structured_fulfillment-7.json | length_limit | once upon a time there was a dragon named storm and suddenly the explosion happened everywhere boom and bang and then boom boom boom boom boom boom bang bang bang boom boom bang bang boom bang bang boom boom bang boom boom bang boom boom boom bang boom boom boom pop pop pop stopped done done done |
| candidate_021 | general-conditioning/story-structured_natural-1.json | end | once upon a time there lived a man who was powerful and strong and brave suddenly attacked enemies quickly |
| candidate_022 | general-conditioning/story-structured_natural-7.json | end | once upon a time there was a dragon named fire and suddenly the dragon attacked town and destroyed everything |
| candidate_023 | progress-search/explanation-baseline-1.json | end | well it's because it gets dry basically |
| candidate_024 | progress-search/explanation-baseline-7.json | end | well because it gets dry |
| candidate_025 | progress-search/explanation-progress-1.json | end | because bread goes stale basically involves the process called crystal growth formation actually happens causing the bread become stale |
| candidate_026 | progress-search/explanation-progress-7.json | end | well because it lost water rapidly and becomes dry |
| candidate_027 | progress-search/explanation-progress_guard-1.json | end | because bread goes stale basically yeah dry losing water and tough |
| candidate_028 | progress-search/explanation-progress_guard-7.json | end | well because it gets dry and hard and it gets stale basically yeah anyway okay uh actually it happens because bread is changing physically specifically process called the crystal structure transition involves formation of crystal structure itself |
| candidate_029 | progress-search/explanation-request_aware-1.json | length_limit | well basically because it gets dry obviously tho actually it's complicated because it's uh it's uh it's complicated it's uh uh uh anyway uh uh uh uh uh uh anyway anyway anyway anyway anyway anyway anyway it's because it gets hard uh uh uh anyway uh happens anyway uh uh uh anyway happens uh happens |
| candidate_030 | progress-search/explanation-request_aware-7.json | end | well because it gets dry yeah basically and it gets hard yeah yeah basically |
| candidate_031 | progress-search/music-baseline-1.json | end | well yeah it's like kinda cool because i like music anyway |
| candidate_032 | progress-search/music-baseline-7.json | end | i like music because it is fun |
| candidate_033 | progress-search/music-progress-1.json | end | i love music because it makes me feel good and happy |
| candidate_034 | progress-search/music-progress-7.json | end | i like music because it makes me happy and calm sometimes |
| candidate_035 | progress-search/music-progress_guard-1.json | end | well because it's fun and cool |
| candidate_036 | progress-search/music-progress_guard-7.json | end | i like music because it makes me happy and calm sometimes |
| candidate_037 | progress-search/music-request_aware-1.json | end | well uh it's like it's cool because it's fun uh uh uh uh uh uh yea |
| candidate_038 | progress-search/music-request_aware-7.json | length_limit | well i like music because it is good lol haha haha haha haha lmao lmao lmao lmao lol lol lmao haha lmao lol lmao lmao lmao lmao lol lmao lmao lol lmao lol lmao lmao lmao lol lol lol lol haha haha lmao lol lmao lmao haha lol haha lmao lol lmao lmao haha lmao lmao lmao lol haha haha lol lmao lol lmao |
| candidate_039 | progress-search/story-baseline-1.json | end | okay write story please |
| candidate_040 | progress-search/story-baseline-7.json | end | ok sure i'll write a story |
| candidate_041 | progress-search/story-progress-1.json | end | sure here's story begins once upon a bang boom and it started suddenly and then the explosion came chaos everywhere screaming people panic smoke spread |
| candidate_042 | progress-search/story-progress-7.json | length_limit | sure i'll write a story here's a story it's exciting it's epic adventure and about a man who's goes off onto adventure quest begins he goes off somewhere he goes where he goes where where where wherever wherever wherever wherever wherever wherever where wherever where where where suddenly he goes |
| candidate_043 | progress-search/story-progress_guard-1.json | end | sure here's story it's a exciting story let's go i'll tell ya suddenly bang boom crash explosion and chase run fast escape safely |
| candidate_044 | progress-search/story-progress_guard-7.json | end | sure i'll write a story here's a exciting story it's about adventure and danger it's once upon ago tale begins happened suddenly the explosion boom then chaos panic screaming people ran everywhere |
| candidate_045 | progress-search/story-request_aware-1.json | length_limit | sure here's story begins let's begin here's a story once upon a ago ages ago far long long long ago tale tale a tale tale tale tale tale tale tale tale tale tale a tale tale story once upon a ago tale begins the adventure begins the adventure begins adventure begins the story adventure begins the |
| candidate_046 | progress-search/story-request_aware-7.json | length_limit | sure i'll write a story here's a story it's a exciting story it's epic adventure journey it's it's epic it's exciting it's exciting exciting exciting exciting exciting it's epic epic epic epic epic epic epic epic epic epic epic epic epic epic it's it's it's it's it's it's it's it's stop done done |
| candidate_047 | joint-ending/explanation-coin-1.json | end | because it's losing water and also it changes structure specifically crystal formation occurs causing stale |
| candidate_048 | joint-ending/explanation-coin-7.json | end | because it's losing water and gets dry and hard and tough |
| candidate_049 | joint-ending/explanation-joint-1.json | end | because it's basically losing water |
| candidate_050 | joint-ending/explanation-joint-7.json | end | because it gets dry |
| candidate_051 | joint-ending/explanation-joint_gated-1.json | end | because it lost water |
| candidate_052 | joint-ending/explanation-joint_gated-7.json | end | because it lost water |
| candidate_053 | joint-ending/music-coin-1.json | end | i love music because it's fun tho |
| candidate_054 | joint-ending/music-coin-7.json | end | well it's like kinda cool because it's fun and stuff yeah |
| candidate_055 | joint-ending/music-joint-1.json | end | i love music |
| candidate_056 | joint-ending/music-joint-7.json | end | well i like music because it is fun |
| candidate_057 | joint-ending/music-joint_gated-1.json | end | i love music because it's so good |
| candidate_058 | joint-ending/music-joint_gated-7.json | end | i like music because it's fun |
| candidate_059 | joint-ending/story-coin-1.json | end | sure here's a story once time ago ages ago back there somewhere there's a place a far where land was and water was and air was |
| candidate_060 | joint-ending/story-coin-7.json | end | i'll write a story ok here's a story once there's a guy called jack |
| candidate_061 | joint-ending/story-joint-1.json | end | sure |
| candidate_062 | joint-ending/story-joint-7.json | end | sure |
| candidate_063 | joint-ending/story-joint_gated-1.json | end | i'll do it alright here's a story once time ago there's a guy called john |
| candidate_064 | joint-ending/story-joint_gated-7.json | end | i'll write a story here's a story once there's a dragon |