machine
Parametric learning in your code harness.
Star
Fork
Watch
machine examples project-euler docs EXPERIMENTS.md
6.3 kB · 134 lines
Markdown
at letta-code

Experiment plan #

Phase 0 — infrastructure validation #

Goal: prove importer, isolation, scorer, and reports work without training.

  1. Import the local ZIP.
  2. Run one rollout for problem 1 in each language through the pinned Docker boundary.
  3. Verify the answer manifest never enters workspaces.
  4. Inspect raw event streams and executed programs.
  5. Confirm inline execution, execution of an unrelated file, nonzero exits, output embedded in error text, and blocked escape attempts all fail functional scoring.

Exit criterion: deterministic tests pass and the three language runtimes execute.

Phase 1 — baseline frontier #

Model: Qwen/Qwen3.5-4B or a Tinker inference model available in Pi.

  • 8 canonical rollouts per language.
  • Fixed thinking level and timeout.
  • Two consecutive rounds required.
  • No gradient updates.
  • Record first exposure to every problem.

This establishes forward-transfer and contamination baselines.

Phase 2 — exploration study #

At the first failed frontier, run strategy-conditioned fan-outs:

  • mathematical derivation;
  • brute force then optimize;
  • tests first;
  • recurrence/dynamic programming;
  • readability.

Compare correctness, execution, latency, and token use. Do not use strategy prompts for the canonical unlock gate.

Phase 2b — tool-protocol bootstrap #

If a target model can solve native prompts but does not call Pi tools, run synthetic bootstrap SFT before GRPO:

  1. Generate and inspect data/tool-bootstrap.jsonl.
  2. Execute every final synthetic program and compare its output to the recorded tool result.
  3. Decode all renderer inputs and inspect trainable masks.
  4. Train one full-prefix example per assistant turn with LAST_ASSISTANT_MESSAGE.
  5. Register the sampler and require actual Pi file writes and language execution on unrelated smoke tasks.
  6. Only then evaluate the Euler frontier through Pi.

Do not substitute automatic observations for approved training examples. Do not include Euler answers in protocol bootstrap data.

Phase 3 — GRPO smoke test #

Use Qwen/Qwen3.5-4B, group size 4, two optimizer steps, and the current problem. Validate:

  • non-zero reward variance (stop or change the task/budget if groups are all-good or all-bad);
  • no renderer/parser errors;
  • plausible rollouts;
  • checkpoint creation;
  • no severe KL spike;
  • post-checkpoint Pi evaluation still runs.

Do not interpret a two-step run as an improvement result. Use a token budget large enough for the model to reach its boxed answer. Qwen3.5-4B with the default thinking renderer saturated both 1,024 and 4,096 tokens; 8,192 was the first tested budget to produce fully formatted responses. The non-thinking renderer was faster but still saturated 1,024 tokens for most language arms.

Phase 4 — controlled GRPO #

Suggested starting configuration:

group_size=8
groups_per_batch=6
learning_rate=4e-5
max_tokens=4096
steps=20
languages=python,javascript,julia

Evaluate the base and checkpoint with identical Pi settings. Compare current frontier, previous-problem regression, execution rate, and token cost.

Phase 5 — replay curriculum #

Add replay groups after more than one problem is mastered:

70% current frontier
20% recent mastered
10% older mastered

Ablate replay ratios and measure forgetting.

Phase 6 — sandbox-backed tool RL #

Run preflight_tool_rl.py without network access, then against a loopback SandboxFusion service. Require all bundled Python, JavaScript, and Julia reference implementations to pass before model sampling. Run a two-step tool-GRPO smoke from the strongest tool-SFT checkpoint and verify mixed hidden-case rewards, real tool calls, checkpoint creation, and an unchanged frozen evaluation split.

Do not compare this path with answer-only GRPO solely on training reward. Compare the fixed internal validation variants, functional gate v2, the frozen unrelated transfer suite, token cost, and sandbox failures. Use an untouched external task set for generalization claims.

Freeze the scorer before sampling. A review of preserved trajectories found that an early functional-v2 draft allowed a dummy write plus inline answer command and did not require an explicit zero exit. Corrected retrospective scoring is useful diagnosis but cannot promote a checkpoint. The next promotion attempt must use the corrected scorer from the start.

Preserve evaluation ordering: a candidate that fails the complete agent gate does not advance merely because an internal validation arm improved. A blocked escape attempt also fails the trajectory; successful containment is not successful behavior.

Phase 6b — bounded Pi-shaped remediation #

Use this only after the full Pi and transfer receipts identify a file/shell or language-specific weakness.

  1. Generate disjoint file-task train/evaluation sets and freeze both digests.
  2. Match Pi's read, write, edit, and bash schemas.
  3. Weight the weak language without removing Python/JavaScript replay.
  4. Cap optimizer steps, tool calls, turns, and generation tokens before invoking Tinker.
  5. Require positive held-out behavior in the weak language before external evaluation.
  6. Run corrected functional gate v2 from a fresh state file.
  7. Reject the checkpoint after the first failed round; do not spend a second round to make the failure look ceremonial.

Compiler errors and normal program output may be model-visible in this phase because repair behavior is the target. Keep hidden-variant evaluation aggregate-only; the two environments answer different questions.

Phase 7 — anti-memorization evaluation #

The first Phase 6 scaffold incorporates this requirement instead of postponing it. It trains on deterministic hidden variants of the families behind problems 1 and 2, keeps evaluation seeds separate, and rewards multiple hidden instances. Extend the family set and add untouched external code tasks before making generalization claims.

Required run record #

Every result should preserve:

  • git revision;
  • dataset manifest and license;
  • model/base/checkpoint;
  • renderer and thinking level;
  • complete config;
  • prompt/version hash;
  • gate definition;
  • random/sampling seeds where available;
  • raw responses and tool events;
  • metrics and checkpoint paths;
  • known failures and manual exclusions.