machine
Parametric learning in your code harness.
Star
Fork
Watch
machine examples minimal-sft
2 folders · 4 files
docs
tests
.npmignore
make_corpus.py
README.md
run-step.sh
README.md

Minimal end-to-end: tag, validate, SFT, evaluate #

This is the smallest complete loop through Machine: collect judged examples, validate them, run a real Tinker SFT, and measure a held-out baseline-vs-checkpoint delta with /train eval.

Where project-euler is an evaluation-first research harness, this example demonstrates the interactive core: the loop a single person runs in one afternoon to teach a student model a house style.

What is being taught #

A strict answer style for bare natural-language arithmetic questions:

What is 47 plus 15?
answer: 62

Every reply is exactly answer: <integer> — lowercase prefix, single space, bare integer, no prose, no period. The questions never state the format, so the style can only come from the tagged examples. That is the same shape as tone learned through /good in a real project, compressed into something exactly gradeable: eval rows use "match": "exact", so a baseline model that answers naturally (47 + 15 = 62) scores zero and a checkpoint that learned the style scores by arithmetic correctness.

This example teaches format compliance, not mathematical capability. The arithmetic is trivial on purpose.

Two ways to build the dataset #

A. Tag real turns (the product flow) #

Work in this directory with Machine installed, then judge turns as you go:

What is 47 plus 15?
answer: 62

/good terse house style

Useful variations while collecting:

/good --as "answer: 62"      rewrite an answer that was right but chatty
/bad too much explanation    keep the rejected shape out of SFT
/good --trace                (in a coding session) capture the full tool trajectory
/tagged                      review what has been saved

/good rows land in this project's Machine store (/machine data shows the path); /train sft uses that store by default.

B. Generate the deterministic corpus (what the documented run used) #

For a reproducible example with no interactive steps:

python3 make_corpus.py

This writes 48 self-authored train rows and 24 held-out eval rows (disjoint operand pairs) under the git-ignored data/ directory, plus a manifest with seed, counts, and checksums. Re-run with a different --seed for a new draw. Pass the file explicitly when training from a reviewed corpus that is not the store default.

Setup #

  • Machine installed as a Pi extension or Letta Code mod (see the repository README)
  • The Python environment from the repository README (tinker-cookbook in .venv)
  • TINKER_API_KEY set
  • Baseline model: thinkingmachines/Inkling-Small (all defaults)

The loop #

Run from this directory. Suffix --yes where shown when driving the commands headlessly (pi -p "/train smoke --yes").

# 1. Build the corpus (path B) or tag turns in a session (path A).
python3 make_corpus.py

# 2. Validate: JSONL shape plus tokenizer/renderer readiness.
pi -p "/train validate data/train.jsonl"

# 3. Scaffold the Tinker Cookbook SFT script for this dataset.
pi -p "/train sft data/train.jsonl"

# 4. Smoke: a real 2-step training run against the Tinker API.
pi -p "/train smoke --yes"

# 5. Create eval.py, then use the generated held-out rows.
pi -p "/train eval init --force"
python3 make_corpus.py            # overwrites data/eval.jsonl with held-out rows

# 6. Baseline the base model before any training.
pi -p "/train eval baseline --yes --data data/eval.jsonl"

# 7. Train for real. Small style datasets need several epochs: the default
#    one-epoch scaffold stops after 48 rows / batch 8 = 6 steps, which the
#    recorded run showed is not enough to move eval accuracy at all.
pi -p "/train sft data/train.jsonl --force --epochs 10 --steps 60"
python3 train_sft.py            # use the venv python, e.g. ../../.venv/bin/python

# 8. Evaluate the checkpoint and compare.
pi -p "/train eval checkpoint tinker://.../sampler_weights/final --yes --data data/eval.jsonl"
pi -p "/train eval compare eval_results/baseline.json eval_results/final.json"

In an interactive Pi or Letta session the same steps are the slash commands without the pi -p wrapper; the reports render in the UI there (headless runs still execute, but custom reports are TUI-only).

Measured result #

See docs/RESULTS.md for the recorded run: configuration, cost, baseline vs checkpoint accuracy, and interpretation.

Provenance and license #

  • Questions and answers are generated by make_corpus.py in this repository — self-authored synthetic arithmetic, no external dataset, Apache-2.0 like the rest of Machine.
  • Model outputs in the recorded run were produced through the Tinker API for this example.
  • data/, logs/, eval_results/, and the generated train_sft.py, tinker.yaml, and eval.py are local-only and excluded from git and the npm package.

Limitations #

  • Exact-match accuracy here measures style compliance first; it is not evidence of improved reasoning.
  • 48 rows / 20 steps is a smoke-scale run. It is enough to learn a format, not enough for statistical claims; the eval is 24 rows.
  • The base model's near-zero baseline is expected by construction (bare questions, strict format) — the interesting number is the delta.
  • The trained style bleeds off-distribution: the recorded checkpoint answered non-arithmetic questions as answer: Paris, answer: William Shakespeare (see docs/RESULTS.md). Content survived everywhere; the format did not stay scoped to arithmetic.