Parametric learning in your code harness.

README.md

machine #

Small, spec'd agent loops with an attached evaluator.

A machine is code: what it is for (its intent), how it is instructed, which model and tools it uses, a rubric that says what good output looks like, and an evaluator that assigns a grader to every criterion of that rubric. Each execution is a run, stored with its full trajectory and, once graded, a scorecard. Runs are what later layers will optimize against: fine-tuning on good runs, revising rubrics, and splitting a machine into specialized sub-machines.

New here? Start with docs/getting-started.md: install, talk to a machine, write your own, specialize it, measure and improve it, and change how a model reasons.

Using a machine #

bun src/cli.ts pirate                 # talk to it in Pi's terminal UI
bun src/cli.ts julia/fast --continue  # resume the last conversation with it

machine <name> opens the machine in Pi's interactive UI. The model sees exactly the machine's instructions and tools, with its model (Tinker ids, openrouter/..., or any Pi provider). Tools run in this process against the current directory, which is the session's workspace (commands isolated by bubblewrap, with the directory writable). Pi's own tools, your ~/.pi setup, and context files stay out. Every exchange is stored as a run with its full trajectory, so interactive use feeds the same grading and training as suites do. In the session:

/score grades the last exchange against the machine's rubric
/good, /bad mark it; /correct <note> records what should change
/machine shows the machine, its criteria and tools, and the session's spend

--eval grades every exchange as it finishes. Judges default to Inkling-Small on Tinker; set MACHINE_JUDGE_MODEL to grade elsewhere.

Primitives #

Primitive	What it is
`defineRubric`	Criteria with descriptions and weights, in their own file
`defineEvaluator`	One grader per criterion; TypeScript rejects a missing or extra one
`check(name, fn)`	Deterministic grader
`judge()`	Model grader; uses the workspace harness and judge model (the run's model unless one is set). Retries a reply without a usable score twice
`jev()`	Graders backed by Jev, TypeSafe's decision model: `noul` (yes/no), `score` (ordered levels), `choice` (options with scores). Much cheaper than a model judge; one request per run for all of a client's questions. Needs `TYPESAFE_API_KEY`
`defineMachine`	Name, intent, instructions, model, rubric, evaluator, tools
`judgePanel(rubric, ids)`	Model graders that score several criteria in one judge call; for when a call per criterion costs too much
`counterpart`	Another machine that plays the other side of a conversation (e.g. a simulated user); each case is then a whole conversation
`defineChild`	A machine that specializes a parent: inherits its criteria and graders, instructions, tools, and model, and adds its own
`Harness`	The agent loop for one task: `chatCompletionsHarness` (Tinker, Ollama, OpenRouter, any OpenAI-compatible endpoint), `piHarness` (Pi), `opencodeHarness`, `fxHarness` (libfx, Vercel AI Gateway), `scriptedHarness` (tests)
`Budget`	A spending limit charged for every run and every judge call; refuses new work once reached
`Sandbox`	Where a run's tools act. `localSandboxes` gives each run its own working directory and, with bubblewrap, isolates its commands (see below)
`Store`	Runs and named chat sessions, addressed by readable names; every write is an upsert. `diskStore` writes private JSON files
`createWorkspace`	Binds machines to a store, sandboxes, and a default harness; `run`, `chat`, `evaluate`

A machine #

machines/chatbot/
  rubric.ts    what good looks like
  machine.ts   instructions, model, and the grader for each criterion

// machines/chatbot/rubric.ts
export const rubric = defineRubric({
  answers: { description: "Addresses what the user actually asked…", weight: 3 },
  concise: { description: "Uses as few words as the question needs.", weight: 1 },
});

// machines/chatbot/machine.ts
export default defineMachine({
  name: "chatbot",
  intent: "Answer one person's questions in a running conversation, plainly and honestly.",
  instructions: "…",
  model: "Qwen/Qwen3.8-27B",
  rubric,
  evaluator: defineEvaluator(rubric, {
    answers: judge(),
    concise: check("words", (run) => wordCount(run.output) <= 150),
  }),
});

Use the cheapest grader that is reliable for a criterion: a check when code can decide it, Jev for a simple question about the text, and a judge when it takes open-ended reasoning:

const j = jev(); // TYPESAFE_API_KEY
evaluator: defineEvaluator(rubric, {
  in_french: j.noul(),                                   // the criterion's description is the question
  tone: j.score(["rude", "neutral", "warm"]),
  length: j.choice({ short: { score: 1 }, long: { score: 0 } }),
  accurate: judge(),
}),

Before relying on a cheap grader for a criterion, measure it against the machine's own grader on stored runs:

bun src/cli.ts agree coder does_task --with jev      # or --with judge

agree reports agreement, Cohen's kappa, the false-pass rate (a bad run passed: the error that poisons training data and rewards the wrong behavior) and false-fail rate (a good run failed: wasted samples), precision, bits of information per label, and the share of bad samples a filter would keep at 10% and 50% base pass rates, at several thresholds. It reuses stored reference grades, lists the runs where the graders disagree, and counts runs where the candidate is unsure (0.3 to 0.7) and should escalate to a stronger grader. Jev's scores vary slightly between identical calls, so unsure runs can land on either side of 0.5.

Nested machines live in nested directories and are named with /, for example machines/chatbot/triage/ is chatbot/triage.

Watching machines #

machine dashboard serves a live view of the workspace on 127.0.0.1:4242: what is running now and each step it takes, every stored run with its scorecard, reasoning, and tool calls, each machine's recent quality and spend, and fine-tunes under .machine/training/ with their loss curves and checkpoints. Every process that runs a machine appends events to .machine/events.jsonl; the dashboard follows it, so nothing else has to run.

Child machines #

A child machine specializes its parent within the parent's constraints:

// machines/julia/parametric/machine.ts
import julia from "../machine.ts";

export default defineChild(julia, {
  name: "julia/parametric",
  intent: "Write Julia that is parametric: type parameters let one definition serve every element type.",
  instructions: instructionsFrom(import.meta.url), // appended to the parent's
  rubric,                                          // only the child's own criteria
  evaluator: defineEvaluator(rubric, { parametric: onProgram("parametric", isParametric) }),
});

A child inherits every criterion of its parent with its grader, and cannot redefine or drop one: what the parent requires, every child is graded on. It also inherits the parent's instructions (its own follow them), tools (it may add more), and model (unless it names its own). Because the family shares inherited criteria and a model, it can be measured and trained as one:

bun src/cli.ts tree julia                 # each machine, what it inherits, what it adds
bun src/cli.ts suite julia --tree         # every machine's cases, then the inherited criteria across all of them
bun src/cli.ts export julia --tree --out julia.jsonl   # passing runs of the whole family, one dataset for the shared model

optimize on a child rewrites only the child's own instructions.md; the optimizer sees the parent's instructions as fixed.

A machine that writes its own specialist #

machine specialize has an author model (default thinkingmachines/Inkling) write a child for a focus, and the cases to rehearse it on:

machine specialize dev --focus "Python datetime and timezone bugs" \
  --model thinkingmachines/Inkling-Small --slug datetime --budget 2
machine suite dev/datetime --trials 4 --effort high --budget 2     # rehearse
machine train dev/datetime --best-of 4 --effort high --budget 3    # fine-tune on the best rehearsals, gate on held-out

The author sees the parent's job, rubric, instructions, tools, and source, and writes machine.ts (defineChild), rubric.ts (1 to 3 new criteria), and instructions.md. Nothing it writes is trusted: the child must typecheck, load as a child on the requested model, and its check graders must run on sample runs of the parent; failures go back to the author (two retries by default), and a check that scores every sample run alike, or reads only the files a run started with, gets a second look. Then it writes cases in batches over its own topic list: each case's test must fail on the starting files and pass with the author's reference fix, in the sandbox runs use, or the case is discarded. Kept cases split into cases.json and holdout.json. For dev, a child that grades with hiddenTests also gets held-back edge-case tests per case (hidden.json, never shown to the agent), which must fail alone before the fix and pass with it. m.specialize({ focus, model, budget }) is the SDK form; --reuse-cases rewrites only the child. Review what it wrote before training: a criterion that cannot be met on some cases shows up in rehearsal's failure reasons, and a scoped description fixes it.

CLI #

Runs on Bun, which executes the TypeScript directly.

bun install
bun src/cli.ts run chatbot                        # interactive, in Pi (--continue resumes)
bun src/cli.ts run chatbot --eval "Explain LoRA in two sentences."
bun src/cli.ts runs chatbot
bun src/cli.ts eval chatbot <run-id>
bun src/cli.ts suite red --harness opencode -v      # run and grade machines/red/cases.json
bun src/cli.ts compare red --harness opencode --trials 5   # working tree vs HEAD, 5 trials each
bun src/cli.ts compare red --ref <commit> --cases holdout.json --trials 3  # on held-out cases
bun src/cli.ts converse <machine> 3 --eval       # one conversation with its counterpart, printed live

Machines that talk to each other #

A machine can name a counterpart: another machine that plays the other side of a conversation, such as a simulated user with a hidden scenario.

counterpart: {
  machine: simulatedUser,                          // another machine
  until: (run) => isDone(run),                     // the conversation ends after this turn
  maxTurns: 8,
},

Each case is then the counterpart's starting point, and running it is a whole conversation: the counterpart speaks first, they alternate, and it ends when until holds or after maxTurns. Every turn of each side is an ordinary stored run (tagged with its conversation), and the machine's last turn, whose history is the whole conversation, is the run that is graded. So suite, compare, optimize, and export work on conversations unchanged, and --model swaps only the machine's model, never the counterpart's. machine converse <machine> [case] runs one and prints it as it happens; with --eval it grades both sides.

Changing a machine #

There is one copy of each machine. Change it in place and let git hold the versions:

Edit the machine (its instructions, model, or tools).
machine compare <machine> --trials 5 runs the working tree's version and the version at --ref (default HEAD) on the same cases, interleaved.
Commit if it wins; git checkout -- machines/<machine> if it does not.

The old version is loaded from git into a temporary directory. Both versions are graded with the current rubric and evaluator, so the comparison measures the change in behavior and nothing else; compare notes when the rubric has changed since the ref, or when the machine has not changed at all. Each run records its revision (a commit, or <commit>+working-tree).

Models are not deterministic, so use --trials to run each case several times: compare reports each version's mean and pass rate, and the mean difference with its standard error over cases. A difference within about two standard errors of zero is not evidence that either version is better.

Optimizing instructions #

A machine declares its instructions an optimization target by keeping them in instructions.md beside machine.ts:

instructions: instructionsFrom(import.meta.url),

machine optimize <machine> --trials 3 --rounds 2 then runs the suite, hands the failures to the built-in optimizer machine (purpose, tools, rubric, current instructions, each failing case with what the machine did and why the graders failed it, and the passing cases), runs the suite again with its proposed instructions, and keeps the proposal only if it scores higher. A kept proposal is written to instructions.md and left uncommitted: commit to accept it, git checkout to reject it. The optimizer only proposes; the measurement decides, and you approve. It may rewrite instructions, never the rubric.

A rewrite must also win by at least two standard errors (--min-z); a smaller gain is indistinguishable from trial noise and is discarded.

optimize only ever shows the optimizer cases.json. Keep a few cases it never sees in holdout.json (or pass --holdout <file>): a proposal is kept only if it scores higher on cases.json and no lower on the held-out cases. A rewrite that wins on its own failures but loses on unseen cases has memorized them, and is discarded. Without a held-out set optimize warns and falls back to cases.json alone. compare --cases holdout.json runs the same check by hand.

Training a machine in place #

A machine version is what decides how it behaves and is graded: its own instructions, its model, and its rubric. Versions are numbered in machines/<name>/versions.json, checked in beside the machine, and every run records the version that made it. The machine's files stay the truth: instructions.md, model.json (a machine that declares model: modelFrom(import.meta.url, "<default>") takes its committed model from there), and rubric.ts.

machine versions dev                      # list versions; the first call records the current one as v1
machine train dev --samples 4 --trials 3 --budget 5
machine propose dev --model Qwen/Qwen3.6-35B-A3B --trials 3 --budget 1   # another model, as a candidate
machine commit dev                        # the newest pending candidate, only if it passed the gate
machine revert dev                        # the previous live version

train never changes the machine. It samples the cases in cases.json (--samples runs each, default 0 to use stored runs), grades whatever is pending, and fine-tunes the live model's base on Tinker with the good runs (--min-score, default 1; the best --best-of per case, default 2; never a held-out case; only runs by the live model unless --any-model, which distills from other models' runs). Each step of a tool-using run becomes its own training example, with the context the model saw at that step and the tools it was offered; steps whose tool calls all failed are left out. On Tinker a run's later prompts hold its earlier steps with their reasoning, so for Inkling each step's prompt is a token prefix of the whole run (checked against the endpoint's prompt tokens), and a run is trained as one example with every good step a target, at a fraction of the tokens. Inkling's prompt carries its thinking effort: training renders at the effort runs sampled at (--effort; the endpoint's default is high, 0.9), and its learning rate defaults to 2e-4. It then runs the checkpoint and the live version on holdout.json (--trials each; stored graded runs of the live version are reused) and prints a report: the data, the losses, each criterion before and after, and the decision. The result is a pending candidate.

commit makes a candidate the live version only if it passed the promotion gate: a higher mean on held-out cases by at least --min-z (default 2) standard errors of the per-case difference, the margin optimize uses. It refuses otherwise and prints the report; --force overrides. Committing writes model.json (the checkpoint and its base model, which prices it) and a new version; revert writes back the previous one. --budget caps sampling, grading, training, and the comparison together. Training runs scripts/train_sft.py with $MACHINE_PYTHON, a Python with tinker-cookbook installed; its output goes to train.log in the run's directory under .machine/training/, where the dashboard charts it.

SDK #

The same loop from code, shaped like an agent SDK: run() streams messages and can be awaited for the finished run.

import { formatReport, machine } from "machine";

const dev = await machine.load("dev", { judgeModel: "openrouter/thinkingmachines/inkling-small:free" });

for await (const message of dev.run("The tests fail. Fix calc.py.", { files: { "calc.py": "…", "test_calc.py": "…" } })) {
  if (message.type === "tool_call") console.log("→", message.name);
  if (message.type === "result") console.log(message.run.version, message.run.output);
}

const chat = dev.session({ autoGrade: true, cwd: "/tmp/demo-project" }); // multi-turn, one working directory
await chat.send("Add a --shout flag to greet.py");
await chat.close();

await dev.grade();                                     // runs never graded, or graded under an older rubric
const candidate = await dev.train({ budget: 5, samples: 4, trials: 3 });
console.log(formatReport(candidate.report));           // the live machine is unchanged
await candidate.commit();                              // throws CommitRefused unless it won on held-out cases
await dev.revert();                                    // back to the previous version

machine.new(spec) does the same for a machine defined in code, keeping its versions under .machine/versions/. m.update({ instructions, model }) records a new version by hand, m.versions() lists them, m.pending() lists runs due for grading, and m.optimize({ budget }) returns a candidate that rewrites the instructions, committed the same way.

Example machines #

chatbot: answers questions in a running conversation. Graded on answering the question and honesty (judged), concision, and no filler (checked).
pirate: answers anything as a golden-age pirate. Graded on still answering the question, pirate voice, naming nothing modern (modern things get period names: a phone is a "talkin' slate"), and seafaring metaphor where it fits (judged), and one to four pirate emoji (checked).
red: calls its red_mentioned tool exactly once when the message contains "red" as a discrete word (any case, no letter or digit directly on either side), and never otherwise. That rule is the whole rubric, checked deterministically from the input and the trajectory; the reply text is not graded. cases.json holds near misses such as "bored", "redo", "Fred", and "I read it".
coder: writes a program with write_file and runs it with run in the run's sandbox. run refuses unless the sandbox is isolated. Graded on whether it wrote and successfully ran a file (checked from the real tool results) and whether the program's actual output does the task (judged against the tool results).
dev: a development assistant for an existing project. Lists, searches, reads, and edits files (edit_file replaces one exact snippet), writes new ones, and runs things with run. It answers questions, explains, and plans, and changes files only when asked. Graded on whether a command passed after its last change (checked from the tool results), whether it did what was asked, whether it stayed on task, and whether it acted on the request: editing only for a change, answering or asking for questions and unclear input like "test" (judged). Its cases are small Python projects in cases.json and holdout.json, including questions and requests to run and report. cd your-project && machine dev --workspace ~/code/machine.
assistant: a general-purpose assistant in the folder you start it in (cd somewhere && machine assistant --workspace ~/code/machine, or demo/assistant.sh). It answers questions, researches, reads and writes files, and runs commands, with dev's file tools and run (isolated: no network, only the folder) plus web_fetch, which runs in the machine process: http and https only, private, loopback, link-local, and internal addresses refused (on every redirect too), a 15 s timeout, scripts and styles stripped, and 20,000 characters a call with an offset to read on. It has no search tool: no web search API is configured. A sibling of dev, not a child: dev's verified and focused are about code. Graded on helpful, grounded (claims trace to tool results and cite the file or URL; judges see whole fetched pages), dev's own acts_on_request (judged), and concise (word count; an empty reply scores 0). Its cases cover file questions, page fetches, a refused localhost fetch, a one-word request, writes and edits, run and report, and general knowledge.
machinist: makes and improves machines, in Pi (machine machinist, or demo/machinist.sh). Its tools run in the machine process: read-only list_machines, show_machine, and recent_runs; specialize, rehearse, and train, which spend money, each with a required budget (at most $10 a call, or MACHINIST_MAX_BUDGET); and commit (through the promotion gate) and revert. Spending and changing tools run only in an interactive session, and only after the person agreed: their latest message names the budget, or says yes to a cost the machinist stated. Graded on that (checked from the conversation), and on grounding its claims in tool output, choosing sensible specializations, and reporting results honestly (judged).
wordmath: solves word problems; the answer on its last line is checked against a known answer. wordmath/fullwidth must also reason in fullwidth letters and digits (checked from the captured reasoning), and its transformReasoning rewrites reasoning into fullwidth when runs are exported, so export wordmath/fullwidth --from wordmath turns the parent's correct runs into training data for thinking that way.
julia and its children: writes a Julia program and runs it with run_julia in the isolated sandbox. The parent requires that the program runs (checked), does the task, and is idiomatic (judged). Each child adds a specialty: julia/parametric (type parameters, checked), julia/generic (no concrete argument types, checked; shown on several input types, judged), julia/fast (times its work, checked; written for speed, judged), and julia/inline (only short-form or @inline functions, checked). Juliaup installs under your home directory, which the sandbox hides, so run_julia mounts only the Julia installation read-only, never the rest of ~/.julia.

Choose the agent loop with --harness (or MACHINE_HARNESS):

Harness	Needs	Models
`tinker` (default)	`TINKER_API_KEY`	Tinker models and fine-tuned checkpoints, e.g. `Qwen/Qwen3.8-27B`
`pi`	`pi` on PATH	Tinker ids as above (or `tinker/<id>`) with `TINKER_API_KEY`, or Pi's own `provider/model` ids
`opencode`	`opencode` on PATH, `bun add @opencode-ai/sdk`, and the provider's key (e.g. `OPENROUTER_API_KEY`)	opencode's `provider/model` ids, e.g. `--model openrouter/thinkingmachines/inkling:free`
`openrouter`	`OPENROUTER_API_KEY`	OpenRouter ids, e.g. `--model thinkingmachines/inkling-small`
`fx`	`bun add libfx`, `AI_GATEWAY_API_KEY`	Vercel AI Gateway ids, e.g. `--model anthropic/claude-sonnet-5.5`
`openai`	`--base-url` or `OPENAI_BASE_URL`, optional `OPENAI_API_KEY`	Whatever the endpoint serves (Ollama, OpenRouter, vLLM)
`echo`	nothing	Replies with the input; free dry run of the plumbing

Free Inkling (thinkingmachines/inkling:free and inkling-small:free on OpenRouter) is served only to agentic harnesses, so call it through --harness pi or --harness opencode with OPENROUTER_API_KEY set; the plain openrouter harness gets a 403. The pi harness turns on Pi's own OpenRouter attribution headers (HTTP-Referer: https://pi.dev) for OpenRouter models, which is what admits it, while keeping Pi offline so no install telemetry is sent. Free models count as $0 against the budget. To grade for free too, pass --judge-model openrouter/thinkingmachines/inkling-small:free (or set MACHINE_JUDGE_MODEL):

bun src/cli.ts suite pirate --harness pi \
  --model openrouter/thinkingmachines/inkling-small:free \
  --judge-model openrouter/thinkingmachines/inkling-small:free

Prompts and outputs sent to the free endpoint are logged and used by Thinking Machines to improve its models, so keep private material off it.

The opencode harness runs one local opencode serve per CLI invocation, gives each run a fresh session in the run's sandbox, and deletes the session after the run. Runs use a custom machine agent whose one-line prompt replaces opencode's coding-agent prompt, so the model sees only the machine's instructions plus a short environment block (model id, working directory, platform, date) that opencode always adds. The server runs with a private, empty config home and opencode's switches for project config, Claude Code instruction files, skills, plugins, and sharing turned off, so your own opencode setup never leaks into machine runs. Pass extra opencode config (for example a custom provider) with opencodeHarness({ config }). Machine tools are served to opencode over MCP from the machine process (127.0.0.1 only, behind a per-run bearer token), so they run with the run's sandbox as on any other harness; only that run's tools are enabled.

The pi harness runs pi --mode json with a private, empty agent directory, so your ~/.pi settings, keys, extensions, skills, and context files never load, and the model sees exactly the machine's instructions. Machine tools are registered in a generated per-run extension that forwards each call to the machine process (127.0.0.1, per-run bearer token), so they run in the run's sandbox as on any other harness.

--eval grades each run. Judge graders call a model, so they cost money on a paid harness. On tinker and pi the default judge is thinkingmachines/Inkling-Small; pick another with --judge-model or MACHINE_JUDGE_MODEL.

Judges on tinker think at reasoning_effort medium (--judge-effort; --effort sets it for runs). Lower effort is cheaper but can grade worse: check a judge's effort with agree --with judge --judge-effort before lowering it.

Spending #

On tinker and pi, every command runs under a budget, $1 unless you pass --budget <usd> (or --budget none). Each run and each judge call is charged at Tinker's listed prices (src/pricing.ts), with prompt-cache hits at the cached rate (Tinker caches prefixes automatically, and tool loops resend their whole conversation, so most of a long run's input is cached; Inkling's temporary discount is ignored, so estimates err high), new work is refused once the limit is reached, and the command prints what it spent. Runs already in flight finish, so a command can overshoot by the cost of --concurrency runs. Usage at an unknown price is refused under a budget; price a fine-tuned tinker:// checkpoint as its base with --price-as <model>.

suite, compare, and optimize run up to --concurrency runs at once (default 4). A run that fails after the harness's retries (429s, 5xx, and network errors are retried with backoff) is reported and left out of the scores instead of ending the command.

Fine-tuning data #

machine export <machine> [--tree] [--min-score 1] --out data.jsonl writes graded runs that score at least --min-score as chat conversations (the instructions the run was given, the input, and its assistant turns with tool calls and results), the format tinker-cookbook's FromConversationFileBuilder reads. The files hold prompts and outputs: read them before training. For rejection sampling, --best <k> keeps each case's best k runs, --since limits to one sampling batch, and --cases cases.json keeps held-out cases out. --per-turn writes one example per assistant message, each ending at it, and every turn of a conversation, for trainers such as scripts/train_sft.py that learn only each example's last assistant message.

Runs, sessions, and sandboxes are stored under .machine/ in the workspace directory. They contain your prompts and outputs: .machine/ is git-ignored, and files are written readable only by you.

Not yet #

Training methods beyond SFT on good runs (RL against the rubric, DPO from good and bad pairs)
Rubric drafting, and approval of workspace edits through the CLI
Cloud sandboxes (Modal) and a cloud store
Restoring fx conversations from libfx checkpoints; history is passed as a transcript for now

Sandboxes #

Each run gets a working directory under .machine/sandboxes/, removed after the run. Tools write and read files there (writeFile, readFile, confined to the directory, symlinks included) and run commands with exec.

Where bubblewrap works (Linux with unprivileged user namespaces), commands run isolated: no network, a read-only system, /home, /root, /tmp, and other user-data mounts hidden except the run's own directory, an empty environment (no API keys), and the whole process tree killed on timeout. sandbox.isolated says whether that is in force; tools that run model-written code should refuse when it is false. Without bubblewrap the sandbox still scrubs the environment, but commands can reach your files and the network.

Bubblewrap shares your kernel and does not cap memory or CPU. For stronger isolation, run code in a remote sandbox (Modal is planned) behind the same Sandbox interface.