Parametric learning in your code harness.
machine docs getting-started.md
30 kB · 741 lines
Markdown
at main

Getting started with machine #

A machine is a small agent written as code. It says what it is for, how it is instructed, which model and tools it uses, and what good output looks like: a rubric of criteria, each with its own grader. Every time you use a machine, the exchange is stored as a run with its full trajectory, so the same records serve everyday use, measurement, and training.

This guide takes you from a fresh checkout to talking to a machine, writing your own, specializing it with child machines, measuring and improving it, and changing how a model reasons. Each section builds on the last, but you can skip to what you need.

  1. Install
  2. Keys
  3. Talk to a machine
  4. Run a suite
  5. Write your first machine
  6. Specialize with child machines
  7. Choose graders you can trust
  8. Improve a machine
  9. Change how a model reasons
  10. Models, harnesses, and spending
  11. Watch it live
  12. Troubleshooting
  13. Command reference

1. Install #

You need:

Tool Why Install
Bun 1.2 or later Runs machine (TypeScript, no build step) curl -fsSL https://bun.sh/install | bash
Pi The interactive terminal UI for machine <name> npm install -g @earendil-works/pi-coding-agent
bubblewrap (Linux) Isolates commands that machines run, such as model-written code sudo apt install bubblewrap
Julia, optional Only for the julia machines juliaup

Then:

git clone https://tangled.org/cameron.stream/machine
cd machine
bun install
bun link          # optional: puts a `machine` command on your PATH
bun test          # every test runs offline and calls no model

Without bun link, write bun src/cli.ts wherever this guide says machine.

macOS: everything works except running model-written code. There is no bubblewrap on macOS, and tools such as run_julia refuse to run code without isolation rather than run it on your machine. The pirate and wordmath machines need no code execution and work everywhere.

2. Keys #

Machine reads provider keys from the environment. Add the ones you have to ~/.zshrc (or your shell's profile) and open a new terminal afterwards:

export TINKER_API_KEY=...       # Tinker: the default for runs and judges
export OPENROUTER_API_KEY=...   # OpenRouter: free Inkling through Pi
Variable Used for
TINKER_API_KEY --harness tinker (the default) and Tinker models in Pi
OPENROUTER_API_KEY openrouter/... models, including the free inkling:free and inkling-small:free
MACHINE_JUDGE_MODEL The model judges use, if you want something other than the default
MACHINE_HARNESS The default harness, if not tinker

To grade for free, add:

export MACHINE_JUDGE_MODEL=openrouter/thinkingmachines/inkling-small:free

Free endpoints log prompts and outputs, and Thinking Machines uses them to improve its models. Keep private material off them.

3. Talk to a machine #

machine pirate

This opens the pirate in Pi's terminal UI. Ask it anything:

> How do I reset my wifi router?

Aye, matey! To reset yer signal-box that catches the great web o' whispers,
first cut the mains, unplug it from the wall, like casting off a heavy
anchor... ⚓ 🏴‍☠️ 🦜

The model sees exactly the machine's instructions and tools, nothing of Pi's own. Tools run against your current directory, isolated by bubblewrap. Every exchange is stored as a run in .machine/, which is git-ignored and readable only by you.

Inside a session:

Command What it does
/score Grades the last exchange against the machine's rubric
/good, /bad Marks the last exchange; stored as feedback on its run
/correct <note> Records what should change, e.g. /correct too terse
/machine Shows the machine, its criteria and tools, and what the session has spent

Useful flags:

machine pirate --model openrouter/thinkingmachines/inkling-small:free   # free
machine pirate --eval        # grade every exchange as it finishes
machine pirate --continue    # resume the last conversation with this machine

machine run pirate with no input does the same.

A coding agent #

dev works on a real project: it answers questions about the code, explains and plans, runs things, and, when you ask for a change, edits files and runs your tests before it says it's done. It changes files only when asked: "test" or "explain shop.py" gets an answer or a question, not an edit. Tools act on the directory you start it in, so start it in the project and point --workspace back at this repo, where the machine is defined and its runs are stored:

cd ~/some-project
machine dev --workspace ~/code/machine
> test_calc.py fails. Fix calc.py so the tests pass.

Commands run under bubblewrap: the project is writable, the rest of your home is hidden, and there is no network.

A general assistant #

assistant is for everything else: questions, research on the web, reading and writing files, running commands. Like dev, it works in the directory you start it in:

cd ~/notes
machine assistant --workspace ~/code/machine     # or ~/code/machine/demo/assistant.sh
> What does functools.cache do? Check https://docs.python.org/3/library/functools.html

It cites the file or URL its answers come from, and changes files only when you ask. web_fetch reads public pages only: it refuses localhost, private networks, and cloud metadata addresses, and a long page comes back in 20,000-character pieces. Commands still run without network.

To see what you've stored:

machine runs pirate          # newest first, with scores
machine eval pirate <run-id> # grade one stored run

4. Run a suite #

A suite runs every input in a machine's cases.json and grades each run:

machine suite pirate -v

The output has this shape:

  1. <score>  How do I reset my wifi router?
  2. <score>  What's 12 times 12?
       trial 1 metaphor: <why the grader marked it down>
  ...
<cases> cases × <trials> trial · mean <score> · pass rate <percent>
  answers          <mean>
  pirate_voice     <mean>
  ...

Each line is a case and its score. -v shows why an imperfect score was imperfect. The summary gives the mean, the pass rate (runs that met every criterion fully), and each criterion's average.

Models are not deterministic, and judges are noisy. A single run can pass or fail by chance, so run each case more than once before believing a number:

machine suite pirate --trials 3 --concurrency 8

Every command on Tinker runs under a budget of $1 unless you set one with --budget 5 (or --budget none). It prints what it spent at the end.

5. Write your first machine #

A machine is a directory under machines/. Here is a complete one that writes haiku.

machines/haiku/
  rubric.ts        what good looks like
  instructions.md  how the model is instructed
  machine.ts       model, graders, tools
  cases.json       inputs to measure it on

rubric.ts says what good looks like, criterion by criterion. Keep it separate from the graders: deciding what you want and deciding how to check it are different steps.

import { defineRubric } from "../../src/index.ts";

export const rubric = defineRubric({
  three_lines: {
    description: "The reply is exactly three non-empty lines and nothing else.",
    weight: 2,
  },
  seasonal: {
    description: "The poem names or clearly evokes a season.",
    weight: 1,
  },
});

instructions.md is the system prompt. Keeping it in its own file makes it something machine optimize can rewrite for you later.

You write haiku about whatever the user names. Reply with the poem only: three
lines, about five, seven, and five syllables, with a clear sense of season.

machine.ts ties it together. Every criterion needs a grader, and TypeScript rejects an evaluator that misses one.

import { check, defineEvaluator, defineMachine, instructionsFrom, judge } from "../../src/index.ts";
import { rubric } from "./rubric.ts";

export default defineMachine({
  name: "haiku",
  intent: "Write a haiku about whatever the user names.",
  instructions: instructionsFrom(import.meta.url),
  model: "Qwen/Qwen3.6-35B-A3B",
  rubric,
  evaluator: defineEvaluator(rubric, {
    // Code can decide this one exactly.
    three_lines: check("three-lines", (run) => run.output.trim().split("\n").filter((l) => l.trim()).length === 3),
    // This one takes judgment.
    seasonal: judge(),
  }),
});

cases.json is a list of inputs:

["the first snow", "a broken umbrella", "my cat at 6am", "a long train ride"]

Try it:

machine haiku                        # talk to it
machine suite haiku --trials 2 -v    # measure it

To check the wiring without spending anything, use the echo harness, which replies with the input and gives judges a fixed score:

machine suite haiku --harness echo

Tools #

A tool is a name, a description, a JSON schema, and a function. It runs in the machine's process with the run's sandbox, the only place it may read, write, or run commands:

import type { Tool } from "../../src/index.ts";

const wordCount: Tool = {
  name: "word_count",
  description: "Count the words in a text.",
  inputSchema: { type: "object", properties: { text: { type: "string" } }, required: ["text"] },
  async execute(input: { text: string }) {
    return String(input.text.split(/\s+/).filter(Boolean).length);
  },
};

Add it with tools: [wordCount] in defineMachine. For tools that run commands, use sandbox.exec, and refuse when sandbox.isolated is false, as machines/julia/tools.ts does.

6. Specialize with child machines #

A child machine does its parent's job within the parent's constraints, plus something more. It lives in a subdirectory of its parent:

machines/julia/            write and run correct, idiomatic Julia
machines/julia/parametric/ ...that uses type parameters
machines/julia/generic/    ...with no concrete argument types
machines/julia/fast/       ...optimized for runtime, and measured
machines/julia/inline/     ...using only one-line or @inline functions
import { defineChild, defineEvaluator, instructionsFrom } from "../../../src/index.ts";
import julia, { onProgram } from "../machine.ts";
import { rubric } from "./rubric.ts";

export default defineChild(julia, {
  name: "julia/parametric",
  intent: "Write Julia that is parametric: type parameters let one definition serve every element type.",
  instructions: instructionsFrom(import.meta.url),   // appended to the parent's
  rubric,                                            // only the child's own criteria
  evaluator: defineEvaluator(rubric, { parametric: onProgram("parametric", isParametric) }),
});

What a child inherits:

From the parent Rule
Every rubric criterion, with its grader A child adds criteria; it can never redefine or drop one
Instructions The child's own follow the parent's
Tools A child may add more, never with a name it already inherits
Model, step limit, reasoning transform Unless the child sets its own
cases.json, holdout.json Unless the child has its own

Because the family shares criteria and a model, you can look at it and measure it as one:

machine tree julia             # each machine, what it inherits, what it adds
machine suite julia --tree     # every machine's cases, then the inherited criteria across all of them

An author model can write a child for you, with cases to rehearse it on:

machine specialize dev --focus "Python datetime and timezone bugs" \
  --model thinkingmachines/Inkling-Small --slug datetime --budget 2

It validates what the author writes (the child typechecks, loads, and its check graders run; each case's test fails before the author's reference fix and passes after it) and prints the next commands: rehearse with suite, then train. Read the child before training it: a criterion that cannot be met on some cases will show up in the rehearsal's reasons.

Keep inherited criteria true for every child. An inherited criterion that fights a child's job (say, a parent's "idiomatic" criterion that penalizes the explicit loops a performance child needs) drags down the whole family.

7. Choose graders you can trust #

Use the cheapest grader that is reliable for each criterion:

Grader When Cost
check(name, fn) Code can decide it: format, counts, a known answer, a tool's real result Free, exact
jev().noul(), .score(), .choice() A simple question about the text Cheap (needs TYPESAFE_API_KEY)
judge() It takes open-ended reasoning A model call per criterion per run

judge() takes options when you need them:

judge({ model: "thinkingmachines/Inkling-Small" })   // a specific judge model
judge({ reasoningEffort: "medium" })                 // how hard the judge thinks
judge({ seesReasoning: true })                       // grade the model's reasoning, not only its reply
judge({ guidance: "Ignore spelling." })              // extra instructions for the judge

On Tinker, judges default to Inkling-Small at medium reasoning effort.

Measure a grader before trusting it. agree grades your stored runs with a candidate grader and compares it with the machine's own:

machine agree pirate no_modern --with judge --judge-model openrouter/thinkingmachines/inkling-small:free --harness pi

It prints, at several thresholds, agreement, Cohen's kappa, the false-pass and false-fail rates, and more. Kappa is agreement beyond chance: 1 is perfect, 0 is no better than chance. A grader that passes every run can show high raw agreement on mostly good runs and still have a kappa of 0. A cheaper grader that hasn't been measured shouldn't decide what a model trains on.

8. Improve a machine #

There is one copy of each machine. Change it in place, measure the change, and let git keep the versions.

Compare the working tree's machine with any git revision of it, on the same cases, graded by today's rubric:

machine compare pirate --trials 3                  # against HEAD
machine compare pirate --ref a1b2c3d --trials 3    # against any revision

The end of its output gives the mean difference between the versions:

  difference     <mean difference> ± <standard error> (standard error over cases)

A difference within about two standard errors of zero is not evidence either way. Commit if it wins; git checkout -- machines/pirate if it doesn't.

Optimize rewrites the machine's instructions.md from its graded failures:

machine optimize pirate --trials 2 --optimizer-model thinkingmachines/Inkling

The optimizer only proposes. A rewrite is kept only if it scores higher by at least two standard errors (--min-z) and no lower on the cases in holdout.json, which the optimizer never sees. A kept rewrite is left uncommitted for you to read with git diff. On a child, optimize rewrites only the child's own instructions.

Export graded runs as training data, in the chat JSONL format tinker-cookbook's supervised trainer reads:

machine export pirate --out pirate.jsonl                 # runs that scored 1
machine export julia --tree --min-score 0.9 --out julia.jsonl

The files hold your prompts and outputs. Read them before training on them or sharing them.

Train it in place #

machine train does the export, the fine-tune, and the measurement in one step, and leaves the decision to you. The machine needs a model Tinker can train (a machine that declares model: modelFrom(import.meta.url, "…") keeps its committed model in model.json) and a holdout.json:

export MACHINE_PYTHON=~/code/tinker-cookbook/.venv/bin/python   # a Python with tinker-cookbook
machine versions dev                                   # records the current definition as v1
machine suite dev --trials 4 --budget 1                # sample and grade the training cases
machine train dev --trials 3 --budget 5                # fine-tune, then checkpoint vs live on holdout.json
machine commit dev                                     # live only if it won by 2 standard errors
machine revert dev                                     # changed your mind

train prints a report: how many good runs it trained on, the losses, and each criterion on the held-out cases for the live version and the checkpoint. The candidate stays pending until you commit it, and commit refuses one that did not clearly win (--force overrides). Every run records its version, so machine runs and the dashboard show which version did what. --any-model trains on good runs by other models too: run the suite with --model <a stronger model> first to distill it into the live one.

From code: the SDK #

Everything above is also a library. run() streams what the machine does and can be awaited for the finished run:

import { formatReport, machine } from "machine";

const dev = await machine.load("dev");
for await (const message of dev.run("The pagination tests fail. Fix it.", { files: { "pagination.py": "…", "test_pagination.py": "…" } })) {
  if (message.type === "tool_call") console.log("→", message.name);
}

const candidate = await dev.train({ budget: 5, samples: 4, trials: 3 });
console.log(formatReport(candidate.report));
if (candidate.report.decision.keep) await candidate.commit();

dev.session({ autoGrade: true }) keeps a multi-turn conversation in one working directory, dev.grade() grades every run that is ungraded or was graded under an older rubric, and dev.optimize({ budget }) proposes an instructions rewrite as a candidate committed the same way.

Ask the machinist #

machinist is a machine whose job is the steps above. Open it in Pi from this repository and talk to it:

demo/machinist.sh          # or: machine machinist
> dev keeps fumbling timezones, make it a specialist

It looks before it suggests anything (list_machines, show_machine, recent_runs), then proposes a change with its cost: a child for a kind of task the parent struggles with, a rehearsal, or a training run. A weakness the parent's own criteria already check is fixed in the parent, so it says that instead of proposing a child.

specialize, rehearse, and train spend money. Each takes a budget in dollars, enforced like --budget, and at most $10 a call (MACHINIST_MAX_BUDGET raises the cap). They run only after you agree: your message names the budget ("yes, specialize dev for slicing bugs with a $3 budget"), or says yes to a cost the machinist just stated. One yes covers one spending call. commit makes a candidate live only if it passed the gate (force only when you say so), and revert undoes it. Long calls show their progress in Pi as they run. In suites none of these tools run; the machinist can only describe what it would do.

/score grades the exchange: asks_before_spending is checked from the conversation by the same rule the tools enforce, and grounded, sensible_focus, and reports_honestly are judged.

9. Change how a model reasons #

Reasoning models think before they answer, and Tinker returns that thinking apart from the reply. Machine keeps it on every run, so you can grade it and train on it. This section walks through the wordmath/fullwidth machine, which tries to make a model think only in fullwidth characters (think like this).

Grade the reasoning #

reasoningOf(run) returns everything the model reasoned during a run. A code check can read it:

import { check, defineChild, defineEvaluator, instructionsFrom, reasoningOf } from "../../../src/index.ts";
import wordmath from "../machine.ts";
import { rubric } from "./rubric.ts";

export default defineChild(wordmath, {
  name: "wordmath/fullwidth",
  intent: "Solve word problems while reasoning entirely in fullwidth characters.",
  instructions: instructionsFrom(import.meta.url),
  rubric,
  evaluator: defineEvaluator(rubric, {
    fullwidth_reasoning: check("fullwidth-reasoning", (run) => fullwidthShare(reasoningOf(run))),
  }),
  transformReasoning: toFullwidth,
});

For criteria that need judgment, judge({ seesReasoning: true }) shows the judge the reasoning as well as the reply.

Ask first, and measure #

The child inherits the parent's correct criterion and cases, so one suite measures both whether it still gets the right answer and whether it reasons the way you asked:

machine suite wordmath --trials 2
machine suite wordmath/fullwidth --trials 2 -v

If instructions alone don't move the model's reasoning, or moving it costs accuracy, that is the case for training.

Translate, then train #

Start from the parent's correct runs: rewrite their reasoning into the new script, and train the model on the result. transformReasoning on the child does the rewriting, and export --from writes the parent's runs as the child's training data, with the child's instructions:

machine export wordmath/fullwidth --from wordmath --out fullwidth.jsonl

Each example is a conversation whose assistant message carries the reasoning as a thinking part:

{"messages": [
  {"role": "system", "content": "You solve word problems... While you think, write only in fullwidth characters..."},
  {"role": "user", "content": "A baker makes 7 trays of 24 cookies and sells 131. How many cookies are left?"},
  {"role": "assistant", "content": [
    {"type": "thinking", "thinking": "A baker makes 7 trays. Each tray has 24..."},
    {"type": "text", "text": "7 × 24 = 168 cookies, and 168 − 131 = 37.\n\nAnswer: 37"}
  ]}
]}

Train a LoRA on it with scripts/train_sft.py, which uses tinker-cookbook's supervised trainer. Run it in a cookbook checkout's Python environment:

cd ~/code/tinker-cookbook
uv venv && source .venv/bin/activate && uv pip install -e .   # once
python ~/code/machine/scripts/train_sft.py fullwidth.jsonl ~/code/machine/.machine/training/fullwidth \
  --model Qwen/Qwen3.6-35B-A3B --epochs 2

It writes per-step losses to metrics.jsonl and checkpoints to checkpoints.jsonl in the log directory. Evaluate the final checkpoint with the same suite, priced as its base model so the budget can bound it:

machine suite wordmath/fullwidth --trials 3 \
  --model "tinker://<run-id>:train:0/sampler_weights/final" --price-as Qwen/Qwen3.6-35B-A3B
machine suite wordmath/fullwidth --trials 3 --cases holdout.json \
  --model "tinker://<run-id>:train:0/sampler_weights/final" --price-as Qwen/Qwen3.6-35B-A3B

Compare the two suites with the base model's: the held-out run shows whether the new way of reasoning carries over to problems the model never trained on.

Things to watch:

  • Accuracy is the guardrail. The inherited correct criterion tells you whether thinking in a new script cost the model its answers.
  • Fullwidth characters cost more tokens than ASCII, so the same thinking is longer and more expensive.
  • Use LAST_ASSISTANT_MESSAGE for multi-turn runs on Qwen3.5 and Qwen3.6. Their renderer can't train every assistant turn of one conversation at once correctly. The wordmath data is single-turn, so either setting works there.
  • More data generalizes better. Add problems before trusting a harder change.
  • Interactive sessions in Pi don't capture reasoning yet. Use suite or run for runs you want to train on.

The same pattern changes reasoning in other ways: a transformReasoning that translates to another language, strips filler, or enforces a format, together with a check that measures it.

10. Models, harnesses, and spending #

A harness is the agent loop that runs a machine. Choose one with --harness (or MACHINE_HARNESS):

Harness Needs Models
tinker (default) TINKER_API_KEY Tinker ids, e.g. Qwen/Qwen3.6-35B-A3B, thinkingmachines/Inkling-Small, tinker:// checkpoints
pi pi on PATH Tinker ids, openrouter/... (free Inkling), or any Pi provider such as openai/gpt-5.6-luna
openrouter OPENROUTER_API_KEY Paid OpenRouter models (free Inkling refuses this harness)
opencode opencode on PATH opencode's provider/model ids
fx bun add libfx, AI_GATEWAY_API_KEY Vercel AI Gateway ids
openai --base-url Any OpenAI-compatible server (Ollama, vLLM)
echo nothing Replies with the input; free dry runs

What things cost on Tinker, per million tokens:

Model Input Output
Qwen3-8B $0.20 $0.60
Qwen3.6-35B-A3B $0.54 $1.34
Qwen3.8-27B $1.86 $5.60
Inkling-Small (the default judge) $1.16 $2.88

Tinker caches repeated prompt prefixes automatically, and machine prices cached input at the cached rate. Tool loops resend their whole conversation, so most of a long run's input is cached. Free OpenRouter models count as $0.

Budgets:

  • Commands on tinker and pi stop starting new work at --budget dollars (default $1). Runs already in flight finish, so a command can overshoot by the cost of --concurrency runs.
  • A model with no known price is refused under a budget. Price a checkpoint as its base with --price-as, or run without a budget with --budget none.

11. Watch it live #

machine dashboard        # http://127.0.0.1:4242 (--port to change it)

The dashboard shows the workspace as it changes:

Section What it shows
Overview Runs, spend, and training at a glance; each machine's recent mean and pass rate; what is running now; live activity
Activity Every step machines report as it happens: runs starting, model steps, tool calls and results, replies, scores, feedback
Machines Each machine's intent, criteria (inherited ones marked), graders, tools, and model
Runs Every stored run; select one for its scorecard, reasoning, tool calls, and reply
Training Each fine-tune's loss curves, settings, token count, cost, and checkpoints to copy into --model

Every process that runs a machine (a suite, a Pi session, a one-off run) appends what it is doing to .machine/events.jsonl, and the dashboard follows that file. Nothing else needs to be running: start the dashboard in one terminal and anything you run in another appears in it. Training shows up when its log directory is under .machine/training/. The dashboard listens on 127.0.0.1 only, since runs hold your prompts and outputs.

Echo-harness dry runs appear in the activity but are left out of quality numbers.

12. Troubleshooting #

You see Do this
Set TINKER_API_KEY. Open a new terminal after editing ~/.zshrc, or source ~/.zshrc
Could not start pi (is Pi installed and on PATH?) npm install -g @earendil-works/pi-coding-agent
Refusing to run code: this sandbox is not isolated Install bubblewrap (Linux). macOS can't isolate commands.
inkling-small:free is only available on agentic harnesses Use --harness pi; the plain openrouter harness is refused
Failed to start server. Is port 4242 in use? machine dashboard --port 4300
Budget of $1.00 reached Pass --budget <dollars>
used tokens at an unknown price --price-as <base model> for a checkpoint, or --budget none
Could not read .../cases.json Add cases.json to the machine, or to an ancestor for a child
A model returns no reply It may have spent its whole output limit thinking; try --effort low or a different model

13. Command reference #

machine <machine>                    Use it interactively in Pi
machine dashboard                    Live view of runs, sessions, and training (--port)
machine run <machine> [input...]     Run once and print the reply; without input, open it in Pi
machine runs <machine>               Stored runs, newest first
machine eval <machine> <run-id>      Grade a stored run
machine suite <machine>              Run and grade cases.json (--trials, --tree, --cases, -v)
machine compare <machine>            Working tree vs a git revision (--ref, --trials)
machine optimize <machine>           Rewrite instructions.md from failures (--rounds, --min-z, --holdout)
machine agree <machine> <criterion>  Measure a grader against the machine's own (--with judge|jev)
machine tree <machine>               A machine, its children, and what each inherits
machine export <machine>             Graded runs as training JSONL (--tree, --from, --min-score, --best, --since, --per-turn, --out)
machine converse <machine> [case]    One conversation with the machine's counterpart, printed live (--eval grades both sides)
machine train <machine>              Fine-tune on good runs; report a candidate vs the live version (--budget, --samples, --trials)
machine propose <machine> --model <id> Another model as a candidate, measured on holdout.json (--budget, --trials)
machine specialize <machine> --focus <area>  An author model writes a validated child and its cases (--model, --author-model, --slug, --budget)
machine commit <machine> [candidate] Make a candidate live if it passed the gate (--force)
machine revert <machine>             Back to the previous live version
machine versions <machine>           The machine's versions and pending candidates

Options that apply everywhere: --harness, --model, --judge-model, --judge-effort, --effort, --budget, --price-as, --concurrency. machine --help lists them all.