docwriter
A project to fine-tune a model specifically for technical writing.
Star
Fork
Watch
docwriter docs PI_TRAJECTORIES.md
8.1 kB · 133 lines
Markdown
at main

Pi trajectory training #

Docwriter's product-shaped task is repository work, not isolated completion. The model should inspect a codebase, identify evidence, edit one documentation target, inspect the effect, and stop. A page-composition completion remains useful as a component test, but it is not the primary training boundary.

Runtime roles #

  • Pi 0.83.0 supplies the coding-agent loop and stable read, grep, find, ls, edit, and write tools.
  • pi-tinker 0.8.0 at 0428c2431109eb855456f2ff782685756dd25c31 supplies Tinker operation and checkpoint registration inside Pi.
  • Docwriter owns source licensing, pinned repository snapshots, target masking, task construction, trajectory custody, tool-schema snapshots, effect evaluation, and promotion.
  • Tinker Cookbook owns model-specific rendering and training primitives.

pi-tinker is deliberately thin. It does not export Pi sessions as training trajectories, enforce Docwriter provenance, or evaluate repository effects. Its raw docs-directory converter produces summarization pairs and is not suitable for this project.

Task lifecycle #

  1. Select a provenance-complete Docwriter record.
  2. Materialize its exact Git commit into a new workspace without .git metadata.
  3. Remove the target documentation file and keep the reference outside the agent-visible workspace.
  4. Run a teacher model through Pi with only the six pinned file tools. Bash is excluded.
  5. Reject the run if the target is absent, another file changed, a tool failed, a call is unresolved, the session compacted, or the assistant ended abnormally.
  6. Compare the resulting target with the hidden reference and source-grounded requirements.
  7. Convert the successful Pi session into renderer-neutral messages plus the exact Pi tool schemas.
  8. At training time, render the schemas in the model's native tool prefix and create one supervised datum per assistant decision.
  9. Compare base and checkpoint inside the same Pi task harness, with identical tools, prompts, limits, and repository snapshots.

The first task family reconstructs a missing page. The more important next family is historical maintenance: check out the parent of a code/documentation co-change commit, expose the implementation change, and ask the agent to update the stale page. That trains targeted edits rather than only page creation.

Files and commands #

The Pi runtime is locally pinned by package.json, pnpm-lock.yaml, and pnpm-workspace.yaml. The pnpm policy patches known vulnerable transitive versions and denies unneeded dependency build scripts.

config/pi-tools.json is generated from the pinned Pi package:

corepack pnpm run pi:tool-specs

Prepare a target-masked task:

docwriter prepare-pi-task \
  --dataset artifacts/qwen36-smoke-source/pydantic/dataset.jsonl \
  --record-id <record-id> \
  --output-dir artifacts/pi-tasks/<task>

After Pi finishes, evaluate filesystem effects:

docwriter evaluate-pi-task artifacts/pi-tasks/<task>

Prove the filesystem namespace before any model call:

docwriter verify-pi-isolation artifacts/pi-tasks/<task>

Export the successful session:

docwriter export-pi-trajectory \
  --session artifacts/pi-tasks/<task>/sessions/<session>.jsonl \
  --task artifacts/pi-tasks/<task>/task.json \
  --output artifacts/pi-trajectories/<trajectory>.jsonl

Artifacts and oracles are ignored and written privately. A trajectory stores the session and tool-spec hashes, Pi and pi-tinker versions, source provenance, teacher model identity, tool calls/results, and quality receipts. Teacher thinking blocks are discarded rather than converted into training data.

Validate native tool rendering and per-decision loss masks:

uv run --group train docwriter validate-pi-trajectories \
  artifacts/pi-trajectories/train.jsonl \
  --profile qwen36-35b-a3b-tinker

Create a cost plan without calling Tinker:

uv run --group train python -m docwriter.pi_tinker_train \
  --trajectories artifacts/pi-trajectories/train.jsonl \
  --profile qwen36-35b-a3b-tinker

Training requires both --execute and --max-cost-usd. It only creates an unpromoted checkpoint; base-versus-checkpoint Pi effect evaluation is a separate gate.

Rendering and loss masks #

Pi stores assistant tool calls as content parts and tool results as toolResult messages. Docwriter converts them to Tinker Cookbook's structured ToolCall and tool messages. Tool definitions remain a top-level provider-neutral field until the selected renderer creates its native tool prefix.

Do not train a Qwen trajectory as one conversation with TrainOnWhat.ALL_ASSISTANT_MESSAGES. The Qwen3.5/3.6 non-thinking renderer does not satisfy the sequence-extension property; earlier assistant turns can see a different prefix when the whole trajectory is rendered at once. Docwriter instead creates a separate datum for every assistant turn using the exact preceding prefix and TrainOnWhat.LAST_ASSISTANT_MESSAGE.

The stock pi-tinker validator does not yet handle this record shape. Its Python pass treats tool-only assistant turns as empty and passes JSON dictionaries directly where Tinker Cookbook expects ToolCall objects, producing AttributeError: 'dict' object has no attribute 'function'. Its generated SFT script also selects ALL_ASSISTANT_MESSAGES. Use Docwriter's validator and trajectory trainer for the dataset; use pi-tinker for Pi integration, Tinker operation, checkpoint registration, and interactive inspection.

This gives loss to:

  • evidence-selection and navigation tool calls;
  • edit/write tool calls and their arguments;
  • the final completion message.

User instructions and tool results are context, not targets.

The offline Qwen3.6 proof currently renders a three-decision edit trajectory into three datums of 1,363, 1,439, and 1,468 tokens, with 27, 56, and 8 trainable tokens. Every datum includes the same six native tool schemas; later decisions include prior tool calls and results.

Isolation requirement #

Removing .git and the target page prevents accidental history leakage, but it is not a complete sandbox. Pi's ordinary file tools accept absolute paths. A real teacher or checkpoint run must therefore execute in a filesystem namespace or container that exposes only:

  • the task workspace as writable;
  • the pinned Pi runtime as read-only;
  • a private session directory as writable;
  • the minimum OS libraries and certificate files needed for the model API.

The oracle, source cache, Docwriter repository, home directory, credentials, and unrelated host files must not be mounted. Bash remains unavailable. API credentials must enter through a private file descriptor or equivalent secret mount, not command-line arguments or copied .env files.

The first live proof used a target-masked Pydantic task. Pi 0.83.0 started inside Bubblewrap; /workspace and /sessions were visible, while the target page, .git, the dynamically detected host home, and the exact host-side oracle path were absent. The environment was cleared and the Pi runtime was mounted read-only. artifacts/pi-tasks/<task>/isolation.json owns the private receipt.

Teacher execution still requires credential delivery through a private file descriptor or equivalent secret mount. No teacher, base-agent, checkpoint-agent, or training call should run until that path and its cost gate have receipts as well.

Evaluation #

The primary unit is the filesystem effect:

  • requested target exists;
  • no out-of-scope path changed;
  • Markdown structure is valid;
  • required public symbols, defaults, errors, and examples are covered;
  • claims are supported by the pinned source snapshot;
  • repository-specific checks pass when available;
  • tool path is efficient enough to inspect rather than wander.

Reference-text similarity remains diagnostic. Exact reconstruction is neither required nor sufficient. A checkpoint promotes only if it improves the held-out effect metrics under the same Pi harness without new mutation, tool-error, or termination failures.