docwriter
A project to fine-tune a model specifically for technical writing.
Star
Fork
Watch
docwriter docs RESEARCH.md
15 kB · 178 lines
Markdown
at main

Research notes #

Research date: 2026-08-04.

Technical-writing pedagogy #

The first direct-composition run asked a model to imitate complete documentation before isolating the decisions a technical writer learns. The corrective architecture is the staged curriculum under curriculum/.

The strongest conceptual handbook found is Developing Quality Technical Information: A Handbook for Writers and Editors, third edition, from IBM Press. Its public description organizes quality around accuracy, clarity, completeness, concreteness, organization, retrievability, style, task orientation, and visual effectiveness, with review and testing as an explicit final part. The book is all-rights-reserved, and the publisher currently reserves text-and-data-mining and AI-training rights. It remains a human research reference only; its chapters, checklists, and examples are not ingested.

Google Technical Writing One and Two are the cleanest licensed pedagogical sources. They provide an actual sequence, exercises, facilitator material, and objectives covering terminology, active voice, clarity, brevity, lists, paragraphs, audience, scope, self-editing, large documents, illustrations, sample code, document types, and tutorial writing. Covered Google Developers prose is CC BY 4.0 and code samples are Apache-2.0. Any source-derived records still require page-level attribution, content hashes, and modification notices.

Diátaxis provides the useful tutorial/how-to/reference/explanation distinction. Red Hat's Modular Documentation and supplementary style guides provide operational structure and templates. All three are CC BY-SA 4.0 and remain research references until the project decides how share-alike obligations apply to distributed datasets and checkpoints. Open Technical Writing and Write the Docs use CC BY-NC-SA 4.0 and are excluded from the intended commercial-capable training path.

The resulting curriculum teaches local mechanics, reader and purpose, diagnosis, information architecture, grounded composition, editorial review, and finally receipted repository maintenance. Repository trajectories are the practicum, not the first lesson.

Sources:

Initial model candidate #

The project does not require Qwen, Inkling, or Tinker. Model/runtime candidates are configuration, and corpus/evaluation artifacts must remain portable across them. Inkling-Small was the first researched candidate. The first executed candidate is Qwen3.6-35B-A3B because it combines stronger current coding/document performance with only 3B active parameters per token and modest MoE training cost.

Model Tinker ID Total / active parameters Tinker context Train price per million tokens
Qwen3.6-35B-A3B Qwen/Qwen3.6-35B-A3B 35B / 3B 64K $1.177

Qwen3.6 uses the Cookbook's qwen3_5 renderer family. The smoke run used qwen3_5_disable_thinking for direct documentation output, rank-16 LoRA, and the Cookbook-recommended learning-rate formula. The model and the Cookbook are Apache-2.0.

Thinking Machines Lab publishes two Apache-2.0 Inkling checkpoints:

Model Tinker ID Total / active parameters Tinker context Current train price per million tokens
Inkling thinkingmachines/Inkling 975B / 41B 64K or 256K $5.61 at the current discount
Inkling-Small thinkingmachines/Inkling-Small 276B / 12B 64K or 256K $1.73 at the current discount

Both Inkling variants are hybrid multimodal MoE models with controllable reasoning effort. They remain configured comparison candidates rather than the default.

Primary sources:

The current Tinker Cookbook supports thinkingmachines/Inkling-Small directly through tml-renderers. A Tinker comparison must render training and evaluation with the same effort value. The docs recommend sweeping both learning rate and effort rather than assuming the default is appropriate. Those controls belong to a profile/provider adapter, not the dataset.

First training result #

The first run deliberately separated repositories:

  • train: 17 curated Pydantic pages at 2e5f0e2b4218de31709f1cf9c5bc61ea97a68835
  • validation: four DataFrames.jl pages at 9d6601e4759d893b63b2c122425232e97f9bcdbb
  • two epochs, 34 single-example optimizer steps, rank 16
  • 301,960 training tokens
  • $0.4253 conservative upper bound; $0.4099 token-rate estimate after execution

Optimization succeeded in the narrow sense: mean loss fell from 0.7684 in epoch one to 0.4341 in epoch two, with the final observed batch loss at 0.1642. Generalization failed:

Metric Base Adapter
Clean terminations 4/4 1/4
Mean evidence-symbol coverage 1.0000 0.2875
Mean lexical-set F1 diagnostic 0.3229 0.1983
Sampled tokens 7,216 14,116

Three adapter outputs exhausted the 4,096-token ceiling and ended malformed. That ceiling was a harness error for full-page diagnosis: the four frozen reference completions contain 3,906, 2,596, 5,369, and 12,274 tokens. The clean-termination regression is therefore confounded by an output budget that could not contain two references, particularly the longest page. The adapter still entered visibly repetitive constructor/example loops and lost most supplied-symbol coverage, so the checkpoint remains rejected; a larger ceiling would determine how long the loops continue, not turn them into good pages.

The harness now uses a 4,096-token floor plus 1.25 times each frozen reference completion. The same validation set receives per-record ceilings of 4,883, 4,096, 6,712, and 15,343 tokens. Their combined non-cached run still fits under a $0.50 upper bound. Falling training loss established that the API and optimizer worked; it did not establish useful learning. A plausible next experiment is one epoch at a substantially lower learning rate with shorter, more diverse targets, but that requires a new cost-authorized run rather than reinterpretation of this one.

This run was not an agent evaluation. A deterministic builder supplied the evidence bundle, and Tinker sampled one completion directly. There was no repository navigation, tool use, iterative planning, Pi, or persistent-agent execution loop. The result changed the primary boundary: future training should use successful repository-editing trajectories, while direct composition remains a component diagnostic.

Harness #

Pi is an MIT-licensed, minimal coding-agent harness with read, write, edit, and shell tools, a TypeScript SDK, RPC/event-stream modes, and JSONL session state. It is suitable for running the base and adapter against identical pinned repositories while preserving tool-call receipts.

The Apache-2.0 pi-tinker package is now pinned as the Tinker operator and checkpoint-provider bridge inside Pi. It demonstrates a useful execution contract:

data → validation → baseline eval → two-step smoke → training → checkpoint eval

Its package is useful, but its built-in docs converter produces weak summarization pairs and it does not export Pi sessions into trajectory SFT. Docwriter therefore owns a bridge that preserves tool calls/results, exact tool schemas, source receipts, target masking, and filesystem-effect evaluation. For Qwen, each assistant decision becomes a separate last-assistant datum because the renderer warns that one all-assistant trajectory violates its sequence-extension contract. Full design: PI_TRAJECTORIES.md.

Sources:

Corpus findings #

The first-pass corpus requires the documentation and implementation to share one repository and one commit. This removes several famous documentation sites from consideration even when their prose license is permissive: React, Vue, Kubernetes, TensorFlow's separate docs repository, and other documentation-only sites are useful style references but weak source-grounded training pairs.

The seed set emphasizes Python and TypeScript because they dominate the intended product-shaped evaluation, with Rust and C++ present to prevent the model from equating technical writing with one ecosystem's vocabulary.

Strong first-pass sources #

Project License Documentation Why it belongs
Pydantic MIT docs/ Strong concepts and reference tied to typed Python and Rust-backed behavior.
FastAPI MIT docs/en/docs/ Task-oriented guides with maintained runnable examples.
Django BSD-3-Clause docs/ Clear separation of tutorial, how-to, explanation, and reference.
Flask BSD-3-Clause docs/ Concise guides and patterns with little ceremonial prose.
pytest MIT doc/en/ Explicit how-to, explanation, and reference taxonomy.
uv MIT OR Apache-2.0 docs/ Precise CLI/configuration documentation in the implementation repository.
marimo Apache-2.0 docs/ Modern guides for state, reactivity, APIs, and application deployment.
Node.js MIT doc/api/ Deep hand-maintained reference tied to exact runtime behavior.
Vite MIT docs/ Compact TypeScript/JavaScript guides and configuration reference.
Vitest MIT docs/ Modern test-tool guides and API/configuration reference.
Next.js MIT docs/ Large TypeScript corpus; useful only after aggressive deduplication and MDX cleanup.
Cargo MIT OR Apache-2.0 doc/book/src/ Rust CLI, configuration, package, and workspace reference tied to command implementations and tests.
rustup MIT OR Apache-2.0 doc/user-guide/src/ Concise Rust toolchain concepts tied to installer implementation and behavioral tests.
Julia MIT doc/src/manual/, doc/src/base/ Language manual and Base documentation in the implementation repository.
Pkg.jl MIT docs/src/ Package, environment, artifact, and application guides with Julia implementation and tests.
DataFrames.jl MIT docs/src/man/ Mature table-manipulation guides with dense public API and behavioral coverage.
Documenter.jl MIT docs/src/man/, docs/src/lib/ Documentation-system guides, doctesting, cross-reference, and deployment behavior.
Flux.jl MIT docs/src/guide/, docs/src/tutorials/, docs/src/reference/ Modern ML guides with Julia definitions, tests, and maintained examples.

The manifest records candidate locations, not automatic endorsement of every page. Each page still has to pass alignment and quality review.

JuMP was initially considered as a candidate due to its optimization tutorials, but JuMP is MPL-2.0. It remains outside the software-license allowlist unless the corpus policy is deliberately expanded; it was not smuggled in under the broad and frequently abused category of “open source.”

Excluded from the first pass #

  • CC-BY prose: legally usable with attribution, but outside the conservative software-license-only gate.
  • Generated API reference: teaches renderer output and templates more than writing judgment.
  • Books: often excellent teaching, but weakly aligned to bounded software source evidence.
  • CodeSearchNet and generic docstring corpora: function summaries are not documentation pages, and aggregate dataset licensing does not replace source-file provenance.
  • Public documentation without a cloneable source: cannot produce reproducible target/evidence pairs.
  • Current product prose as training gold: target products should remain held-out product-shaped evaluations rather than leak into the training set.

Prior datasets and benchmarks #

CoDocBench #

CoDocBench mines commits where code and its associated documentation changed together. It contains 4,573 Python code/documentation pairs from 200 projects with repository-disjoint train/test splits. The method is useful for the later maintenance task family because co-change is stronger evidence than proximity. The published dataset is function/docstring-oriented and does not preserve a sufficiently conservative per-file license chain for direct ingestion here.

CodeWiki and CodeWikiBench #

CodeWiki evaluates repository-level documentation with hierarchical decomposition, architecture context, and multi-dimensional rubrics. CodeWikiBench stores exact repository commits, documentation trees, structured pages, and generated rubrics. Its useful lesson is to evaluate coverage and cross-module understanding at repository scale. Its LLM-judge scores are not sufficient as a promotion gate, and the benchmark repository's license was not clearly exposed during this review, so no code or data is imported.

SWD-Bench #

SWD-Bench evaluates whether documentation helps a separate model detect, locate, and complete repository functionality. That is a stronger utility test than asking a judge whether prose looks good. A smaller version belongs in the held-out evaluation: give a reader model only the generated page, then measure whether it can answer functionality questions and locate the relevant source.

RepoDoc #

RepoDoc uses a repository knowledge graph for semantic retrieval and incremental updates. The relevant idea is selective evidence based on imports, calls, implementations, and module boundaries rather than whole-file dumping. This project starts with high-precision symbol and link signals; a graph index becomes worthwhile only if those signals fail on cross-module guides.

Evaluation consequences #

The literature points to four separate questions:

  1. Factuality: Is each behavioral claim supported by the supplied code evidence?
  2. Coverage: Does the page include the public surface and critical usage constraints?
  3. Utility: Can another developer or model complete a bounded task using the page?
  4. Writing: Is the page ordered, concise, and legible enough that the evidence can actually be used?

One scalar “documentation quality” score hides the failure mode. Promotion requires all four, reported separately.