Build #

Notes on corpora build that the code cannot state for itself: the determinism decision, what is validated and by whom, and the two places where the emitted bytes are a contract rather than a convenience.

src/build/** is the only code that touches the signing key, and the only code allowed to depend on npm packages at runtime. See INTERFACES for the normative dist/ layout.

Order of operations #

1. Import the key (--key-file or CORPORA_SIGNING_KEY) — 64 hex characters, never logged.
2. Read every .json file under --lexicons, recursively, and validate all of them, reporting every problem at once rather than one per run.
3. Check that each authority domain the NSIDs imply has a _lexicon TXT record naming --did. How strictly is --dns-check's business; see below.
4. Build the DID document and round-trip it through @atproto/identity's own parser.
5. Build the MST, sign the commit, generate the full repo CAR and one partial proof per record.
6. Compare against the previous build and skip the write when nothing changed.
7. Emit dist/.

Steps 3 and 4 come before signing on purpose. A missing TXT record or a document naming the wrong key makes every signature worthless, and both are failures that would otherwise surface as invalid_scope on somebody else's login rather than as an error here.

Reproducibility, and the rev #

Identical lexicons plus an identical key produce a byte-identical dist/. This is an M1 acceptance criterion and there is a test that builds twice and compares every emitted byte.

Everything in the repository is content-addressed except one field. Repo.create takes the commit rev from a TID, which is clock-based, so an unmodified rebuild would otherwise churn a new commit CID, a new signature and a new set of proof CARs every time it ran.

Corpora derives the rev from the MST root instead: build the tree, hash the root, and format a TID from a fixed anchor (2020-01-01) plus a content-derived offset of up to about thirteen days. Same records, same rev, same bytes, on any machine at any time.

Two consequences worth knowing:

• The rev's timestamp component is not a publish time. Do not read it as one. It is an arbitrary point near an anchor that exists only to keep the value a syntactically valid TID.
• Revs do not increase across publishes. Nothing in the resolution chain reads them — each build is a fresh genesis commit with prev: null and since: null, there is no firehose and no incremental sync — but if you want monotonic revs anyway, pass --rev. It accepts a TID (3lxaaaaaaaa22) or an ISO 8601 instant to derive one from, so --rev "$(date -u +%FT%TZ)" in CI gives you the conventional behaviour at the cost of reproducibility.

The signature itself is deterministic: @atproto/crypto signs with @noble/curves, whose secp256k1 nonces are RFC 6979 derived rather than random.

Validation #

Three layers, none of which subsumes the others.

Ours, first. Non-integer numbers anywhere in a document, naming the file and the JSON path. DRISL (the encoding formerly called DAG-CBOR) has no float type, so these fail at encode time with a message that names neither. Also: lexicon: 1, a syntactically valid id, no duplicate ids, and a filename that does not contradict the id inside it.

lexiconDocumentSchema from @atproto/lex-document, second. The official TypeScript parser and the one the reference stack uses. Note that it is imported directly while being only a transitive dependency — it arrives with @atproto/repo, and is not listed in package.json.

goat lex lint, out of process and still worth running. The TypeScript parser's permission schema is {type, resource} intersected with an open dictionary of arbitrary keys to scalars, so it accepts field names and shapes that goat's concrete structs reject — "aud": ["*"] being the verified example. Corpora carries targeted checks for the permission fields that matter (aud is a single string; lxm and collection are arrays of NSIDs) but does not attempt to reimplement goat. See INTERFACES and the publish loop in PUBLISHING.

Green in TypeScript is necessary, not sufficient #

Two divergences between the Go and TypeScript stacks have already surfaced here, and both point the same way: Go is stricter, and the TypeScript path is the permissive one. A build that passes every in-process check can still produce something half the ecosystem cannot read.

Case	@atproto/*	goat / indigo
"aud": ["*"] in a permission	accepted by lexiconDocumentSchema	rejected: aud is a single string
did:web:example.com%3A3000	resolved by @atproto/identity	rejected: not a simple hostname
did:web:localhost%3A2583	resolved (downgraded to http)	rejected: any port at all

Corpora rejects the first two outright. The third it permits, because it is the blessed local development form and the tests need it, but the build says so in the summary and in a warning.

The lesson generalises: when a validation only exists on one side, treat the stricter side as the specification. Adding a check here is cheap; the alternative is a namespace that resolves for some clients and not others, which is the failure mode this project exists to eliminate.

did:web shape #

did:web:<host> and nothing else. A path (did:web:example.com:lexicons) is unsupported by atproto. A percent-encoded port is rejected unless the host is exactly localhost, quoting the DID spec: "Port numbers (with separating colon hex-encoded) are only allowed for localhost, and only in testing and development." The check runs for every did:web, including when --service supplies the endpoint and nothing needs to be derived from the host.

The id wins over the filename #

The id inside the document is authoritative for the NSID and the record key. A path agrees with it when the path's trailing segments spell the NSID, which covers com.example.thing.json, com/example/thing.json, and either under grouping directories. It is an error only when the filename is itself a well-formed NSID naming something else, since that is the case that genuinely misleads. Anything else, like permissions/authFull.json, is a warning.

The DNS check has three modes #

Step 3 derives the authority domains from the NSIDs it just read and looks up _lexicon.<domain>. Two different things can come back wrong, and they are not the same fact about the world:

--dns-check	Record missing	Record names a different DID
strict (default)	fails the build	fails the build
cutover	fails the build	warns
off	nothing is looked up	nothing is looked up

--no-dns-check is an alias for --dns-check off and is not going anywhere; passing both is an error unless they agree. build() takes the same three values as dnsCheck, where true and false still mean strict and off.

Why cutover exists. Moving a live namespace to a new authority is sequenced so that nothing is at risk until the new authority has been verified — ROADMAP steps 2 and 3 stand it up and check it while DNS still points at the old DID, and only step 4 flips the records. Under strict that build cannot be made at all: every record still names the DID being replaced, so every authority domain is a failure. The only escape used to be --no-dns-check, which also throws away the missing-record check at the moment it is most valuable, because a cutover is exactly when new authority domains appear and one of them is quietly absent.

So the rule the modes encode: a mismatch is expected mid-cutover; a missing record never is.

• A mismatch means the lexicons resolve. They resolve through the DID the record still names, which during steps 2 and 3 is the correct answer and the reason the sequence is safe.
• A missing record means they resolve through nothing. There is no old answer to fall back on — NSID authority has no hierarchical fallback (RESOLUTION) — and no step of any cutover where that is the intended state.

cutover warns once per mismatched domain, naming the DID found and the DID expected, and the build report carries those warnings so the CLI prints them. Flip the records once the new authority verifies, then go back to strict: leaving cutover on permanently means never noticing that the flip did not happen.

Emitted bytes that are contractual #

Records are served round-tripped, not copied. DRISL canonicalizes map key order, so a record does not survive a JSON round trip byte-identically — content is preserved, key order changes. records/<nsid>.json therefore holds the document as it comes back out of the encoder, so what is published is what verifies.

list/page-N.json is in descending rkey order, which is what a reference PDS returns from listRecords by default and the opposite of MST order. The serving handler cannot reorder prebuilt pages, so the order is decided here or not at all.

Proofs are partial, never the full repo CAR. getRecords generates one inclusion proof per record: roughly 1 KB against a 2.3 KB full CAR for the four test fixtures, and 2.1 KB against 36.8 KB for atcr.io's real 28. See RESOLUTION.

manifest.totalBytes counts the served artifacts only. manifest.json and data.js both embed the number and cannot count themselves.

manifest.didDocHash fingerprints the did.json that gets served, not the one on disk. It is sha256-<hex> over the DID document as the serving core writes it: JSON.stringify with no whitespace and no trailing newline. dist/did.json holds the same document pretty-printed, so hashing that file gives a different and useless value.

The wire encoding is the one that matters because there is exactly one consumer: corpora verify --origin <origin> --expect-did-doc <hash> fetches /.well-known/did.json and hashes the response bytes. Producer and verifier have to agree on the encoding or the check is worse than no check, so a build test hashes the handler's own response and compares.

did:web has no operation log, which means host compromise alone can forge a document, a key and every signature under it. ARCHITECTURE accepts that risk with one compensating control: publish the expected fingerprint and MST root somewhere the serving host does not control — the source repository, a release tag, a commit message. Both values are in manifest.json, which is build metadata and is never served, so quoting them from there is quoting something the deployed host cannot rewrite.

Not rebuilding for nothing #

A rebuild whose MST root, DID and signing key match the manifest already in --out writes nothing and says so. --force overrides. --against <origin> compares against a deployed authority instead by fetching com.atproto.sync.getLatestCommit and comparing commit CIDs — manifest.json is build metadata and is never served, so the deployed commit is the only thing there is to compare.

Reformatting a source file without changing what it encodes to moves its sources hash but not the MST root; the build still skips, and says that is what happened.

The key #

corpora keygen generates a secp256k1 key, round-trips it through Secp256k1Keypair.import before printing anything, and writes it with mode 0600 when given --out (refusing to overwrite an existing file). Given --out it does not echo the key: writing a secret to a file and to the terminal in the same breath is worse than either.

Rotation is a rebuild. Generate a new key, run corpora build, deploy: the new did.json and the new signatures ship as one bundle, so there is no window where they disagree.