pds #

Local learning project; implementation in progress.

We are learning Gleam while building a single-account AT Protocol Personal Data Server. The entry program prints a greeting and starts a local HTTP listener on 127.0.0.1:2583. GET /health returns {"status":"ok","service":"pds"}; unknown paths return 404 and unsupported health methods return 405. A separate module parses the structure of repository paths; identifier modules validate record keys and collection NSIDs. repo_path.parse combines them into validated paths with typed errors; split_path checks structure only.

The roadmap is in IMPLEMENTATION_PLAN.md. Session progress, verification results, and the exact next exercise are in LEARNING_LOG.md.

Local workflow #

Run these commands from this directory with Gleam, Erlang/OTP, and Rebar3 installed. The first build downloads dependencies from Hex.

gleam run
gleam check
gleam test
gleam format --check

gleam run stays running until stopped. In another terminal:

curl --fail-with-body -i http://127.0.0.1:2583/health

src/pds.gleam contains the entry point. src/pds/syntax/repo_path.gleam contains the structural path parser. test/pds_test.gleam runs gleeunit and discovers syntax tests in test/pds/. The normative parser suite, record-key fixtures, invalid NSID fixtures, XRPC error envelopes, local storage contracts and JSON data-model fixtures are covered by the current suite: all 190 tests pass. GET describeServer, structured errors and route boundaries are also verified over local HTTP. The development-only WebSocket binary echo passes the independent client check: exact bytes, disconnect cleanup, and ordinary HTTP afterward.

The required strict NSID fixture comparison is separate:

gleam run -m pds/nsid_interop

It checks all 52 original fixture expectations and currently exits nonzero: 51 pass, 1 fails, zero skipped. The remaining mismatch is NSID-001, a conflict between the specification's authority-length limit and an upstream valid fixture. See the discrepancy tracker. Run both verification commands; unit success alone does not prove compatibility.

Keep the generated manifest.toml to pin dependency versions.

The local WebSocket experiment is at /dev/ws and is gated to Development. Start gleam run, then run in another terminal:

node integration/dev_ws.mjs

The client uses Node 24.16.0's built-in WebSocket API, with no npm dependencies. It checks exact binary bytes, disconnect, and the health response afterward. This development experiment is separate from the repository event stream.

Storage checkpoint (M3 complete; local gate L passed) #

Database opening and checksum-tracked transactional migrations are verified in isolated temporary databases, including reopen and rollback. The parameterized insert exercise passes and runs in the normal unit suite; M3.1 is complete. The HTTP server does not open the local default ./data/pds.sqlite yet.

Development create/get/replace/delete and keyset listing are implemented in src/pds/storage/dev_records.gleam. JSON validation, duplicate/missing targets, isolation, reopen, cursors and live-pagination behavior are verified. See the storage contract. Run the original retrieval checks explicitly with:

gleam check
gleam run -m pds/storage_get_exercise

That explicit runner checks retrieval, missing records, owner/collection isolation, reopen and SQL errors, then closes and removes its temporary database. Retrieval is also included in the 190 passing unit tests. Unexpected multiple matches return a typed UnexpectedRowCount error, verified with an isolated test schema that deliberately omits the primary key. The pagination walkthrough is recorded in the learning log. M3.3's typed transaction helper, eight contract tests and the asserted rollback demonstration are verified. The owner actor's write/concurrency/restart/timeout checks and the learner's Get reply handler also pass. Run the rollback demonstration:

gleam run -m pds/storage_transaction_exercise

It compares the same two writes with and without a transaction in separate temporary files. The second insert deliberately duplicates the first key: plain writes leave the first record; grouped writes roll it back. This runner also runs in the default unit suite. Gate L passed with fresh local HTTP and binary WebSocket checks plus isolated storage persistence/rollback evidence.

src/pds/storage/owner.gleam opens and owns its connection inside an actor. Each write batch is one message and one transaction. Expected request errors are typed; a caller timeout does not cancel a queued/running write. WAL, synchronous FULL and a 250ms busy timeout are configured on each file connection. The owner is currently used only in isolated tests, without HTTP integration or automatic supervision. Run the verified actor lesson:

gleam run -m pds/storage_owner_exercise

It checks record replies, missing records, continued processing and reopening durable data, and also runs in the default unit suite. Next is M4.1's recursive data model and checked JSON boundary. Local gate L does not imply repository, sync or client interoperability; the known NSID disagreement remains open.

Data-model lesson (M4.1 in progress) #

src/pds/data_model.gleam is an initial recursive Value teaching scaffold. Array and Object contain more Values; objects currently retain string/value field pairs so duplicates cannot disappear during dictionary conversion. The field helper distinguishes missing, explicit Null and duplicate targets. Links now contain checked CIDs. The full protocol-value boundary, encoder budgets, special object interpretation and complete JSON conversions remain pending. All 20 pinned data-model fixture cases are copied unchanged with provenance; their adapter is not implemented.

The learner's example_record and as_text implementation is verified and included in the unit suite. Run the completed lesson explicitly:

gleam run -m pds/data_model_exercise

The runner verifies a nested object/list/null value, present null vs missing fields, and safe Text extraction including empty strings/non-text rejection. The learner's recursive encode_array in src/pds/encoding/json.gleam is verified and included in the default suite, alongside four initial boundary regression tests. Run the completed encoding lesson:

gleam check
gleam run -m pds/json_encoding_exercise

The runner checks empty/mixed/nested arrays and child-error propagation. The encoder checks integer bounds, byte alignment, duplicate keys, checked CID link encoding, and prevents ordinary Objects from impersonating $bytes or $link wrappers. Unknown reserved fields remain ordinary retained data. Encoder resource budgets, complete decoding and fixture coverage remain pending. M4.1 remains in progress.

src/pds/encoding/json_syntax.gleam now parses a bounded raw JSON tree with duplicate-key rejection and exact numeric information. Its integer helper accepts integral decimals/exponents and rejects fractions or signed64-bit overflow without rounding. Nine parser/numeric regression tests pass.

The learner's numeric branch in decode_scalar is verified and included in the default suite. It maps exact integers into model.Integer and numeric errors into InvalidNumber. Run the completed scalar lesson:

gleam check
gleam run -m pds/json_decoding_exercise

The runner checks scalar round trips, integral decimals/exponents, large integers and typed fractional/overflow failures. decode_scalar deliberately rejects compound inputs with ExpectedScalar; decode is the new recursive entry point. Array traversal is supplied and verified.

The learner's decode_object_fields in src/pds/encoding/json.gleam is verified and included in the default suite. It preserves field names/order, recursively converts children and propagates errors. Run the completed lesson:

gleam check
gleam run -m pds/json_object_decoding_exercise

The runner verifies nested round trips, unknown field retention, null vs missing and child errors. Checked link wrappers and nonempty type tags are now supported; modern blob value checks are also verified.

The learner's decode_bytes_wrapper in src/pds/encoding/json.gleam is verified and included in the default suite. It checks the one-field string shape, returns model.Bytes on success and retains typed shape/base64 failures. Run the completed bytes lesson:

gleam check
gleam run -m pds/json_bytes_decoding_exercise

The runner verifies empty/binary/nested round trips, optional padding, escaped JSON text, wrong value kinds, extra fields and malformed base64. Six base64 helper regressions also pass: standard alphabet, optional complete padding, malformed inputs, single-byte round trips and the1MiB helper limit. Decoder tolerance of unused pad bits is a local policy; the encoder emits canonical output. Complete protocol boundaries and official fixture coverage remain pending. Checked CID links and $link wrappers are described below.

The learner's validate_codec is verified and included in the default suite. Run gleam run -m pds/cid_exercise for both codecs, unsupported codes and exact text/binary round trips. Four base32 and three CID rejection tests also pass. The opaque CID checks the AT Protocol/DASL profile: four metadata bytes plus32 digest bytes, represented as lowercase b-base32. Content hashing/verification and immutable blocks come later.

Value.Link(cid.Cid) and JSON link conversion are verified. At the learner's request, the agent completed $link handling: exactly one string field, a checked CID, and typed shape/CID errors. The completed runner is in the default suite and verifies both codecs, escaped field names, nested round trips, extra fields, wrong kinds, invalid CIDs and duplicate keys. Run it explicitly:

gleam check
gleam run -m pds/json_link_decoding_exercise

$type decoding now preserves typed objects and their fields, rejecting empty or wrong-kind tags. It recognizes the special "blob" tag. Two typed-object regressions pass, including the selected123.0 integer-like case. The unchanged official JSON fixture adapter reports all 5 valid and 12 invalid cases passing; its three CBOR/CID vectors await M4.2/M4.3.

The learner's validate_fields in src/pds/blob.gleam is verified and included in the default unit suite. It checks field kinds, Raw CID codec, nonempty MIME and positive signed64-bit size before constructing opaque Blob metadata. Wrong kinds and invalid values produce distinct typed errors. The complete Object survives JSON conversion, preserving its additional fields. Run:

gleam check
gleam run -m pds/blob_exercise

The runner verifies metadata getters, kind/value failures, size boundaries, required field checks and nested JSON blob round trips. Contextual assertion messages identify input and expected/actual errors in the invalid-ref loop. Generic decoding remains intentionally broad: text, arrays, and untyped objects are valid data-model Values. A repository record is narrower: it must be an object with an NSID $type. The learner's from_value in src/pds/record.gleam is verified and constructs an opaque RepositoryRecord only after model.field and nsid.parse succeed. Run the completed exercise:

gleam check
gleam run -m pds/record_exercise

The runner checks generic-versus-record roots, missing/wrong/duplicate tags, NSID record types, blobs and definition fragments. This does not validate a record against a Lexicon schema. The completed reserved-wrapper lesson rejects manually constructed $bytes/$link Objects while preserving unknown $ fields. Encoder tree/output budgets and the JSON fixture adapter are complete. M4.2's bounded unsigned varints and deterministic CBOR are also complete: canonical scalars/arrays/text-key maps/CID links, strict definite decoding, consumed-byte accounting, local input/tree budgets, and all three imported CBOR byte vectors pass.

M4.3 now derives checked Raw/DAG-CBOR CIDs by SHA-256 hashing the exact content bytes. All three imported CBOR vectors produce their expected CIDs. The pure in-memory block store accepts only verified CID/content pairs, is idempotent for the same bytes, and does not permit replacement with changed bytes. Durable blocks remain a later repository-storage concern.