subdermal
Gleam PDS
Star
Fork
Watch
subdermal docs STORAGE.md
9.8 kB
Markdown
at main

Development storage contract #

This is the M3 learning table, not a signed repository. These functions are called from isolated tests; no public storage route or HTTP startup database integration exists yet. Identity/authentication comes in later cards. The M3.3 connection-owning actor and learner read reply branch are verified in isolated tests. M3 is complete and local gate L has passed.

dev_records.create/get/replace/delete identify a record by owner and a validated collection/key path. Create returns AlreadyExists for a duplicate. Get, replace and delete return NotFound for a missing target. Replace does not insert missing records; repeating a deletion returns NotFound.

Create and replace validate JSON before writing and preserve its original text, including whitespace. All record and scope values are bound parameters. Replace/delete use SQLite RETURNING so existence and mutation are determined by one statement. Replacing a value with the same JSON still succeeds. SQL errors stay typed; single-record lookups report unexpected multiple matches.

Listing and continuation #

dev_records.list_page takes owner, validated collection, direction, a page size and an optional cursor. It returns records with validated keys and stored JSON plus an optional next cursor. Local resource policies are 1–100 records per page and at most 8192 bytes per cursor; these are project choices, not AT Protocol constants.

Ordering is SQLite COLLATE BINARY: forward is ascending, reverse is descending. For example, 10 precedes 2, and A precedes a. A forward continuation uses rkey > last_key; reverse uses rkey < last_key. Both comparisons are strict, so the last returned row is excluded. The listing fetches one extra row to detect another page, returns at most the requested size, and makes the next cursor from the last returned key. Exact-size final pages have no cursor. The size may change between requests.

The v1 token is unpadded base64url of canonical JSON containing version, owner, collection, direction and last key. Reuse checks the scope and key syntax. Unsupported versions, malformed tokens, duplicate/extra fields, noncanonical encodings and cross-scope reuse return typed errors. Treat it as opaque when calling the API. It is not signed or secret; scope checking does not authorize access to an account. Changing the token representation requires versioning and an explicit compatibility decision.

Changes between pages #

Each page reads current committed data in one query. Pages do not share a historical snapshot. This is the documented behavior for writes between pages:

  • Later keys in the chosen direction can appear on the next page.
  • Newly inserted keys behind the cursor are not revisited.
  • Deleted records disappear; a next page may become empty.
  • Deleting the cursor's record does not break continuation: its key is still a comparison boundary, and need not exist in the database.
  • A replacement is visible when that record is read on a later page.
  • With an unchanged record set, walking either direction yields each record once, without gaps or duplicates.

These semantics are tested with inserts/deletes/replacements between requests; they do not promise snapshot isolation across requests.

Write transactions #

transaction.run(connection, callback) uses BEGIN IMMEDIATE, passes the same connection to the callback, and commits an Ok result together with its return value. An Error result triggers explicit rollback of all earlier operations in that transaction. A duplicate insert alone does not establish this all-or-nothing behavior: without an explicit transaction, the earlier insert remains committed.

Begin, operation, commit and rollback failures have distinct typed errors; a rollback failure retains the original error too. Close the connection on RollbackFailed before further work. A programmer exception triggers cleanup and is re-raised; if that cleanup's rollback fails, the connection is closed. Ordinary expected failures should be returned as Error, not raised. Nested calls fail at BEGIN without rolling back the outer transaction.

Callbacks must not issue transaction-control SQL, perform network requests, or share the connection with another caller. This helper does not serialize access itself. The owner described below serializes complete write batches.

The isolated gleam run -m pds/storage_transaction_exercise runner prints the two-write result and subsequent first-record lookup, both without and with a transaction. The second insert deliberately duplicates the first key. The runner asserts that plain writes leave the first record while grouped writes remove it, and also runs in the normal unit suite. The agent completed this demonstration at the learner's request; no learner rollback exercise is pending.

Connection ownership and actor replies #

owner.start(settings) initialises the connection inside a Gleam actor. The opaque Owner is a message address with a process identifier for monitoring; clients cannot extract its connection. Private Write/Get/Shutdown messages carry reply Subjects whose types specify the response. A handler receives the connection state and one message, sends the response with process.send, then returns actor.continue(connection) for the next message. Returning a continuation and sending a reply are separate steps.

owner.write sends an ordered batch of Create/Replace/Delete mutations as one message. The entire batch executes through transaction.run before another message is handled. Expected errors reply and leave the owner available; RollbackFailed closes the uncertain connection and stops normally. Shutdown closes before acknowledgement and normal exit. The actor is linked to its starter. Tests explicitly stop/start it and verify persisted rows; automatic supervision and HTTP startup integration remain deferred. Start one writer per application database; this local API does not register a global singleton.

File connections verify WAL and set foreign_keys ON, synchronous FULL, and busy_timeout250ms. FULL is the chosen SQLite durability setting; our tests check configuration and reopen behavior, not simulated power failure. Busy returns an explicit error after the bounded lock wait; no retries loop forever. The sqlight/esqlite execution API still uses its default execution timeout; busy_timeout bounds lock contention, not arbitrary query or disk work.

The monitored request wait is 1000ms, with a disposable relay and a 1100ms outer deadline. Timeout and dead owner are typed errors rather than caller panics. Relay/alias cleanup discards late replies. A timeout does not cancel an accepted write: after a suspended actor resumes, that write can still commit. Caller time bounds do not imply bounded queue size, transaction size, or guaranteed cancellation. These are local choices, not protocol constants.

The learner's Get branch sends dev_records.get's Result after wrapping storage errors with RecordError, then continues with the same connection. gleam run -m pds/storage_owner_exercise checks success, missing-record errors, continued handling and persistence after an explicit owner restart. It now runs in the normal suite; a timeout fails the assertion.

Verification #

The ordinary gleam test suite covers create/get, duplicate/missing behavior, scope isolation, malformed JSON, reopen persistence, replace/delete, forward and reverse walks, ordering, limits, malformed/cross-scope cursors, cursor reuse after reopen and the live mutation cases above. Eight transaction tests also cover successful writes/reopen, duplicate rollback and connection reuse, the nontransactional baseline, rollback of replace/delete on validation failure, nested BEGIN rejection, deferred-constraint commit failure, preservation of rollback errors, and cleanup before re-raising a programmer exception. Temporary files and connections are removed/closed even when a test assertion fails. Ten actor tests also verify concurrent duplicate commits, atomic conflicting batches, rollback/recovery, explicit restart persistence, dead-owner errors, bounded lock contention, suspended-owner timeout and late-reply cleanup, and typed startup failure, full read scope/case preservation, and typed SQL read errors with continued actor handling. The actor lesson is also in the default suite. Fresh/reopened configuration checks verify FK/WAL/FULL/busy. The full default suite passes 141 tests; compiler and format checks pass.

Gate L passed on2026-10-05: fresh eight-request HTTP checks and independent binary WebSocket/close/HTTP-after-close verification pass, alongside persisted CRUD, pagination, restart and rollback contracts. The owned listener was stopped and its port released. This establishes the plan's local learning-server gate only. Runtime database integration, public protocol storage routes, automatic supervision and later repository/sync/client/operational gates remain future work. Strict NSID fixture compatibility remains separately unresolved.

Implementation references checked for this lesson: SQLite RETURNING, scrolling window queries, and collating sequences. Transaction semantics were checked against SQLite transactions on 2026-10-05. Owner references checked on the same date: installed gleam_otp1.3.0 sources and actor documentation, SQLite WAL, SQLite pragmas, and Erlang aliases.