Spec: db layer restructuring #

Status: draft Tracking: beads epic (see bd show, filed alongside this spec)

Motivation #

The db layer works, but its correctness rests on conventions that every callsite must independently remember. Concretely:

• Db (src/db/mod.rs:48) exposes every keyspace as a public Keyspace field plus pub inner: Arc<Database>. Any module can write raw bytes to any keyspace with hand-built keys; db.inner.batch() appears in 27+ files.
• Four access styles coexist: half-dead static async helpers (Db::get, with insert/remove/contains_key marked #[allow(dead_code)]), direct sync calls inside spawn_blocking closures, direct sync calls on dedicated blocking threads, and free functions taking (batch, db, ...).
• Key formats live in free functions returning Vec<u8> (src/db/keys/) with prefix constants scattered per module — db/filter.rs maintains its own SEP/prefix constants separate from keys::SEP. The authoritative documentation of key formats is comments (duplicated again in AGENTS.md).
• Serialization helpers exist (ser_repo_state etc.) but ~31 files call rmp_serde directly, several deserializing RepoState raw and bypassing the helpers (backfill/worker/process.rs:58, backfill/manager.rs:122, backfill/worker/task.rs:187).

None of these are hypothetical risks:

1. The record-import invariant is duplicated three times. A record write must keep in sync: records ks, blocks ks, per-collection counts, backlinks (cfg), a StoredEvent, and a jetstream event (cfg). This is implemented once properly in ops::apply_commit (via RecordEmitter), then re-implemented by hand in backfill/sparse.rs (persist_sparse_backfill builds StoredEvent + event key + jetstream::stage_event manually, and carries its own cfg'd backlinks block) and twice more in backfill/worker/process.rs. Divergence between these is silent index drift — missing events, missing backlinks, wrong counts.
2. Three manual registries per keyspace. keyspace_by_name (db/keyspaces.rs), the /debug name→type deserializer match (api/debug.rs:99), and the /stats endpoint must each be updated when a keyspace is added. open.rs:209 admits this with a "when adding new keyspaces, make sure to..." comment. That comment is the bug report.
3. The counts protocol is convention-only. Every write site must remember: stage deltas into the batch → batch.commit() → apply_count_deltas (plus lifecycle reservation apply). The pairing is unenforced; a missed apply drifts in-memory counts from persisted state until restart replay. Three separate subsystems (count deltas + watermarks, LifecycleCountBatch, per-collection record count free fns) each carry their own call-order rules.
4. Nothing pairs a keyspace to its key format or value type. Inserting RepoState bytes into cursors, or building a record key against the repos keyspace, compiles fine and fails at read time — or worse, doesn't.
5. Db::open is a 260-line monolith; check_poisoned is called at some iteration sites and not others; every key construction heap-allocates.

What is already good and stays untouched: the migration framework (frozen versioned wire types, ordered forward migrations), the per-mode keyspace groups (IndexerDb/StreamDb/JetstreamDb/RelayDb + OpenCx), the RecordEmitter ZST facade pattern, the count-delta crash-safe replay design, and the per-keyspace tuning rationale.

Design #

One principle: a keyspace is one type, declared in one table, and its invariant-bearing writes have exactly one owner. Everything else derives.

1. Keyspace schemas: Ks<S> #

Each keyspace gets a schema type pairing name, key codec, value codec, tuning, and debug rendering:

pub trait Schema {
    const NAME: &'static str;
    type Key: KeyCodec;
    type Value: ValueCodec;

    /// per-keyspace tuning, applied at open (moves out of open.rs).
    fn options(cx: &OpenCx) -> KsOptions;

    /// debug rendering; default impl transcodes msgpack -> json.
    fn render(value: &[u8]) -> Result<serde_json::Value> { ... }
}

/// zero-cost typed handle; Clone is cheap (Keyspace is an Arc'd handle).
pub struct Ks<S: Schema> {
    inner: Keyspace,
    _s: PhantomData<S>,
}

impl<S: Schema> Ks<S> {
    pub fn get(&self, key: &S::Key) -> Result<Option<S::Value>>;
    /// zero-copy escape for CAS blocks and other raw-value reads.
    pub fn get_raw(&self, key: &S::Key) -> Result<Option<Slice>>;
    pub fn contains(&self, key: &S::Key) -> Result<bool>;
    pub fn insert(&self, txn: &mut Txn, key: &S::Key, value: &S::Value);
    pub fn remove(&self, txn: &mut Txn, key: &S::Key);
    pub fn prefix(&self, p: impl Prefix<S>) -> impl Iterator<Item = Result<(S::Key, S::Value)>>;
    /// greppable escape hatch (migrations, debug endpoints).
    pub fn raw(&self) -> &Keyspace;
}

Poison checks (check_poisoned) move into the read/iterator adapters so every access is covered instead of the current ad-hoc sites.

KeyCodec replaces the keys/ free functions with structs that are the documented format:

pub trait KeyCodec: Sized {
    /// KeyBuf = SmallVec<[u8; 64]>: typical keys never touch the heap.
    fn encode_into(&self, buf: &mut KeyBuf);
    fn decode(bytes: &[u8]) -> Result<Self, KeyError>;
}

/// {DID}|{COL}|{RKey} — the definition is the documentation.
pub struct RecordKey<'a> {
    pub did: &'a str,
    pub collection: &'a str,
    pub rkey: &'a str,
}

This deletes the duplicate SEP/prefix constants in db/filter.rs and makes "wrong key against wrong keyspace" a type error.

ValueCodec has three impls: msgpack via rmp_serde (blanket over Serialize + DeserializeOwned wire types), RawBytes (CAS blocks), and BeU64/BeI64 (cursors, counts). The ~31 scattered rmp_serde callsites and the ser_/deser_ helper pair in mod.rs collapse into this.

2. One declarative registry #

A single table declares every keyspace, grouped exactly like the existing mode groups:

keyspaces! {
    core {
        repos: Repos,
        repo_metadata: RepoMetadata,
        cursors: Cursors,
        counts: Counts,
        filter: Filter,
        crawler: Crawler,
    }
    #[cfg(feature = "indexer")]
    indexer {
        records: Records,
        blocks: Blocks,
        pending: Pending,
        resync: Resync,
        resync_buffer: ResyncBuffer,
    }
    #[cfg(feature = "indexer_stream")]
    stream { events: Events, ... }
    #[cfg(feature = "jetstream")]
    jetstream { ... }
    #[cfg(feature = "relay")]
    relay { relay_events: RelayEvents, ... }
    #[cfg(feature = "backlinks")]
    backlinks { backlinks: Backlinks }
}

The macro generates:

• the Db field groups (matching today's IndexerDb etc. — this extends the existing seam, it does not replace it),
• open-time creation with S::options(cx) tuning (guts the open.rs monolith),
• keyspace_by_name(&str),
• fn all() -> impl Iterator<...> for /stats,
• /debug rendering via S::render.

Adding a keyspace becomes: write the schema type, add one row. Forgetting the row means the keyspace doesn't exist at all — no silent partial registration. The open.rs:209 comment dies.

3. Visibility is the enforcement mechanism #

Registry-generated fields are private to db::. Read access is exposed per keyspace where legitimately needed; the invariant-bearing keyspaces (records, blocks, counts, events, jetstream_events, backlinks) get no public write access at all. The only way to mutate a record is the op type below. raw() escape hatches exist for db::migration and the /debug write endpoint — both inside or routed through db::, both greppable.

This is the "use the type system to encode correctness constraints" rule applied with module privacy, which is the only enforcement Rust actually gives us.

4. One write path for records: RecordTxn #

Layered write handles:

/// typed batch over fjall::Batch; plain writes (cursors, filter, pds_meta...).
pub struct Txn<'db> { batch: fjall::Batch, db: &'db Db }

impl Txn<'_> {
    pub fn commit(self, mode: PersistMode) -> Result<()>;
}

/// the ONLY path that may mutate records/blocks/counts/events/backlinks.
pub struct RecordTxn<'db> {
    txn: Txn<'db>,
    counts: CountDeltas,
    lifecycle: LifecycleCountBatch,
    emitter: RecordEmitter, // existing ZST facade, reused as-is
}

impl RecordTxn<'_> {
    pub fn put_record(&mut self, did, coll, rkey, cid, block, action) -> Result<()>;
    pub fn delete_record(&mut self, did, coll, rkey, prev_cid) -> Result<()>;
    pub fn update_repo_state(&mut self, did, f: impl FnOnce(&mut RepoState)) -> Result<()>;
    /// stage deltas -> batch.commit -> apply_count_deltas -> lifecycle apply.
    /// the ordering exists exactly once, here.
    pub fn commit(self, mode: PersistMode) -> Result<()>;
}

• put_record/delete_record own the full invariant: records ks, blocks ks, per-collection counts, backlinks (one cfg site, inside), StoredEvent emission and jetstream staging via RecordEmitter.
• ops::apply_commit, backfill/sparse.rs::persist_sparse_backfill, and both backfill/worker/process.rs sites route through it. The three hand-rolled StoredEvent constructions and sparse.rs's private backlinks cfg block are deleted, not deprecated.
• Drop-without-commit is the only failure mode, and it is exactly the case the existing count-delta restart replay already handles. No new crash-safety machinery — the existing design, minus the per-callsite ceremony.

This is not a transaction abstraction over fjall; batches are already atomic. It is the existing protocol given a shape that cannot be half-followed.

5. Blocking discipline #

One rule: Ks methods are sync. Async contexts use one wrapper:

impl Db {
    /// spawn_blocking with the Db handle; the one async<->sync seam.
    pub async fn run<T>(&self, f: impl FnOnce(&Db) -> T + Send) -> T;
}

Dedicated blocking threads (ingest/indexer/shard.rs, backfill retry worker) call sync methods directly, unchanged. The static async helpers on Db (get/insert/remove/contains_key, mostly dead code today) are removed.

Non-goals #

• No storage-engine abstraction, no fjall replacement, no mock-store trait for tests (fjall in a tempdir is fast; tests keep using the real thing).
• No runtime mode selection. Modes stay compile-time; the registry table keeps cfg at the composition root per the feature-gating conventions.
• The migration framework is untouched. Migrations deliberately read old formats and keep raw keyspace access via raw().
• No change to on-disk formats, key layouts, or wire types. This is a code-shape refactor; a fresh build must open an existing database and produce byte-identical writes.

Phasing #

Each phase lands independently, warning-free through nu tests/feature_matrix.nu (all 13 combos) and nu tests/run_all.nu. Within a phase, old and new APIs may coexist; at each phase boundary the old path is deleted — no lingering dual conventions.

1. Schema + registry. Schema, Ks<S>, ValueCodec, the keyspaces! table; port every keyspace; derive keyspace_by_name, /stats, /debug from it; move tuning out of open.rs. Pure mechanical, no behavior change. (Fields stay pub(crate) during this phase.)
2. Key codecs. Key structs replace keys/ free functions; delete the duplicate separator/prefix constants in db/filter.rs; poison checks fold into accessors/iterators.
3. RecordTxn. Route apply_commit, sparse backfill, and backfill worker processing through put_record/delete_record; delete the duplicated StoredEvent/backlinks/counts blocks; flip invariant keyspaces to private.
4. Counts + cleanup. Fold count staging/apply and lifecycle reservations into Txn/RecordTxn commit; remove loose apply_count_deltas callsites; Db::run replaces the static async helpers; decompose what remains of open.rs.

Ordering: 1–2 are low-risk mechanical enablers; 3 is the highest-value correctness fix and depends on them; 4 rides on 3's types.

Verification #

• Per phase: nu tests/feature_matrix.nu, nu tests/run_all.nu --skip-creds.
• Phase 3 specifically (invariant-touching): repo_sync_integrity.nu, backlinks.nu, stream_live_backfill.nu, stream_cursor_replay.nu, and repo_count_resync.nu with credentials — these are exactly the tests that observe the record-import invariant from outside.
• Phase 4: repo_count_resync.nu again (count drift is its purpose), plus ephemeral_ttl_*.nu for event pruning paths.
• Open an existing pre-refactor database directory and verify /stats and /debug/iter parity against a pre-refactor build (no format changes means byte-identical output).

Risks #

• Callsite breadth. Dozens of files touch keyspaces directly. Mitigation: phase boundaries are keyspace-by-keyspace mechanical ports; the compiler drives (private fields = exhaustive callsite list as errors).
• Iterator decode overhead. Typed iterators decode per item; hot scans that only need raw slices use prefix_raw variants. CAS block reads keep zero-copy Slice via get_raw.
• Monomorphization. Ks<S> per keyspace is ~20 instantiations of small methods; negligible.
• RecordTxn shape vs. sparse backfill. Sparse import builds records from proof CARs, not commit diffs; put_record must accept a pre-parsed block without forcing a re-parse. The API above takes (cid, block) raw for this reason — verify during phase 3 that sparse's zero-copy Bytes::slice path survives.