# Handoff — sync coverage and the datastore/server track Covers work through 2026-08-18. Two other handoffs exist and are NOT this one: `handoff-mcp.md` is the standing reference for the MCP remote backend design and its known gaps, and `handoff-mcp-seam.md` covers the MCP seam work. **This track lives in `.worktrees/sync`, on branch `sync`.** Work there, not in the primary checkout, which stays on `main`. **`.worktrees/fsm-window-kernel`, `.worktrees/bg-run-tmp-paths` and `.worktrees/spindles-review` belong to other initiatives.** Nothing here touches them. ## Resume Here ### Goal Everything in the datastore must reach the server, so that losing a device loses nothing. Sync coverage for items, events and tags is complete and proven against production, the test foundation under it is repaired and documented, and the bulk pull is paged. The desktop and server halves are landed and deployed; what remains is the phone, which still cannot sync, and the durability work behind it. ### Current state Landed on `main`. The desktop and server work is done, gated and deployed; the phone is the remaining half. The commits that carry it: - `fe14a3e6` — the end-to-end suite addresses one server profile from both sides. It had been 2 passed / 11 failed since 2026-06-24 because its own `serverRequest()` sent no `profile` param (resolving the legacy slug `"default"`) while the sync module sent the desktop's local profile UUID, and an unregistered UUID gets its own isolated profile. Now 13/13. - `93cf0b5e` — `docs/sync-coverage-gaps.md` section 8: what each of the five sync suites exercises for real and what is invisible to it by construction. Adds `yarn test:sync:all`. - `dc19ffd0` — the two end-to-end cases that printed a pass on every path replaced with real assertions: a deletion tombstone reaching the server, and an item the server permanently rejects being parked and then un-parked by an edit. - `0a6e8216` — the three `since` routes accept `limit` and hand back `nextCursor`; the desktop's three pull loops follow it, merging page by page. A full pull now goes through `/items/since/0` instead of `GET /items`. - `ad9d6495` — a note being typed in stays open when the notes pane rebuilds. - `48c89956` — the coverage audit's per-suite counts refreshed for the paged-pull cases. - `78e7a0fd` — an edited note's preview shows the new text immediately instead of the pre-edit text. The six older ones (`ca55f2ae`, `0234630d`, `e17ed401`, `64c5df6b`, `740cffea`, `4aeecd1d`) carry per-event push tracking in `item_events.pushedAt`, the parked-item columns, the batched item push through `POST /items/batch`, and their docs. Everything in How to verify is green, and the full Electron gate's failing set is inside `apps/desktop/tests/gate-baseline.md` across three consecutive runs. The gate did catch one regression, fixed in `e22c0c16`: the notes pane restored an interrupted editor's focus and caret in a `requestAnimationFrame`, a frame after the rebuild, so a second rebuild landing in that window read the not-yet-applied caret through `_captureOpenEditor()` and stored it as the user's own — a caret mid-note came back at the end of it. Focus and caret are now applied synchronously at the end of `_render()`. **It reproduced only under the full desktop project**, never in a single-spec run or `test:grep`, and it was intermittent there — four failures across the three runs before the fix, none in the three after. **The server is deployed** and the new routes are verified live: `/items/since/0?limit=2` returns two rows with a `nextCursor`, a bad `limit` is a 400, and `POST /items/batch` answers rather than 404s. Both changes are backward-safe in either direction — the client probes the batch route once per run and falls back on a 404, and a server that ignores `limit` returns one unpaged page with no cursor, ending the client's loop after one round trip. ### Next steps 1. **Ship a mobile build.** The phone is locked out of `/items` entirely until a version-3 build ships: the shipped iOS build sends `X-Peek-Datastore-Version: 1` and `checkVersionHeaders()` in `apps/server/index.js` 409s it. Profile registration is not behind that check, which is why the device still shows a recent last-seen. Mobile source already declares version 3 — `apps/mobile/peek-core/src/types.rs` `DATASTORE_VERSION`, with `apps/mobile/src-tauri/src/lib.rs` sending it on both the pull and push paths — so this is a stale shipped binary, not a source gap. The App Store path is `apps/mobile` `npm run archive` (`build-archive.sh`), which needs Xcode's Distribute step by hand; `yarn mobile:ios:xcodebuild:device:full` only side-loads a connected device. Verify with `scripts/verify-ios-build.sh` — a green build does not prove the binary on the device is yours. The first pull will be slow but not destructive. `pull_from_server()` uses plain `GET /items?includeDeleted=true` on a first sync — no `limit`/`cursor`, no chunking, and the paging added on this track covers only the `since` routes — so the whole set arrives in one response, measured around 10 MB and growing. Tombstones are safe: `merge_server_item()` in `apps/mobile/peek-core/src/sync.rs` handles a matched tombstone with an in-place `UPDATE items SET deleted_at = ... WHERE id = `, keeping the row id, so the `ON DELETE CASCADE` on `blobs` and `item_visits` never fires and visit/frecency history and offline blobs stay attached. The delete-and- reimport branch runs only when no local row matches at all. Record the pending count before and after from `get_sync_status()` (Settings shows it), and take a server backup first regardless. **needs the Mac, which holds Xcode.** 2. **Finish the dispatch loop wiring.** The grant exists — minted against production, scope tag `mcp-test`, read-write, profile `4c1b5b4d-d613-42eb-a570-7775e6f938bd` — and `GET /mcp/whoami` confirms it resolves with that scope. What is missing is the MCP client config pointing at the remote backend: `peek-mcp-init` assumes a local `Peek.app` that does not exist on a Linux machine, so the three files it would write have to be produced against the server's `/mcp/*` routes instead. See `handoff-mcp.md` for the remote-backend design. Note `GET /mcp/tags` returns empty, because nothing carries the `mcp-test` tag — the loop will see no work until something is tagged. 3. **Give the restore path a production rehearsal.** Backups are proven readable and the download route works against real data, but nothing has ever restored one. Until an archive is put back and verified, a backup is proven as an archive and unproven as recovery. Backups also still live on the same Railway volume as the live database with a 7-deep purge, so they do not protect against volume loss — the download route is the first half of fixing that; nothing pulls them off-box on a schedule. `docs/server-backup-and-deploy.md` → the "Restoring an archive" section has the route, its body, dry run, ordering, WAL handling and retention. A rehearsal restores into the live server, so take a fresh backup first. 4. **Close the gaps the coverage audit names**, listed in `docs/sync-coverage-gaps.md` → the "The gaps this leaves" subsection of section 8: no suite exercises the tag or event stages over a real socket, none exercises real authentication end to end (the end-to-end suite spawns the server with `E2E_TEST=true`, which short-circuits `apps/server/auth.js`), `item_events.pushedAt` and the `POST /items/batch` 404 fallback are proven only in-process, and `sync-version-compat.test.js` asserts version-bump behaviour against a copy of `datastore.ts checkAndWriteDatastoreVersion()` rather than the real one. 5. **Sync `rules` — still deferred until something can write a user rule.** There are zero `source='user'` rows and no code path that creates one; `tile:rules:register` hardcodes `source: 'runtime'` and there is no rule editor. All existing rules are `source='manifest'` and regenerate, so they must not sync. Two constraints the rule editor must satisfy first: user-rule ids must be globally unique across devices, since the id is the only viable sync identity and manifest rules use a content hash that would collide by design; and rule deletion is a hard `DELETE` with no tombstone column, so sync would resurrect deleted rules until `rules` gains `deletedAt`. Separately, `scriptId` resolution fails silently — a synced rule naming a script the other device lacks never fires and logs nothing. ### Key files & symbols - `apps/desktop/main/sync.ts` — `syncAll()` runs six stages: pull tags, pull items, pull events, push tags, push items, push events. - `pullPages()` walks a `since` pull one page at a time, yielding each page's rows so the caller merges and discards rather than accumulating the profile. It stops on a missing `nextCursor`, an empty page, a repeated cursor, or `MAX_PULL_PAGES`. **Paging is opt-in on both sides — a server that ignores `limit` ends the loop after one request, which is what lets this ship before the server does.** - `pullFromServer()` uses `/items/since/0` for a full pull, not `GET /items`. **Consequence: a row whose `updatedAt` is missing or unparseable reads as 0 and is never returned.** Every write path stamps `updatedAt`, so this bites legacy rows only. - `pushToServer()` batches through `pushItemsBatch()` in chunks of `MAX_ITEMS_PER_BATCH` (250), applies each chunk's per-row verdicts in one local transaction, and falls back to `pushSingleItem()` for the rest of the run on a 404. Images always take the single-item route. - `parkRejectedItem()` records a permanent rejection as `syncError` plus `syncErrorAt` = the item's `updatedAt` at that moment. **The push query skips a parked item only while `updatedAt` has not moved past `syncErrorAt`, which is what makes any edit un-park it — no edit path in `datastore.ts` knows parking exists. Do not replace this with a flag that edit paths must clear.** - `pushEventsToServer()` selects `WHERE i.syncId != '' AND e.pushedAt = 0` and stamps `pushedAt` per delivered chunk. **Only delivered rows are stamped; that is the entire retry mechanism.** - `resetPerItemSyncState()` takes a `clearSyncId` flag — **keep `syncId` on a version bump or items duplicate.** It also clears `pushedAt` and the park columns. - `serverFetch()` carries `timeoutMs` (`DEFAULT_FETCH_TIMEOUT_MS` 15 s, `BATCH_PUSH_TIMEOUT_MS` 60 s, `BULK_PULL_TIMEOUT_MS` 120 s). - `apps/server/index.js` — `resolvePageOptions()` and `nextCursorFor()` are shared by the three `since` routes; `MAX_SYNC_PAGE` is 1000. A bad `limit` or an undecodable `cursor` is a 400, deliberately, rather than a silent restart from the beginning. `POST /items/batch` validates per row and never rejects a whole chunk over one bad item, unlike `POST /events`. `checkVersionHeaders()` is mounted on `/items` and `/items/*` ONLY, so a version-mismatched client still reaches `/events` and `/tags`. - `apps/server/db.js` — `encodeCursor()` / `decodeCursor()` / `appendCursorClause()` back the keyset scan. **The cursor carries the last row's timestamp AND id; rows share a millisecond, and a timestamp-only cursor either skips the rest of that group or repeats it forever.** `saveItems()` wraps `saveItem()` in one transaction and returns a per-row result; `saveItem()` matches a pushed item by server row id, then by the `syncId` column. **No content fallback; syncId is canonical.** - `apps/server/users.js resolveProfileId()` — an unregistered UUID gets its own isolated profile. **Deliberate (a cross-profile data-leak fix, `a448079c`); do not revert it to make a test pass.** - `apps/desktop/main/datastore.ts` — `migrateEventPushedAtColumn()` adds `item_events.pushedAt` and backfills from the retired watermark; `migrateItemSyncErrorColumns()` adds the park columns. Both are gated on the column being absent, which is what makes the backfill one-shot. - `apps/desktop/renderer/page/page-notes.js` — `_render()` sets `_rebuilding` while it replaces the pane's DOM, `_captureOpenEditor()` snapshots the open editor's text and caret, and the textarea's `blur` handler keeps the note open when `_rebuilding` is set. Its closing branch also writes the new text onto the in-memory note before re-rendering, because `_render()` reads `this._notes` and the datastore save is fire-and-forget. **Chrome fires that blur while the textarea is still in the document, so `isConnected` cannot distinguish a rebuild from the user leaving — the flag is the only signal that works.** Verified by live trace, not by reasoning. - `apps/desktop/tests/gate-baseline.md` — the failing-spec baseline; diff against it rather than reading the gate's failure count. - `docs/sync-coverage-gaps.md` → section 8, "What the sync test suites can and cannot catch" — per suite: what is real, which stages it touches, what it cannot see. - `docs/sync.md` → the "Paging a pull" section — the cursor contract and its compatibility story. ### How to verify ``` yarn test:sync:all # 9 + 8 + 16 + 13 yarn server:test # 245 node --test apps/server/test-backup.js apps/server/test-restore.js # 52 node --test packages/schema/fidelity.test.js # 32 pass, 0 fail, 9 skip node packages/schema/check-freshness.js # all fresh node --test packages/sync/test.js # 96 npx tsc -p apps/desktop/tsconfig.json --noEmit # clean yarn build && node --test apps/desktop/dist/main/sync.test.js # 41 node --test apps/desktop/dist/main/datastore.test.js # 45 NODE_ENV=test node --test apps/desktop/tests/unit/*.test.js # 743 yarn test:grep 'an open note survives a reload' # 1 passed yarn test:grep 'leaving a note still closes it' # 1 passed ``` `yarn test:desktop:electron` is the full gate, takes ~15 minutes and pins the CPU. Run it whenever a change warrants it where `PEEK_GATE_UNATTENDED=1` is set, and ask first where it is not — see `apps/desktop/CLAUDE.md`. Judge it by diffing the failing spec set against `apps/desktop/tests/gate-baseline.md`, never by the count. **`test:desktop:electron:spec` does not build.** It runs `PACKAGED=1` against the last `electron-builder` output, so after any source edit it tests a stale binary. Build and repackage first, or use `yarn test:grep`, which rebuilds and runs from source. `npm install --prefix apps/server` is needed once in a fresh worktree, for plain JS dependencies only — without it the desktop sync tests fail with `Cannot find module 'hono'`, because they drive the real server app in-process. ## Reference ### Decisions & gotchas - **A local bookkeeping column needs no datastore version bump.** The rule that adding a synced column requires one exists because the push query only re-sends rows whose `syncedAt` is stale, so a bump is what forces the backfill. `pushedAt`, `syncError` and `syncErrorAt` are all declared `"sync": false` in `packages/schema/v1.json`: never on the wire, no server column, nothing a peer can observe. Adding them cost no bump and no full re-push. - **`packages/schema/v1.json` is the source of truth, and the fidelity test only checks one direction per table.** For `item_events` the completeness test parses the literal `CREATE TABLE` text in `apps/desktop/main/datastore.ts` and does not union in `ALTER TABLE` statements, unlike the `items`/`tags` equivalents — so a column added only by a migration fails it. Add new columns to both the schema file and the `CREATE TABLE` literal, then run `yarn schema:codegen`, or `check-freshness.js` fails. - **A permanent server rejection now costs one item, not the datastore.** Historical context for anyone reading the server's `CONTENT_REQUIRED_TYPES` comment: it used to stall the whole event stage, because the event watermark was held below every event behind the rejected item. Per-event stamps and item parking each removed one half of that. The server still accepts an empty `text` item deliberately. - **`createdAt` is client-authoritative on INSERT; `updatedAt` stays server-stamped.** Preserving creation dates is what "lose nothing" means, while `updatedAt` is the key last-write-wins compares on. - **Mobile has no version-change reset at all**, which is why it cannot reproduce the item-duplication bug. If it ever gains one it needs the same syncId-preserving rule the desktop has. - **Mobile's park mechanism is not the desktop's.** `apps/mobile/peek-core/src/sync.rs mark_item_sync_failed()` nulls its own `sync_error` column from each update statement in `items.rs`; one missed statement there is an item that silently never syncs again. The desktop keys on `updatedAt` instead, so no edit path participates. - **A parked check is skipped, never stubbed green.** Skips use the per-test form; `describe.skip` makes `node:test` drop subtests from the summary entirely. - **The sync progress log is in-memory in the main process**, so an app restart loses it. `getSyncStatus().lastError` already persists the last failure across restarts. Whether the log itself should persist is an open product call. - **Tag push has no per-tag error isolation**, unlike item push — a chunk failure there throws and aborts the whole push. Known and unaddressed. - **Deleting an event does not propagate.** `tile:datastore:delete-item-event` is a live IPC any tile can call; a deleted observation returns on the next pull. Propagating it needs a tombstone column that no schema declares. - **Tauri desktop is paused** and its schema fidelity checks are skipped with a stated reason. `apps/tauri-desktop/CLAUDE.md` records what to re-run before unpausing. Its `Item.sync_source` has no backing column: every SELECT building an `Item` in `apps/tauri-desktop/src-tauri/src/datastore.rs` reads `row.get(6)`, which is `createdAt`, an INTEGER — a runtime type mismatch that also shifts every later column index. Whether to add a real column or delete the phantom field is a design call. - **Tauri desktop and mobile Rust do not compile on this machine** — both need GTK/glib development packages requiring privileges this environment does not grant. `apps/mobile/peek-core` does build and passes 107 tests. - **The two IPC tests for `tile:sync:push`/`tile:sync:pull` replay the handler's stage order rather than invoking the registered closure**, because `tile-ipc.ts` imports `electron` statically, which throws at parse time outside a live Electron process. They prove the stages deliver; they would not catch a rewiring of `tile-ipc.ts` itself. Recorded in `docs/sync-coverage-gaps.md` section 8. The settings panel's live status line and log have no automated coverage at all for the same reason, and that one is not recorded anywhere else. - **A sync library was reviewed and rejected.** ElectricSQL, PowerSync and Zero all require Postgres, which this server is not. Zero and Evolu have no Rust client, so `apps/mobile/peek-core` could not speak them; Automerge's Rust and JS implementations are not wire-compatible with each other. Replicache is archived, CR-SQLite is ~2 years stale and self-declared not production ready, and Yjs solves concurrent text editing, which is not the problem here. Evolu otherwise fits (MIT, self-hostable, SQLite) but stores opaque encrypted mutations in its relay, which would end the server's own query surface, the MCP grants and the restore path. - **The MCP remote backend is on the back burner** by decision, though its blocking question is answered: a real client (version 2.1.233) expands an embedded `${VAR}` in a `.mcp.json` url. The `#scope` fragment is never transmitted (HTTP does not send fragments); it is only a local credential-file key, and scope reaches the server as a column on the grant row via `apps/server/grants.js resolveMcpGrant()`. Zero MCP grants exist on the account, and a device key cannot authenticate `/mcp/*` (verified: 401 there, 200 on `/profiles`). ### Environment notes - The Railway CLI is installed and authenticated; the repo is linked to project `amusing-courtesy`, service `peek-node`. Read-only commands need no confirmation. `railway variables` prints secrets — use `railway run --service peek-node -- node