peek
experiments in a post-browser web
Star
Fork
Watch
peek handoff-session.md
22 kB
Markdown
at main

Handoff — sync coverage and the datastore/server track #

Covers work through 2026-08-18. Two other handoffs exist and are NOT this one: handoff-mcp.md is the standing reference for the MCP remote backend design and its known gaps, and handoff-mcp-seam.md covers the MCP seam work.

This track lives in .worktrees/sync, on branch sync. Work there, not in the primary checkout, which stays on main.

.worktrees/fsm-window-kernel, .worktrees/bg-run-tmp-paths and .worktrees/spindles-review belong to other initiatives. Nothing here touches them.

Resume Here #

Goal #

Everything in the datastore must reach the server, so that losing a device loses nothing.

Sync coverage for items, events and tags is complete and proven against production, the test foundation under it is repaired and documented, and the bulk pull is paged. The desktop and server halves are landed and deployed; what remains is the phone, which still cannot sync, and the durability work behind it.

Current state #

Landed on main. The desktop and server work is done, gated and deployed; the phone is the remaining half. The commits that carry it:

  • fe14a3e6 — the end-to-end suite addresses one server profile from both sides. It had been 2 passed / 11 failed since 2026-06-24 because its own serverRequest() sent no profile param (resolving the legacy slug "default") while the sync module sent the desktop's local profile UUID, and an unregistered UUID gets its own isolated profile. Now 13/13.
  • 93cf0b5e — docs/sync-coverage-gaps.md section 8: what each of the five sync suites exercises for real and what is invisible to it by construction. Adds yarn test:sync:all.
  • dc19ffd0 — the two end-to-end cases that printed a pass on every path replaced with real assertions: a deletion tombstone reaching the server, and an item the server permanently rejects being parked and then un-parked by an edit.
  • 0a6e8216 — the three since routes accept limit and hand back nextCursor; the desktop's three pull loops follow it, merging page by page. A full pull now goes through /items/since/0 instead of GET /items.
  • ad9d6495 — a note being typed in stays open when the notes pane rebuilds.
  • 48c89956 — the coverage audit's per-suite counts refreshed for the paged-pull cases.
  • 78e7a0fd — an edited note's preview shows the new text immediately instead of the pre-edit text.

The six older ones (ca55f2ae, 0234630d, e17ed401, 64c5df6b, 740cffea, 4aeecd1d) carry per-event push tracking in item_events.pushedAt, the parked-item columns, the batched item push through POST /items/batch, and their docs.

Everything in How to verify is green, and the full Electron gate's failing set is inside apps/desktop/tests/gate-baseline.md across three consecutive runs.

The gate did catch one regression, fixed in e22c0c16: the notes pane restored an interrupted editor's focus and caret in a requestAnimationFrame, a frame after the rebuild, so a second rebuild landing in that window read the not-yet-applied caret through _captureOpenEditor() and stored it as the user's own — a caret mid-note came back at the end of it. Focus and caret are now applied synchronously at the end of _render(). It reproduced only under the full desktop project, never in a single-spec run or test:grep, and it was intermittent there — four failures across the three runs before the fix, none in the three after.

The server is deployed and the new routes are verified live: /items/since/0?limit=2 returns two rows with a nextCursor, a bad limit is a 400, and POST /items/batch answers rather than 404s. Both changes are backward-safe in either direction — the client probes the batch route once per run and falls back on a 404, and a server that ignores limit returns one unpaged page with no cursor, ending the client's loop after one round trip.

Next steps #

  1. Ship a mobile build. The phone is locked out of /items entirely until a version-3 build ships: the shipped iOS build sends X-Peek-Datastore-Version: 1 and checkVersionHeaders() in apps/server/index.js 409s it. Profile registration is not behind that check, which is why the device still shows a recent last-seen. Mobile source already declares version 3 — apps/mobile/peek-core/src/types.rs DATASTORE_VERSION, with apps/mobile/src-tauri/src/lib.rs sending it on both the pull and push paths — so this is a stale shipped binary, not a source gap. The App Store path is apps/mobile npm run archive (build-archive.sh), which needs Xcode's Distribute step by hand; yarn mobile:ios:xcodebuild:device:full only side-loads a connected device. Verify with scripts/verify-ios-build.sh — a green build does not prove the binary on the device is yours.

    The first pull will be slow but not destructive. pull_from_server() uses plain GET /items?includeDeleted=true on a first sync — no limit/cursor, no chunking, and the paging added on this track covers only the since routes — so the whole set arrives in one response, measured around 10 MB and growing. Tombstones are safe: merge_server_item() in apps/mobile/peek-core/src/sync.rs handles a matched tombstone with an in-place UPDATE items SET deleted_at = ... WHERE id = <local id>, keeping the row id, so the ON DELETE CASCADE on blobs and item_visits never fires and visit/frecency history and offline blobs stay attached. The delete-and- reimport branch runs only when no local row matches at all. Record the pending count before and after from get_sync_status() (Settings shows it), and take a server backup first regardless. needs the Mac, which holds Xcode.

  2. Finish the dispatch loop wiring. The grant exists — minted against production, scope tag mcp-test, read-write, profile 4c1b5b4d-d613-42eb-a570-7775e6f938bd — and GET /mcp/whoami confirms it resolves with that scope. What is missing is the MCP client config pointing at the remote backend: peek-mcp-init assumes a local Peek.app that does not exist on a Linux machine, so the three files it would write have to be produced against the server's /mcp/* routes instead. See handoff-mcp.md for the remote-backend design. Note GET /mcp/tags returns empty, because nothing carries the mcp-test tag — the loop will see no work until something is tagged.

  3. Give the restore path a production rehearsal. Backups are proven readable and the download route works against real data, but nothing has ever restored one. Until an archive is put back and verified, a backup is proven as an archive and unproven as recovery. Backups also still live on the same Railway volume as the live database with a 7-deep purge, so they do not protect against volume loss — the download route is the first half of fixing that; nothing pulls them off-box on a schedule. docs/server-backup-and-deploy.md → the "Restoring an archive" section has the route, its body, dry run, ordering, WAL handling and retention. A rehearsal restores into the live server, so take a fresh backup first.

  4. Close the gaps the coverage audit names, listed in docs/sync-coverage-gaps.md → the "The gaps this leaves" subsection of section 8: no suite exercises the tag or event stages over a real socket, none exercises real authentication end to end (the end-to-end suite spawns the server with E2E_TEST=true, which short-circuits apps/server/auth.js), item_events.pushedAt and the POST /items/batch 404 fallback are proven only in-process, and sync-version-compat.test.js asserts version-bump behaviour against a copy of datastore.ts checkAndWriteDatastoreVersion() rather than the real one.

  5. Sync rules — still deferred until something can write a user rule. There are zero source='user' rows and no code path that creates one; tile:rules:register hardcodes source: 'runtime' and there is no rule editor. All existing rules are source='manifest' and regenerate, so they must not sync. Two constraints the rule editor must satisfy first: user-rule ids must be globally unique across devices, since the id is the only viable sync identity and manifest rules use a content hash that would collide by design; and rule deletion is a hard DELETE with no tombstone column, so sync would resurrect deleted rules until rules gains deletedAt. Separately, scriptId resolution fails silently — a synced rule naming a script the other device lacks never fires and logs nothing.

Key files & symbols #

  • apps/desktop/main/sync.ts — syncAll() runs six stages: pull tags, pull items, pull events, push tags, push items, push events.
    • pullPages() walks a since pull one page at a time, yielding each page's rows so the caller merges and discards rather than accumulating the profile. It stops on a missing nextCursor, an empty page, a repeated cursor, or MAX_PULL_PAGES. Paging is opt-in on both sides — a server that ignores limit ends the loop after one request, which is what lets this ship before the server does.
    • pullFromServer() uses /items/since/0 for a full pull, not GET /items. Consequence: a row whose updatedAt is missing or unparseable reads as 0 and is never returned. Every write path stamps updatedAt, so this bites legacy rows only.
    • pushToServer() batches through pushItemsBatch() in chunks of MAX_ITEMS_PER_BATCH (250), applies each chunk's per-row verdicts in one local transaction, and falls back to pushSingleItem() for the rest of the run on a 404. Images always take the single-item route.
    • parkRejectedItem() records a permanent rejection as syncError plus syncErrorAt = the item's updatedAt at that moment. The push query skips a parked item only while updatedAt has not moved past syncErrorAt, which is what makes any edit un-park it — no edit path in datastore.ts knows parking exists. Do not replace this with a flag that edit paths must clear.
    • pushEventsToServer() selects WHERE i.syncId != '' AND e.pushedAt = 0 and stamps pushedAt per delivered chunk. Only delivered rows are stamped; that is the entire retry mechanism.
    • resetPerItemSyncState() takes a clearSyncId flag — keep syncId on a version bump or items duplicate. It also clears pushedAt and the park columns.
    • serverFetch() carries timeoutMs (DEFAULT_FETCH_TIMEOUT_MS 15 s, BATCH_PUSH_TIMEOUT_MS 60 s, BULK_PULL_TIMEOUT_MS 120 s).
  • apps/server/index.js — resolvePageOptions() and nextCursorFor() are shared by the three since routes; MAX_SYNC_PAGE is 1000. A bad limit or an undecodable cursor is a 400, deliberately, rather than a silent restart from the beginning. POST /items/batch validates per row and never rejects a whole chunk over one bad item, unlike POST /events. checkVersionHeaders() is mounted on /items and /items/* ONLY, so a version-mismatched client still reaches /events and /tags.
  • apps/server/db.js — encodeCursor() / decodeCursor() / appendCursorClause() back the keyset scan. The cursor carries the last row's timestamp AND id; rows share a millisecond, and a timestamp-only cursor either skips the rest of that group or repeats it forever. saveItems() wraps saveItem() in one transaction and returns a per-row result; saveItem() matches a pushed item by server row id, then by the syncId column. No content fallback; syncId is canonical.
  • apps/server/users.js resolveProfileId() — an unregistered UUID gets its own isolated profile. Deliberate (a cross-profile data-leak fix, a448079c); do not revert it to make a test pass.
  • apps/desktop/main/datastore.ts — migrateEventPushedAtColumn() adds item_events.pushedAt and backfills from the retired watermark; migrateItemSyncErrorColumns() adds the park columns. Both are gated on the column being absent, which is what makes the backfill one-shot.
  • apps/desktop/renderer/page/page-notes.js — _render() sets _rebuilding while it replaces the pane's DOM, _captureOpenEditor() snapshots the open editor's text and caret, and the textarea's blur handler keeps the note open when _rebuilding is set. Its closing branch also writes the new text onto the in-memory note before re-rendering, because _render() reads this._notes and the datastore save is fire-and-forget. Chrome fires that blur while the textarea is still in the document, so isConnected cannot distinguish a rebuild from the user leaving — the flag is the only signal that works. Verified by live trace, not by reasoning.
  • apps/desktop/tests/gate-baseline.md — the failing-spec baseline; diff against it rather than reading the gate's failure count.
  • docs/sync-coverage-gaps.md → section 8, "What the sync test suites can and cannot catch" — per suite: what is real, which stages it touches, what it cannot see.
  • docs/sync.md → the "Paging a pull" section — the cursor contract and its compatibility story.

How to verify #

yarn test:sync:all                                                  # 9 + 8 + 16 + 13
yarn server:test                                                    # 245
node --test apps/server/test-backup.js apps/server/test-restore.js  # 52
node --test packages/schema/fidelity.test.js                        # 32 pass, 0 fail, 9 skip
node packages/schema/check-freshness.js                             # all fresh
node --test packages/sync/test.js                                   # 96
npx tsc -p apps/desktop/tsconfig.json --noEmit                      # clean
yarn build && node --test apps/desktop/dist/main/sync.test.js       # 41
node --test apps/desktop/dist/main/datastore.test.js                # 45
NODE_ENV=test node --test apps/desktop/tests/unit/*.test.js         # 743
yarn test:grep 'an open note survives a reload'                     # 1 passed
yarn test:grep 'leaving a note still closes it'                     # 1 passed

yarn test:desktop:electron is the full gate, takes ~15 minutes and pins the CPU. Run it whenever a change warrants it where PEEK_GATE_UNATTENDED=1 is set, and ask first where it is not — see apps/desktop/CLAUDE.md. Judge it by diffing the failing spec set against apps/desktop/tests/gate-baseline.md, never by the count.

test:desktop:electron:spec does not build. It runs PACKAGED=1 against the last electron-builder output, so after any source edit it tests a stale binary. Build and repackage first, or use yarn test:grep, which rebuilds and runs from source.

npm install --prefix apps/server is needed once in a fresh worktree, for plain JS dependencies only — without it the desktop sync tests fail with Cannot find module 'hono', because they drive the real server app in-process.

Reference #

Decisions & gotchas #

  • A local bookkeeping column needs no datastore version bump. The rule that adding a synced column requires one exists because the push query only re-sends rows whose syncedAt is stale, so a bump is what forces the backfill. pushedAt, syncError and syncErrorAt are all declared "sync": false in packages/schema/v1.json: never on the wire, no server column, nothing a peer can observe. Adding them cost no bump and no full re-push.
  • packages/schema/v1.json is the source of truth, and the fidelity test only checks one direction per table. For item_events the completeness test parses the literal CREATE TABLE text in apps/desktop/main/datastore.ts and does not union in ALTER TABLE statements, unlike the items/tags equivalents — so a column added only by a migration fails it. Add new columns to both the schema file and the CREATE TABLE literal, then run yarn schema:codegen, or check-freshness.js fails.
  • A permanent server rejection now costs one item, not the datastore. Historical context for anyone reading the server's CONTENT_REQUIRED_TYPES comment: it used to stall the whole event stage, because the event watermark was held below every event behind the rejected item. Per-event stamps and item parking each removed one half of that. The server still accepts an empty text item deliberately.
  • createdAt is client-authoritative on INSERT; updatedAt stays server-stamped. Preserving creation dates is what "lose nothing" means, while updatedAt is the key last-write-wins compares on.
  • Mobile has no version-change reset at all, which is why it cannot reproduce the item-duplication bug. If it ever gains one it needs the same syncId-preserving rule the desktop has.
  • Mobile's park mechanism is not the desktop's. apps/mobile/peek-core/src/sync.rs mark_item_sync_failed() nulls its own sync_error column from each update statement in items.rs; one missed statement there is an item that silently never syncs again. The desktop keys on updatedAt instead, so no edit path participates.
  • A parked check is skipped, never stubbed green. Skips use the per-test form; describe.skip makes node:test drop subtests from the summary entirely.
  • The sync progress log is in-memory in the main process, so an app restart loses it. getSyncStatus().lastError already persists the last failure across restarts. Whether the log itself should persist is an open product call.
  • Tag push has no per-tag error isolation, unlike item push — a chunk failure there throws and aborts the whole push. Known and unaddressed.
  • Deleting an event does not propagate. tile:datastore:delete-item-event is a live IPC any tile can call; a deleted observation returns on the next pull. Propagating it needs a tombstone column that no schema declares.
  • Tauri desktop is paused and its schema fidelity checks are skipped with a stated reason. apps/tauri-desktop/CLAUDE.md records what to re-run before unpausing. Its Item.sync_source has no backing column: every SELECT building an Item in apps/tauri-desktop/src-tauri/src/datastore.rs reads row.get(6), which is createdAt, an INTEGER — a runtime type mismatch that also shifts every later column index. Whether to add a real column or delete the phantom field is a design call.
  • Tauri desktop and mobile Rust do not compile on this machine — both need GTK/glib development packages requiring privileges this environment does not grant. apps/mobile/peek-core does build and passes 107 tests.
  • The two IPC tests for tile:sync:push/tile:sync:pull replay the handler's stage order rather than invoking the registered closure, because tile-ipc.ts imports electron statically, which throws at parse time outside a live Electron process. They prove the stages deliver; they would not catch a rewiring of tile-ipc.ts itself. Recorded in docs/sync-coverage-gaps.md section 8. The settings panel's live status line and log have no automated coverage at all for the same reason, and that one is not recorded anywhere else.
  • A sync library was reviewed and rejected. ElectricSQL, PowerSync and Zero all require Postgres, which this server is not. Zero and Evolu have no Rust client, so apps/mobile/peek-core could not speak them; Automerge's Rust and JS implementations are not wire-compatible with each other. Replicache is archived, CR-SQLite is ~2 years stale and self-declared not production ready, and Yjs solves concurrent text editing, which is not the problem here. Evolu otherwise fits (MIT, self-hostable, SQLite) but stores opaque encrypted mutations in its relay, which would end the server's own query surface, the MCP grants and the restore path.
  • The MCP remote backend is on the back burner by decision, though its blocking question is answered: a real client (version 2.1.233) expands an embedded ${VAR} in a .mcp.json url. The #scope fragment is never transmitted (HTTP does not send fragments); it is only a local credential-file key, and scope reaches the server as a column on the grant row via apps/server/grants.js resolveMcpGrant(). Zero MCP grants exist on the account, and a device key cannot authenticate /mcp/* (verified: 401 there, 200 on /profiles).

Environment notes #

  • The Railway CLI is installed and authenticated; the repo is linked to project amusing-courtesy, service peek-node. Read-only commands need no confirmation. railway variables prints secrets — use `railway run --service peek-node -- node