# Sync coverage gaps — what a server restore would not bring back

Research note. Surveys which parts of the datastore reach the sync server and which
are local-only, on the question: if a laptop is lost, what does restoring from the
server produce?

Nothing here is scheduled work. Each gap names what settles it.

Sources: `packages/schema/v1.json` (the canonical schema, `sync` flag per column),
`apps/desktop/main/sync.ts` (`pushSingleItem()`, `mergeServerItem()`,
`pullFromServer()`, `pushToServer()`, `syncTagsToItem()`, `pushTagsToServer()`,
`pullTagsFromServer()`, `mergeServerTag()`, `pushEventsToServer()`,
`pullEventsFromServer()`, `mergeServerEvent()`), `apps/desktop/main/rules.ts` and
`rule-engine.ts`, and `apps/server/db.js` (`initializeSchema()`, `saveItem()`,
`saveTags()`, `getTagsSince()`).

## Short answer

A restore returns items with all their fields — title, domain, favicon, visit
history, frecency, starred/archived state, deletion tombstones — their tags, tag
frecency, tag metadata, and now item events (the observations behind series and
feed items). It does not return user-created rules, because none exist yet: there
is no rule editor and no code path that creates a `source: 'user'` rule.

The remaining gaps are narrower than they once were: tag rename and tag deletion
don't propagate, event deletion doesn't propagate, and a push with no `tags` field
is a latent (not live) hazard. See below for each.

## 1. Item fields — all now sync

`title`, `domain`, `favicon`, `visitCount`, `lastVisitAt`, and `frecencyScore` are
`"sync": true` on `items` in `packages/schema/v1.json`, carried by `sync.ts
pushSingleItem()` / `mergeServerItem()`, and stored by `apps/server/db.js
saveItem()`. So are `mimeType`, `starred`, `archived`, and `deletedAt`.

`createdAt` is client-authoritative: `pushSingleItem()` sends the item's original
local `createdAt`, and `apps/server/db.js saveItem()` honors it only on INSERT, so
an item's original creation date survives a restore. `updatedAt` stays
server-stamped on every write instead — `mergeServerItem()`'s conflict resolution
(last-write-wins) compares on it, so it can't also carry client-supplied values
without breaking that comparison.

## 2. `item_events` now syncs, stamped per event

`packages/schema/v1.json` marks every `item_events` column `"sync": true` and lists
the table in `validation.required_sync_columns`. `sync.ts pushEventsToServer()` /
`pullEventsFromServer()` / `mergeServerEvent()` carry it, and `apps/server/db.js`
stores it. `series` and `feed` items — whose entire payload lives in `item_events`
rows rather than in the item itself — restore with their observations intact.

Push records delivery per event, in `item_events.pushedAt` (0 = never delivered),
rather than against any time watermark. The reason a watermark cannot serve here:
an event's `itemId` on the wire is the *server's* item id, resolved from the local
item's `syncId` — so an event whose parent item has no `syncId` yet (not pushed, or
still mid-push) has to be skipped for this run. Any watermark still advances at the
end of that run (items tolerate this, since they track their own per-row
`syncedAt`), which would push it past the skipped events' `createdAt` and strand
them permanently.

A dedicated event watermark, held below the oldest skipped event, was the previous
answer and had its own failure: an item the server permanently rejects — or one of a
type that never syncs at all — never earns a `syncId`, so it pinned that watermark
indefinitely and every run re-pushed the whole event tail behind it (24,000 events
per run, measured against production). With a per-row stamp, an event whose parent
cannot sync strands exactly itself and is picked up whenever that parent does sync.

`pushedAt` is declared `"sync": false` in `packages/schema/v1.json` — local
bookkeeping, never sent on the wire, no column on the server — so it needs no
datastore version bump. `datastore.ts migrateEventPushedAtColumn()` adds the column
and stamps everything at or below the retired watermark as already delivered; only
the tail behind it re-pushes, once, and the server ignores ids it already holds.

Deletion does not propagate: a locally hard-deleted event reappears on the next
pull if the server still has it. There is no `deletedAt` (or any tombstone) column
for `item_events` in any backend's schema.

**Settled by:** a tombstone column for `item_events`, plus push/pull handling for
it — unscheduled work, and the one piece of §2 still open.

## 3. Tag metadata now syncs; tag identity and deletion are the remaining gaps

`slug`, `color`, `parentId`, `description`, and `metadata` on `tags` are `"sync":
true` as of DATASTORE_VERSION 3. `sync.ts pushTagsToServer()` /
`pullTagsFromServer()` / `mergeServerTag()` carry them, keyed by the tag's `name` —
the same identity `apps/server/db.js getOrCreateTagWithConn()` already used for tag
rows an item push derives. `parentId` travels on the wire under that same field
name but holds the parent's name, not any id — the server column of the same name
stores that name too, not a row id — resolved back to a local tag id on pull in a
second pass so a parent arriving after its child in the same pull still links
correctly; an incoming cycle of `parentId` (name) links is broken deterministically
(newest `updatedAt` loses its parent link, ties going to the lower `id`) so every
device that pulls the same cycle breaks the same link.

Two gaps remain, accepted rather than fixed:

- **Renaming a tag on one device creates a second tag rather than renaming the
  first.** Identity on the wire is the name; a syncId would give the same row two
  sources of truth to reconcile, and the push/pull code was built to avoid that
  rather than build rename support.
- **Tag deletion does not propagate.** There is no tombstone column on `tags` — a
  tag deleted on one device still exists, and can still be pulled back, on every
  other device.

**Settled by:** a real tag identity beyond name (needed for rename to sync) and a
`deletedAt` column plus tombstone push/pull (needed for delete to propagate) — both
separate, unscheduled work.

## 4. Rules are entirely local — deferred, not rejected

The `rules` table is `"sync": false` in every column. A survey of the code finds
zero rows with `source: 'user'`, and no path that produces one: there is a
`tile:rules:register` IPC handler (`api.rules.register`), but it always writes
`source: 'runtime'`, never `'user'`, and there is no rule editor UI. The only rules
that exist are `source: 'manifest'` (4 of them, across `entities` and
`tag-actions`), reconciled from feature manifests by `rules.ts
syncManifestRules()` on every load — so manifest rules regenerate themselves and
should not sync.

Rule sync isn't rejected, just blocked on a rule editor that doesn't exist yet.
Whoever builds one has to satisfy two constraints first for sync to be possible:

- **User-rule ids must be globally unique across devices.** The id is the only
  viable sync identity for a user rule. Manifest rules can't use plain ids for
  this because `rules.ts computeManifestRuleId()` derives theirs from a content
  hash of the rule's fields — a user rule minted the same way would collide by
  design the moment two devices declare similar-looking rules.
- **Rule deletion needs a tombstone.** `rules.ts deleteRule()` is a hard `DELETE
  FROM rules WHERE id = ?`, and there is no `deletedAt` (or equivalent) column on
  `rules` in any backend's schema. Syncing rules as-is would resurrect a deleted
  rule on every other device that still has it.

Separately, and true regardless of whether rules ever sync: `rule-engine.ts
fireRule()` resolves a rule's `scriptId` by publishing to
`scripts:execute:<scriptId>` — a plain pubsub topic. If no script on that device
subscribes to that topic (because the script doesn't exist there), the publish has
no listener, the rule silently never fires, and nothing logs it. A synced rule
naming a script the receiving device lacks would hit exactly this.

**Settled by:** building a rule editor (giving `source: 'user'` a real origin),
then deciding whether user-authored rules sync — filtered to `source = 'user'`,
once the id-uniqueness and tombstone constraints above are met.

## 5. The two copies of the canonical schema are no longer divergent

`apps/server/schema.json` is generated from `packages/schema/v1.json` by
`packages/schema/codegen.js generateServerSchema()`, and the generated file is
checked in. Generation is necessary because deploys upload `apps/server/` alone
(see `docs/server-backup-and-deploy.md`), so server code can never
`import` from `packages/schema` at runtime — it needs its own copy of whatever it
reads.

The generated file is a narrow projection, not a mirror: `SERVER_TABLES` in
`codegen.js` lists only the tables the server physically creates (`items`, `tags`,
`item_tags`, `item_events` — `rules` is desktop-only and deliberately dropped),
and for each it carries only `validation.required_sync_columns`, because that's
the only part `apps/server/db.js validateSchema()` reads at boot. Freshness is
covered by `packages/schema/check-freshness.js`, which regenerates and diffs
against the checked-in copy.

## 6. The server's `items` table columns — resolved

`apps/server/db.js initializeSchema()` creates `items` with `title`, `domain`,
`favicon`, `frecencyScore`, `visitCount`, `lastVisitAt`, and `mimeType` alongside
the original columns. This was downstream of §1: once those fields started
syncing, the server needed columns to receive them. See
`docs/mcp-remote-backend-design.md` §3.1.

## 7. A push with no `tags` field wipes the item's existing tag links

Every other item column `apps/server/db.js saveItem()` writes goes through
`COALESCE(?, existingColumn)`, so a client that omits a field on a re-push (e.g.
an older client that doesn't carry `title`/`domain`/`starred`/etc.) leaves the
stored value untouched. `tags` is the one exception: `saveItem()` unconditionally
runs `DELETE FROM item_tags WHERE itemId = ?` and rebuilds the links from the
pushed `tags` array, and `POST /items` defaults an absent `tags` field to `[]`. A
push with no `tags` field at all therefore clears the item's tags, where the same
omission on any other field is a no-op.

This is latent, not live: every current client (desktop, mobile, extension) always
sends a full `tags` array on every item push. Pinned by
`apps/server/test.js` "a push with no tags field wipes the item's existing tags
(latent hazard, not desirable)".

**Settled by:** giving `tags` the same COALESCE-style omitted-means-unchanged
treatment as every other column — unscheduled work, since no current client hits it.

## 8. What the sync test suites can and cannot catch

A green suite only means something if it is known what that suite is able to fail
on. Six suites carry sync coverage; this is what each one exercises for real and
what is invisible to it by construction. The occasion for writing it down:
`sync-e2e.test.js` was 2 passed / 11 failed for two months (2026-06-24 to
2026-08-18) while every other suite stayed green, because the only suite able to
see that class of fault was the broken one.

All six pass as of 2026-09-01: 41, 17, 9, 8, 17 and 7 cases respectively.

A worktree needs `yarn workspace @peek/desktop build` before the two suites that
import `apps/desktop/dist/main/` can run at all — without it they die on
`ERR_MODULE_NOT_FOUND` rather than reporting a failure against the code.

### `apps/desktop/main/sync.test.ts` — 41 cases

Runs as runner 1 of the 6-runner desktop gate (`yarn test:desktop:electron`), and
standalone via `yarn test:unit`. Exercises the real `sync.ts` push and pull
functions against the real server app and a real `node:sqlite` datastore, and is
the only suite covering all six stages deliberately rather than incidentally
(`describe('IPC entry points — tile:sync:push / tile:sync:pull must run all three
stages')`). It also owns the paged-pull cases (`describe('Paged pulls — the
client follows the server cursor')`), which answer the item pull from a stub
rather than the real app: what is under test there is the client's loop — how
many requests it makes and when it stops.

Blind by construction to any disagreement between two separately-configured HTTP
clients: `setup()` stubs `globalThis.fetch` and routes it into the server's Hono
`app.request()` in-process, so there is no socket, no header serialization round
trip, and no second fetch implementation. That is exactly the shape of the
`sync-e2e.test.js` fault, which is why this suite could not see it.

Also blind to drift in the real IPC handlers: `replayTileSyncPush()` and
`replayTileSyncPull()` re-implement the body of `tile-ipc.ts`'s handlers, because
`tile-ipc.ts` cannot be imported under `ELECTRON_RUN_AS_NODE`. If the registered
handler's stage order changes, the replay still passes.

Covers all three recent changes: `pushedAt` per-event stamping, the parked-item
columns, and the batched push including its 404 fallback.

### `packages/integration-tests/sync-e2e.test.js` — 17 cases

`yarn test:sync:e2e`. The only suite where a real desktop client and a real server
process talk over a real socket: it spawns `apps/server/index.js` on port 3458 and
drives the compiled `sync.ts` against it. This is the suite that can see a
client/server disagreement, and the reason step 1 of restoring the foundation was
repairing it rather than deleting it.

One limit remains:

- Authentication is bypassed. The server is spawned with `E2E_TEST=true`, which
  short-circuits `deviceMiddleware` and `multiUserMiddleware` in
  `apps/server/auth.js`, so a credential-handling mismatch on the wire is invisible
  *here*. `sync-auth-e2e.test.js` below is the suite that covers it.

Tag and event stages are no longer incidental. `testTagStagesOverSocket()` and
`testEventStagesOverSocket()` call the real `pushTagsToServer()` /
`pullTagsFromServer()` / `pushEventsToServer()` / `pullEventsFromServer()` and
confirm what crossed the wire by requesting `/tags/since` and `/events/since`
directly, in both directions. `testTagsetSync()` still covers only the `tags` array
on `/items`, which is a different thing from the tag-metadata routes.
Two cases used to print a pass on every path and never inject a failure. They
were replaced with assertions of the behaviour that actually holds now:
`testDeletionTombstonePropagates()` proves a soft delete reaches the server as a
`deletedAt` tombstone and drops out of the default listing, and
`testRejectedItemIsParked()` proves a permanently rejected row records
`syncError`/`syncErrorAt`, is not re-attempted while unchanged, and pushes once
edited. That makes the parked-item mechanism the one recent change proven over a
real socket rather than only in-process.

`pushedAt` and the `/items/batch` 404 fallback are now proven here too.
`testEventPushedAtStamping()` asserts a delivered event's `pushedAt` moves off zero
while an event whose parent item has no `syncId` stays at zero — the stranding
invariant §2 describes. The 404 fallback cannot be reached against the spawned
server, which always has the route, so `testBatchFallbackWhenRouteMissing()` puts a
`node:http` proxy in front that answers `POST /items/batch` with a 404 and forwards
everything else. The proxy counts what it sees, and the test asserts on those counts
rather than on the items alone: asserting only that the items arrive would pass just
as well if the batch route had never been attempted.

### `packages/integration-tests/sync-integration.test.js` — 9 cases

`yarn test:sync`. Real HTTP to a real spawned server on port 3457, with requests
built by hand in `serverRequest()`. No desktop code is loaded at all, so this is a
server-route suite: it validates `POST /items` and `GET /items/since/:timestamp`
and nothing else. No `/tags` or `/events` route is requested. Nothing about how
the real client builds a request or reads a response is under test.

### `packages/integration-tests/sync-three-way.test.js` — 8 cases

`yarn test:sync:three-way`. Real HTTP to a real spawned server on port 3459. Both
"mobile" and "desktop" are the same hand-rolled `clientRequest()` helper with
different `profile` query params — neither the desktop `sync.ts` nor the Rust
mobile client is involved. What it actually proves is server-side profile
resolution and isolation, including that an unregistered UUID gets its own profile
rather than falling into `default`. Only `/items` is touched; no tag or event
stages, no batching, no sync-state bookkeeping.

### `packages/integration-tests/sync-version-compat.test.js` — 17 cases

`yarn test:version-compat`. Two mechanisms in one file:

- Header negotiation (`test1_bothV1` through `test8_bothNoHeaders`,
  `testHealthCheck`, `testIncrementalSyncVersionCheck`) — real HTTP to a real
  server on port 3460, but the client side is `versionedRequest()` setting
  `X-Peek-Datastore-Version` and friends by hand, not `serverFetch()`. It proves
  the server's reading of those headers, not the desktop's writing of them.
- Version-bump behaviour (`test9_initWritesVersion` through
  `test13_legacyEncodedVersionFallback`, `testServerDBVersion`) — these now drive the
  real implementations. `checkAndWriteDatastoreVersion()` is module-private in
  `datastore.ts` and returns void, so it is exercised through its real entry point,
  `initDatabase()`, with the outcome read from the `feature_settings` row and from
  `isSyncDisabledDueToVersion()`; `tamperStoredVersion()` writes the stored version
  straight into sqlite between inits to stand in for a database left by another
  binary. `testServerDBVersion()` drives the real `db.js initializeSchema()` instead
  of hand-building a `settings` table. `test13_legacyEncodedVersionFallback` covers
  the bare-numeral encoding that `JSON.parse` rejects and the implementation's
  `parseInt` fallback recovers — a branch the previous copy did not distinguish.

Only `/items` is requested; no tag or event stages.

### `packages/integration-tests/sync-auth-e2e.test.js` — 7 cases

`yarn test:sync:auth`. Spawns the real server on port 3461 in multi-user mode with
no `E2E_TEST`, so `multiUserMiddleware` and `users.resolveDevice()` actually run.
Credentials are minted at runtime through `POST /admin/devices` behind `ADMIN_TOKEN`.
It covers the 401 paths (absent header, well-formed but unminted token), a minted
key succeeding, a revoked device's key failing afterwards — the individual-revocation
guarantee — the public health check, and that device keys and `ADMIN_TOKEN` cannot
authenticate each other's routes.

The case that closes the gap is `testRealDesktopClientAuthenticates()`: it configures
the compiled `sync.js` with a genuinely minted key, pushes, and confirms the item
lands, then reconfigures with an unminted key and asserts the push fails rather than
silently reporting success. That puts the client's own header construction under
test, not just the server's reading of a hand-built header.

### Nothing runs any of this automatically

There is no CI, no `.github/` workflows, no husky config, and no installed git
hook (`core.hooksPath` unset, `.git/hooks` holds only the stock samples). Suites 2
through 5 are standalone `yarn` scripts that no aggregate script calls. Suite 1
rides the desktop gate, which itself requires explicit human approval per
`apps/desktop/CLAUDE.md` before each run. That is why a June breakage surfaced in
August.

Decision: the five non-gate suites are grouped under `yarn test:sync:all` and run
before landing any sync change, rather than being wired into a pre-commit hook.
Each spawns a server process and together they take a few minutes, which is too
slow to run on every commit; the desktop gate stays the only place suite 1 runs.

They share fixed ports — 3457, 3458, 3459, 3460, 3461 — and each suite creates a
fixed-name profile on its own server. So two runs of the same suite at once collide:
`waitForServer()` polls the port and treats any 200 as its own server, and the second
run then fails on an already-existing profile. Run the chain serially.

### The gaps this leaves

The four gaps this section previously named are closed, each by the suite noted above:
tag and event stages now run over a real socket, real authentication is exercised end
to end by `sync-auth-e2e.test.js`, `pushedAt` and the `POST /items/batch` 404 fallback
are proven over the wire, and the version-bump logic drives the real implementation
rather than a copy.

What remains:

- Nothing runs any of it automatically. The aggregate `yarn test:sync:all` is still
  something a person has to remember, which is the condition that let a June breakage
  surface in August.
- Two suites depend on `apps/desktop/dist/` existing. A worktree that has not run
  `yarn workspace @peek/desktop build` gets a module-resolution crash from them, which
  is a louder failure than a silent pass but still not a result about the code.
- The header-negotiation half of `sync-version-compat.test.js` still builds its
  requests by hand in `versionedRequest()`, so it proves the server's reading of the
  version headers, not the desktop's writing of them.
- The mobile client is not covered by any of this. `sync_e2e_mobile` is a separate
  cargo suite, and nothing exercises the Rust client against a spawned server in the
  same way `sync-e2e.test.js` covers the desktop one.

## Verifying any of this

```
grep -n '"sync": false' packages/schema/v1.json
grep -rn "source: 'user'" apps/desktop/main --include='*.ts' --exclude='*.test.ts'   # returns nothing
yarn schema:check
```

The first is the local-only inventory — now just `rules` and the two accepted tag
gaps. The second is the §4 finding: no code path produces a user-authored rule.
The third regenerates `apps/server/schema.json` from `packages/schema/v1.json` and
diffs against the checked-in copy — a clean exit is §5 holding.

## Not covered by this note

- Whether the server's backup path (`apps/server/backup.js`) changes any of the
  above. It backs up what the server holds; it cannot restore what never arrived.
- Blob and image storage coverage.
- Whether any of these gaps have already caused loss in practice — no audit of the
  live server's data was run.