peek
experiments in a post-browser web
Star
Fork
Watch
peek docs sync.md
14 kB
Markdown
at track/spec-mobile-sync-behaviour

Peek Sync #

Cross-platform data synchronization between mobile, desktop, and server.

Architecture #

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Mobile    │────▶│   Server    │◀────│   Desktop   │
│  (Tauri)    │     │   (Hono)    │     │  (Electron) │
└─────────────┘     └─────────────┘     └─────────────┘
      │                    │                    │
      ▼                    ▼                    ▼
   SQLite              SQLite              SQLite

The server acts as the source of truth. All clients push and pull data via the server's REST API.

Item Types #

All platforms use the same unified types:

Type Description Content Field
url Saved URLs/bookmarks The URL string
text Text content/notes The text content
tagset Tag-only items null
image Binary images Base64 data or filename

Sync Protocol #

Pull (Server → Client) #

  1. Fetch GET /items/since/:timestamp — a full pull is the same route with a timestamp of 0, so both go through the one route that pages and that always carries tombstones
    • The request carries limit, and the pull follows the nextCursor in each response until none comes back, merging each page as it lands (see Paging below)
  2. For each server item:
    • If deleted_at is set: mark local item as soft-deleted (tombstone)
    • Find local item by syncId matching server id
    • If not found: insert new item
    • If found and server is newer: update local
    • If local is newer: skip (will push later)

Push (Client → Server) #

  1. Query items where syncedAt = 0 (never synced) OR updatedAt > syncedAt (modified since last sync)
    • Includes soft-deleted items (those with deletedAt set) so tombstones propagate
    • Skips an item parked by a permanent rejection (syncError set) until its updatedAt moves past syncErrorAt
  2. POST /items/batch in chunks, each item carrying type, content, tags and its own local id
    • If an item has deletedAt: include it in the push payload so the server records the tombstone
    • The response is per row, so one refused item does not cost the rest of the chunk
    • A server without that route answers 404, and the run falls back to POST /items per item
    • Images always take the single-item route: their payload is base64 in content
  3. On success: update local syncId and syncedAt from that item's result

Conflict Resolution #

Last-write-wins based on updatedAt timestamp.

API Endpoints #

Server #

All endpoints accept a ?profile={uuid} parameter (defaults to default):

GET  /items?profile=<uuid>&includeDeleted=true  # All items (with tombstones)
GET  /items/since/:timestamp?profile=<uuid>     # Items modified after timestamp (always includes tombstones)
GET  /items/:id?profile=<uuid>                  # Single item
POST /items?profile=<uuid>                      # Create/update one item (supports deleted_at for tombstones)
POST /items/batch?profile=<uuid>                # Create/update many items in one transaction, per-row results
PATCH /items/:id/tags?profile=<uuid>            # Update tags
DELETE /items/:id?profile=<uuid>                # Hard delete item

The includeDeleted=true parameter on GET /items returns soft-deleted items (those with deleted_at set). The /items/since/:timestamp endpoint always includes tombstones so incremental sync propagates deletions.

Paging a pull #

/items/since/:timestamp, /events/since/:timestamp and /tags/since/:timestamp return the whole matching set in one body unless the client asks for a page. That body was measured at 9.91 MB on a real profile and grows without bound, so a client that knows about paging sends limit and follows the cursor:

GET /items/since/0?profile=<uuid>&limit=500              # first page
GET /items/since/0?profile=<uuid>&limit=500&cursor=<c>   # resume after the last row of the previous page
  • limit is 1..1000; anything else is a 400.
  • cursor is opaque — pass back exactly what the previous response's nextCursor held. A cursor the server cannot decode is a 400 rather than a silent restart, because it means client and server disagree about where the pull is.
  • nextCursor is present only while the page was full. An exactly-full final page still carries one, so a pull ends on a page that comes back empty.
  • The cursor carries the last row's timestamp and id. Rows can share a millisecond, and a cursor holding only a timestamp would either skip the rest of that group or repeat it forever.

Paging is opt-in on both sides, which is why it needed no datastore version bump: a client that sends no limit gets the whole set exactly as before, and a client that sends one to a server predating paging gets the whole set in one page with no nextCursor, so its loop ends after one round trip.

One consequence of routing a full pull through the since route: a row whose updatedAt is missing or unparseable reads as 0 and is therefore never returned, where GET /items would have returned it. Every write path stamps updatedAt, so this is a property of legacy rows only.

The profile parameter is a server-side profile UUID. The server uses the UUID directly as the folder name on disk. Legacy slugs are resolved to UUIDs for backward compatibility.

Profile Management:

GET  /profiles                 # List user's profiles
POST /profiles                 # Create profile
DELETE /profiles/:id           # Delete profile

Desktop IPC #

await window.app.sync.getConfig()     // Get server URL, API key
await window.app.sync.setConfig(cfg)  // Save config
await window.app.sync.pull()          // Pull from server
await window.app.sync.push()          // Push to server
await window.app.sync.full()          // Full bidirectional sync

Configuration #

Desktop #

Sync configuration is per-profile and stored in profiles.db:

  • api_key - API key (authenticates with server user)
  • server_profile_slug (mapped to serverProfileId in code) - Server profile UUID to sync to
  • last_sync_at - Last sync timestamp
  • sync_enabled - Enable sync for this profile

Server URL is environment-based (SYNC_SERVER_URL env var or default).

Each desktop profile can independently sync to different server profiles under the same server user account.

Example:

Desktop Profile    API Key        Server Profile
──────────────     ──────────     ──────────────
Work               alice's key    work
Personal           alice's key    personal

Mobile #

Sync configuration is stored in profiles.json in the App Group container:

  • sync.server_url - Server URL (top-level, used for requests)
  • sync.api_key - API key (top-level, used for auth)
  • profiles[].server_profile_id - Server profile UUID for each local profile
  • profiles[].server_url / profiles[].api_key - Per-profile sync config

The mobile sends ?profile=<server_profile_id> on sync requests. If server_profile_id is not set, falls back to the local profile UUID.

Delete Propagation #

Deletions propagate across platforms via tombstones (soft-deletes with a deleted_at timestamp).

Flow #

  1. Delete locally: Item gets deleted_at timestamp instead of being hard-deleted
  2. Push tombstone: Next sync pushes the item with deleted_at to the server
  3. Server stores tombstone: Server records deleted_at on the item
  4. Pull tombstone: Other clients pull items with includeDeleted=true, see deleted_at, and soft-delete locally

Implementation by platform #

Platform Push tombstones Pull tombstones Query param
JS engine (packages/sync/sync.js) getItemsForPush() includes deleted applyPulledItems() checks deleted_at includeDeleted=true
Electron (apps/desktop/main/sync.ts) Push query includes deleted_at IS NOT NULL Pull applies deleted_at to local items &includeDeleted=true
Tauri desktop (apps/tauri-desktop/src-tauri/src/sync.rs) Push includes soft-deleted items Pull handles deleted_at field &includeDeleted=true
Tauri mobile (apps/mobile/src-tauri/src/lib.rs) Push includes soft-deleted items Pull handles deleted_at field &includeDeleted=true
Server (apps/server/db.js) N/A getItems() respects includeDeleted param; getItemsSince() always includes tombstones N/A

Tombstone lifecycle #

  • Tombstones are permanent — they are never hard-deleted by sync
  • The deleted_at field is a Unix millisecond timestamp
  • Items with deleted_at are excluded from normal UI queries but included in sync queries

Server-Change Detection #

When a user changes sync servers (or configures sync for the first time after having synced previously), per-item sync markers (syncedAt, syncId) from the old server would prevent items from being pushed to the new server. Both desktop and mobile detect this:

  1. After every pull or full sync, the current server URL and profile ID are saved to a settings table (lastSyncServerUrl, lastSyncProfileId).
  2. Before syncAll(), the stored values are compared to the current config.
  3. If they differ (server changed), all per-item sync markers are reset — making every item eligible for push.
  4. If no stored values exist (first-time sync), no reset occurs — items pulled in a prior pull-only sync retain their markers. This prevents duplicates when pullFromServer() runs before syncAll() has stored the config.

This ensures no data loss when switching servers and no duplicates on first sync.

Files:

  • packages/sync/sync.js: resetSyncStateIfServerChanged(), saveSyncServerConfig() (unified engine)
  • apps/desktop/main/sync.ts: resetSyncStateIfServerChanged(), saveSyncServerConfig() (desktop)
  • apps/mobile/src-tauri/src/lib.rs: reset_sync_state_if_server_changed(), save_sync_server_config() (mobile)

Deduplication #

Sync Path (syncId-based only) #

When items arrive via sync (POST /items with sync_id), dedup uses syncId matching only:

  • If an item with the same sync_id exists: update it
  • If no match: create a new item (even if content is identical)

This means the same URL saved independently on two devices creates two separate server items (each with its own sync_id). This is intentional — sync_id is the canonical identity.

Non-Sync Path (content-based) #

When items arrive without a sync_id (e.g., browser extension, API calls):

  • URL items: dedup by type + content
  • Tagset items: dedup by matching tag set
  • Text items: dedup by type + content

fromISOString #

The server returns integer timestamps (Unix ms). The fromISOString() helper in both packages/sync/sync.js and apps/desktop/main/sync.ts handles both integer and ISO string inputs.

E2E Full Sync Test #

scripts/e2e-full-sync-test.sh (yarn test:e2e:full-sync) runs a clean-room three-way sync test:

Phase 1 — Three-way sync (6 items):

  1. Starts a fresh temp server with 2 seeded items
  2. Configures a desktop profile and seeds 2 items, pushes to server
  3. Creates a fresh iOS simulator database with 2 items
  4. Polls the server waiting for iOS to push (manual step: build in Xcode, sync in simulator)
  5. Triggers desktop re-sync to pull iOS items
  6. Verifies all three platforms have 6 items

Phase 2 — Sync dedup testing (10 items): 7. Re-pushes an existing item with its own sync_id → verifies dedup (still 6) 8. Seeds identical cross-device URL into desktop + iOS (no sync_id) 9. Seeds identical cross-device tagset into desktop + iOS (no sync_id) 10. Desktop pushes new items to server 11. iOS syncs (manual step) 12. Verifies 10 items on all platforms (no content dedup in sync path) 13. Re-sync stability check: 0 pulled, 0 pushed (no growth)

Phase 3 — One-time dedup cleanup: 14. Seeds 3 duplicate items into each platform's database 15. Restarts server → dedup migration runs, removes duplicates 16. Restarts desktop → dedup migration runs 17. Force-quit/relaunch iOS (manual step) → dedup migration runs 18. Verifies all platforms removed duplicates and set dedup_cleanup_v1 flag 19. Idempotency check: restart server again, verify flag prevents re-run

Phase 4 — Delete propagation (tombstone sync): 20. Deletes a desktop item → desktop sync pushes tombstone to server 21. iOS syncs (manual step) → pulls tombstone, soft-deletes locally 22. Deletes an iOS item (via direct DB write after app termination) 23. iOS relaunches and syncs (manual step) → pushes tombstone to server 24. Desktop sync pulls iOS tombstone 25. Verifies tombstone counts on all platforms

All data is temporary and cleaned up on exit (including iOS simulator backup/restore).

Known Limitations #

Deletes Not Synced (FIXED) #

Deletions now propagate via tombstones. See Delete Propagation above.

Push Failures Not Retried (HIGH) #

Failed push operations are logged but not retried. After sync, lastSyncTime advances, so failed items won't be picked up on next sync.

Workaround: Manually trigger sync if network issues occurred.

Files #

  • packages/sync/sync.js - Unified cross-platform sync engine (JS, used by mobile/web)
  • apps/desktop/main/sync.ts - Desktop sync implementation (Electron-specific)
  • apps/server/db.js - Server database with sync support
  • packages/integration-tests/sync-e2e.test.js - Server + desktop headless e2e sync tests
  • apps/mobile/src-tauri/src/lib.rs - Mobile sync (pull/push/sync_all)
  • scripts/e2e-full-sync-test.sh - Full three-way e2e test (server + desktop + iOS simulator)
  • notes/sync-edge-cases.md - Detailed edge case documentation