hzz
Star
Fork
Watch
hzz docs watch-data-import.md
10 kB · 149 lines
Markdown
at main

Watch-data imports (experimental, recorded-contract acceptance pending) #

accounts/WatchHistoryImport.kt provides offline input adapters and conservative TMDB matching. app/watchimports/RoomWatchImportStore.kt, core/database/WatchImports.kt and webserver/api/WatchImport.kt connect them to owner-gated phone preview, manual-pick and explicit confirmation controls. No provider sign-in or provider writes are added. Existing Letterboxd behaviour is unchanged; see letterboxd-import.md.

Real-export compatibility is not yet verified. The phone labels both inputs experimental; Trakt accepts decoded history JSON page arrays, not an invented general export format. The caller must collect/concatenate all intended pages. Apple TV exports are unsupported.

Input contracts #

Netflix viewing-activity CSV #

The implemented dialect has exactly two English headers, Title and Date, in either order. It accepts UTF-8 text with an optional leading BOM, LF/CRLF records, quoted fields, escaped quotes and embedded newlines. Dates are explicitly month/day/year, with one or two digit month/day and a two or four digit year. Two digit years mean 2000–2099 (no moving pivot). Invalid dates, pre-1970 dates, missing/extra fields, malformed quotes, duplicate/unknown headers, empty files, more than 10,000 rows or more than 4 MiB of UTF-8 refuse the entire input with a fixed sanitized exception. There is no fallback to upload time or device timezone.

Dates are stored as UTC midnight with dateOnly=true: that is a date encoding, not evidence of an actual midnight viewing time. Raw titles are retained, including season/episode decorations. The adapter does not split titles on colons or guess a movie/show/episode from punctuation. Netflix rows have no release year or canonical identity. Exact TMDB title matches are candidates for a person to pick, never automatic matches, even when there is only one result.

Evidence gap: tests use authored, non-personal examples, not a recorded personal Netflix export. This dialect must be checked against a consented, sanitized real export before a UI claims general Netflix support. Other header languages, date orders and privacy-export formats are unsupported; do not silently reinterpret them.

Trakt decoded history #

WatchHistoryInput.trakt takes injected TraktClient.TraktHistoryEntry records. It does not call Trakt or depend on the parallel sign-in work. The caller must collect all intended pages; a list is not proof that a provider history is complete. Timestamps must be whole-second UTC with a literal Z, and at/after the epoch. Invalid timestamps refuse the whole list. The legacy TraktClient.history path is not changed by this adapter.

A movie TMDB ID is resolved in the movie namespace; a show TMDB ID in the series namespace. An episode TMDB ID is retained separately and is never used as a series ID. The existing Trakt decoded record does not carry episode coordinates: such rows preserve the show and episode title/ID but remain unresolved. Episodes without show metadata are refused rather than being mistaken for shows. A movie mixed with a show/episode is refused.

Evidence gap: Trakt tests are authored decoded records, not live recordings. Pagination, fractional timestamps, richer episode metadata, deleted/private titles and actual export formats still require recorded-contract verification. Do not describe these tests as real account reads.

Apple TV #

No parser or format contract. A privacy-export sample has not been verified. Do not invent one.

Preview and matching #

WatchHistoryPreview has no storage/account mutation. It returns every distinct source row with Matched, Ambiguous(candidates) or Unresolved, and a duplicate count. Exact duplicate facts collapse within a preview; different watch dates survive. Repeating a preview is deterministic for unchanged lookup results. Confirmation persists exact source facts idempotently (see below). Matching is cached per identity within one preview, bounded to 1,000 distinct matching keys and 300 seconds total. Bounds are checked before network requests. Lookup failures/cancellation throw and return no partial preview. Empty previews are refused.

TmdbWatchHistoryLookup searches both movie and series namespaces. Only exact normalized titles (case/whitespace), available year/type constraints, and complete search results can match automatically. Multiple fits or an incomplete search remain ambiguous. Direct ID lookups must return the requested ID and namespace with a nonblank title. Episode rows without verified coordinates remain unresolved without lookup. No search rank, first candidate or upload time is used as identity.

The models contain private viewing facts; callers must not log their values or exception causes from transports. The input exception itself has no input or parsing cause.

Durable ownership, semantics and safety #

Schema 13 adds watch_import_rows and watch_import_candidates, owned directly by a user and cascading when that user is deleted. There is no synthetic Letterboxd/provider account. Source facts are stored as deterministic v1 JSON (WatchHistoryRow declaration order, defaults included, UTF-8) with a SHA-256 fact key, unique within the user. This encoding is a persistence contract: future changes need migration/version handling, not a silent serializer/default change. The key includes source, date and precision, original title/type/IDs/year and episode facts, not TMDB match metadata. Exact duplicate facts collapse; different watch dates remain separate. Two identical Netflix watches on the same day cannot be distinguished by that source dialect.

Append, not refresh: confirmation does not remove facts missing from a partial later upload. Repeat imports retain row IDs and manual picks; other matches/candidates may be refreshed by the new preview. Confirmation and saved-row picks are Room transactions including candidate/item writes and history-feed rebuilding. Provider lookups finish before those transactions, so a provider failure does not partially mutate history. Repeated confirmation is safe. Private source facts and candidates survive restart; an unconfirmed preview deliberately does not.

At most four previews are held in memory, expiring after 15 minutes or on process restart (oldest is evicted when full). Tokens are random and user-scoped; they are not authorization. Preview picks only change that in-memory draft. Confirmation appends all facts, including unresolved ones; manual picks can also be corrected afterwards. Every picked title is resolved on the server by TMDB namespace and ID. Browser-provided title/year/metadata is never accepted.

Only movie references without source series/episode identity enter the user-owned HISTORY/watch-import feed. The feed deduplicates titles; rewatches and source dates/precision remain in source rows. Its item-source timestamp is 0 (unknown), deliberately not a fabricated viewing instant from date-only data. Series references and unresolved episodes are retained for review, not translated into series completion. No playback progress, scrobble or provider write is produced, even by a series manual pick.

Phone/API contract #

All these routes require a paired browser acting for the path user, including private GET reads. Existing same-origin/tailnet/body-type checks apply to mutations. Generic sanitized route events use only WATCH_IMPORT, never source titles/dates, row IDs, preview tokens or transport causes.

  • POST /api/v1/users/{id}/watch-import/preview?source=NETFLIX|TRAKT: strict UTF-8, at most 4 MiB; text/csv for Netflix, application/json for a decoded Trakt entry array. Returns a preview token, duplicate count, source dates/precision and matched/ambiguous/unresolved rows. Preview row IDs are one-based positions local to that token, not saved-row IDs.
  • POST .../watch-import/preview/{token}/rows/{rowId}/pick: JSON {"type":"MOVIE"|"SERIES","tmdbId":N}. Returns the updated, still-unsaved preview.
  • POST .../watch-import/confirm: JSON {"token":"..."}; returns saved source rows. The draft remains briefly available for retry; re-confirming never overwrites an existing saved pick.
  • GET .../watch-import: all saved source rows for this user only.
  • POST .../watch-import/rows/{rowId}/pick: the same pick request, resolving a saved source row.

The phone shows a visible preview, candidates/IDs, date-only labels and explicit confirmation. Manual picks are possible before and after confirmation. Lists have 100-row DOM pages; every source row remains reachable, and confirmation retains the full preview, not just the visible page. There is no file logging, analytics payload or account read. Restart/expiry requires a new preview, not an automatic confirm.

Host verification and acceptance still queued #

Host tests use authored, non-personal fixtures, injected lookup failures and local Room: parser/matcher limits, preview non-mutation/picks, provider failure atomicity, owner isolation, repeat imports/confirmations, restart/cascade, migration rollback and no series completion. Route tests cover pairing/owner/origin/content type/upload bounds and sanitized errors. Node DOM fixtures cover preview-pick/confirm, safe text rendering, paging and clearing private preview state; run node --test webserver/src/test/js/watch-import.test.cjs.

Still required through the coordinator (no device/account acceptance performed here):

  1. Obtain consented sanitized recorded Trakt pages and Netflix exports; verify timestamps, pagination, headers/localized dates, missing titles and richer episode metadata. Do not call authored fixtures real exports. Do not broaden parsing until that evidence exists.
  2. Phone/TV smoke check: pair/owner switching, upload and preview without history changes, candidate pick, confirm, repeated upload/confirm, restart and deletion; inspect light/dark layout and large-list paging. No person is currently available; this is queued, not a reason to skip host integration and not a completed acceptance claim.
  3. Apple TV privacy-export evidence is absent; no parser is planned until a format is verified.