ghostoff
Migrating content using the Ghost API to Offprint / Standard Site on atproto
Star
Fork
Watch
ghostoff docs development.md
3.2 kB · 66 lines
Markdown
at main

Working with the GhostOff codebase #

This doc describes the architecture of the migration tool and how the pieces fit together.

Entry point #

src/index.ts is the CLI entry point. It:

  1. Loads configuration from .env / CLI flags (src/config.ts).
  2. Loads the previous run state (src/state.ts).
  3. Authentices with the atproto PDS (src/atproto.ts).
  4. Fetches all public posts from Ghost (src/ghost.ts).
  5. For each post, downloads images, builds blocks, validates the document, writes records, and saves state.

Configuration #

src/config.ts uses commander and dotenv. Any CLI flag overrides its corresponding environment variable. Required values are validated eagerly; missing values throw before any network calls are made.

Ghost client #

src/ghost.ts is a thin wrapper around the Ghost Content API. It fetches /settings and paginates /posts with include=authors,tags&formats=html,plaintext&limit=100.

Asset pipeline #

src/assets.ts:

  • Resolves relative image URLs against the Ghost base URL.
  • Downloads images to ghostoff-export/assets/ with a SHA-256 hash filename, skipping already-cached files.
  • Uses sharp to resize/compress images over ~900 KB.
  • Uploads processed images via the raw com.atproto.repo.uploadBlob XRPC endpoint.
  • Blob references are cached in the state file so reruns don't re-upload identical images.

HTML transformation #

src/html-to-offprint.ts parses each post's HTML with linkedom and converts the body into an app.offprint.content item array. Because linkedom's parseHTML() treats fragments oddly, the HTML is wrapped in <!DOCTYPE html><html><body>...</body></html> before parsing.

src/facets.ts handles inline formatting. It recursively walks inline nodes, builds plaintext segments, and computes UTF-8 byte-offset facets for bold, italic, links, code, etc.

atproto layer #

src/atproto.ts wraps:

  • PasswordSession login/resume.
  • Blob upload (raw fetch because atcute's client is JSON-oriented).
  • com.atproto.repo.createRecord and com.atproto.repo.putRecord for documents and articles.
  • Runtime validation of site.standard.document against @atcute/standard-site.

State and idempotency #

src/state.ts reads/writes ghostoff-state.json. It maps each Ghost post uuid to the created document/article AT-URIs and CIDs. On reruns, existing records are updated via putRecord with swapRecord set to the old CID.

The state file also caches uploaded image blob references so images are only uploaded once.

Rate limiting #

src/rate-limit.ts provides small helpers for parsing RateLimit-* and Retry-After headers, plus exponential backoff. XRPC calls use @atcute/client's retryFetchHandler. Blob uploads have a manual retry loop.

Type checking and conventions #

npm run typecheck

The project uses strict TypeScript, ESM, and Node.js's native fetch. The DOM lib is included because the tool manipulates HTML elements via linkedom.

When adding new block conversions, keep the type definitions in src/html-to-offprint.ts close to the conversion logic, and make sure the emitted object has a $type discriminator matching the real Offprint lexicon.