# Glean - Design Document

## 1. Overview

Glean is a social RSS reader built on the AT Protocol. It operates as an **AppView** for the `at.glean.*` lexicon namespace: it indexes records from Jetstream, serves XRPC query endpoints, and provides the web UI at [glean.at](https://glean.at).

Users store their RSS feed subscriptions as individual lexicon records on their PDS (one record per feed). Glean's AppView consumes Jetstream, indexes those records, fetches the referenced RSS feeds, and serves both the reader UI and public XRPC APIs for the `at.glean.*` namespace.

The core idea: your RSS subscriptions are a strong signal about your interests. When enough people expose theirs, you can discover both **people** (who reads the same things) and **content** (what similar readers follow that you don't).

## 2. Stack

| Layer            | Technology                                                       |
| ---------------- | ---------------------------------------------------------------- |
| Edge             | Cloudflare Worker: SvelteKit SSR + routing to the container      |
| Backend          | Go, running as a single Cloudflare Container                     |
| Database         | Cloudflare D1 (SQL + FTS5, via Worker binding proxy) + Vectorize |
| Frontend         | SvelteKit (adapter-cloudflare, SSR) + TailwindCSS v4             |
| Auth             | AT Protocol OAuth / DID resolution (configurable PLC directory)  |
| AT Protocol role | AppView for `at.glean.*` lexicons                                |
| Data source      | AT Protocol Jetstream → SQLite index                             |

## 3. AT Protocol Lexicons

All user data lives on their PDS. The server does not own user data; it indexes and aggregates it.

### 3.1 `at.glean.subscription`

A single RSS feed subscription. One record per feed per user. Created automatically when a user subscribes to a feed. During onboarding, existing subscriptions can be bulk-imported from an OPML file: the OPML is parsed and individual subscription records are created.

```json
{
  "lexicon": 1,
  "id": "at.glean.subscription",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "A single RSS feed subscription.",
      "record": {
        "type": "object",
        "required": ["feedUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "title": { "type": "string" },
          "category": { "type": "string" }
        }
      }
    }
  }
}
```

#### OPML Import/Export

OPML is **not** part of the lexicon. It is only used as a transport format:

- **Import (onboarding)**: User uploads an OPML file. Glean parses it, validates each feed URL, creates individual `at.glean.subscription` records on the user's PDS, and indexes them locally.
- **Export (offboarding)**: Glean reads the user's `at.glean.subscription` records from their PDS and generates an OPML file for download. The user's repository remains the canonical source.

### 3.2 `at.glean.annotation`

Reading notes on an article: quote a passage, tag it, rate it, or write a note. A user can have many annotations per article. These are public records on the PDS; users can always share an annotation on Bluesky if they want discussion.

```json
{
  "lexicon": 1,
  "id": "at.glean.annotation",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "Reading note on a specific RSS article.",
      "record": {
        "type": "object",
        "required": ["feedUrl", "articleUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "articleUrl": { "type": "string" },
          "quote": { "type": "string", "maxGraphemes": 5000 },
          "note": { "type": "string", "maxGraphemes": 500 },
          "tags": {
            "type": "array",
            "items": { "type": "string", "maxGraphemes": 50 },
            "maxLength": 10
          },
          "rating": { "type": "integer", "minimum": 1, "maximum": 5 }
        }
      }
    }
  }
}
```

### 3.3 `at.glean.like`

A user likes an article. The liked feed surfaces popular articles and feeds into discovery. Likes also feed into the recommendation system.

```json
{
  "lexicon": 1,
  "id": "at.glean.like",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "Like an RSS article.",
      "record": {
        "type": "object",
        "required": ["feedUrl", "articleUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "articleUrl": { "type": "string" }
        }
      }
    }
  }
}
```

### 3.4 `at.margin.note` (External)

Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records: margin notes appear alongside glean annotations on article detail pages.

#### Ingestion (margin.at → Glean)

Margin notes are indexed from both Jetstream and PDS sync, same as glean records. The mapping from margin note to glean annotation:

| Margin note field              | Annotation field | Notes                                                           |
| ------------------------------ | ---------------- | --------------------------------------------------------------- |
| `target.source`                | `article_url`    | W3C SpecificResource source URL                                 |
| `body.value`                   | `note`           | Text content of the annotation                                  |
| `target.selector.exact`        | `quote`          | TextQuoteSelector exact text                                    |
| `tags`                         | `tags`           | Direct mapping                                                  |
| `createdAt`                    | `created_at`     | Direct mapping                                                  |
| _(looked up from articles DB)_ | `feed_url`       | Resolved by matching `target.source` against known article URLs |

When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`.

#### Mirroring (Glean → margin.at)

When a user creates an annotation through Glean, two records are written to the user's PDS:

1. `at.glean.annotation`: the primary record (canonical URI for the annotation)
2. `at.margin.note`: mirror for interoperability with margin.at clients

The conversion from glean annotation to margin note uses `NewMarginNoteRecord`:

| Glean annotation field | Margin note field       | Notes                            |
| ---------------------- | ----------------------- | -------------------------------- |
| `articleUrl`           | `target.source`         | Direct mapping                   |
| `quote`                | `target.selector.exact` | Wrapped in a `TextQuoteSelector` |
| `note`                 | `body.value`            | `body.format` = `"text/plain"`   |
| `tags`                 | `tags`                  | Direct mapping                   |
| _(constant)_           | `motivation`            | Always `"commenting"`            |

The margin note mirror is fire-and-forget: if it fails, the glean annotation still succeeds. The Jetstream consumer will pick up the margin note create event, but `handleMarginNote` skips it via `AnnotationExistsByContent` (checks `author_did` + `article_url` + `quote` + `note`) to prevent duplicate annotations.

### 3.5 `app.skyreader.feed.subscription` (External)

Glean also indexes records from the Skyreader lexicon (`app.skyreader.feed.subscription`). When a user has subscriptions in Skyreader, they are imported as Glean subscriptions during PDS sync and via Jetstream events. This lets users who previously used Skyreader seamlessly transition to Glean without re-subscribing to their feeds.

The mapping from Skyreader subscription to Glean subscription:

| Skyreader field | Glean field | Notes                               |
| --------------- | ----------- | ----------------------------------- |
| `feedUrl`       | `feed_url`  | Direct mapping                      |
| `title`         | `title`     | Direct mapping                      |
| `siteUrl`       | `site_url`  | Stored on the feed record           |
| `createdAt`     | `added_at`  | Direct mapping                      |
| _(none)_        | `category`  | Empty (Skyreader has no categories) |

If a Glean subscription already exists for the same `feed_url`, the existing one is kept. If the existing subscription has no URI (was created locally without PDS sync), the Skyreader URI/CID is backfilled.

### 3.6 `app.bsky.graph.follow` (External)

Follow relationships are tracked from Bluesky and Tangled follow records. The `FollowRecord` struct is validated against the lexicon at `lexicons/app/bsky/graph/follow.json`. The optional `via` field (a strong ref) is preserved as raw JSON but not used by Glean.

### 3.7 Lexicon Constants

All collection NSIDs are defined as constants in `lexicon.go` and used throughout the codebase:

```go
const (
    CollectionSubscription          = "at.glean.subscription"
    CollectionAnnotation            = "at.glean.annotation"
    CollectionLike                  = "at.glean.like"
    CollectionMarginNote            = "at.margin.note"
    CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
    CollectionBskyFollow            = "app.bsky.graph.follow"
    CollectionTangledFollow         = "sh.tangled.graph.follow"
)
```

### 3.8 AppView Query Lexicons

As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.

#### `at.glean.listSubscriptions`

List subscriptions from a repo, with optional filtering.

```
Input:
  repo: string (DID of the user)
  category?: string
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  subscriptions: [{ uri, cid, value: at.glean.subscription#main, indexedAt }]
```

#### `at.glean.listFeedLists`

List subscription lists from multiple repos, with optional filtering.

```
Input:
  actors?: string[] (filter by DIDs)
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  feeds: [{ did, subscriptionCount, subscriptions: [{ feedUrl, title, category }] }]
```

#### `at.glean.listAnnotations`

List annotations for an article, a feed, or by a user.

```
Input:
  feedUrl?: string
  articleUrl?: string
  author?: string (DID)
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  annotations: [{ uri, cid, author: { did, handle }, value, indexedAt }]
```

#### `at.glean.listLikes`

List liked articles, optionally filtered by user or feed.

```
Input:
  author?: string (DID)
  feedUrl?: string
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  likes: [{ uri, cid, author: { did, handle }, value: at.glean.like#main, indexedAt }]
```

#### `at.glean.getTrending`

Articles with the most likes, forming the community feed.

```
Input:
  limit?: integer (default 50, max 100)
  cursor?: string
  since?: string (datetime)

Output:
  cursor?: string
  articles: [{ feedUrl, articleUrl, title, likeCount, annotations: [...] }]
```

#### `at.glean.getRecommendations`

Get feed recommendations for a user based on clustering.

```
Input:
  repo: string (DID of the user, query parameter)
  limit?: integer (default 20, max 50)

Output:
  feeds: [{ feedUrl, title, siteUrl, description, subscriberCount, score }]
  people: [{ did, handle, displayName, avatar, jaccard, commonFeeds, isFollowed }]
```

### 3.9 AppView Jetstream Consumption

Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream1.eurosky.network`):

```
SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note", "app.skyreader.feed.subscription", "site.standard.document"]
```

Every collection the handler processes is subscribed network-wide, including events by Glean's own users, which arrive through those same collections. No `wantedDids` filter is used: the `users` table holds every follow target (see 7.7), so a DID list would grow into megabytes of query string and a rejected WebSocket handshake. This keeps live sync with Skyreader subscriptions, margin.at notes, and the social graph.

The follow firehose is mainnet-scale, so follow edges are storage-gated to the slice the distance BFS actually traverses: edges with a reader user on either end. A `users` row exists only when a DID has a reader record (`at.glean.*`, `skyreader.*`, or `margin.*`: subscriptions, likes, and annotations), so a reader's out-edges resolve their 1-hop recommendations and edges into readers resolve 2-hop paths for whoever follows the follower. Every distance query joins `follow_distances` against `subscriptions` or `likes`, so distances to non-readers are never read. Everything else would mirror the network's graph without influencing any output; 3-hop coverage is partial, which its 0.1 signal weight absorbs. Articles for `site.standard.publication` feeds arrive via the feed scheduler (PDS polling through `StandardSiteFetcher`, atproto feeds, not RSS); the `site.standard.document` stream adds real-time create/update/delete for already-known publication feeds. Documents for unknown publications are dropped so the network-wide stream cannot mirror every publication into the article index. (`site.standard.publication` events are not subscribed: no handler ever consumed them.)

Records created while the consumer is offline are covered by PDS sync. The maintenance pass enforces the storage gates on already-stored rows.

On each event:

- **create**: Insert record into local SQLite, update materialized counts
- **delete**: Tombstone the record (soft delete to preserve foreign key integrity)
- **update**: Replace the record's CID and value

The AppView does not handle writes. Users write records to their own PDS. Glean only reads them from Jetstream.

## 4. RSS Reader

Glean is first and foremost an RSS reader. It fetches, parses, and stores articles from RSS, Atom, and JSON feeds so users can read them in a clean interface.

### 4.1 Feed Fetching

A background scheduler polls subscribed feeds on a configurable tick. Feeds are fetched at most once per cycle regardless of how many users share them.

### 4.2 Fetch Schedule

The scheduler uses a configurable tick interval with in-flight deduplication:

- **Tick interval**: A Worker cron heartbeat (`*/15 * * * *` in `wrangler.jsonc`) invokes `POST /internal/tick`; feed fetching runs on every tick
- **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible
- **Subscriber filter**: Only feeds with at least one subscription are fetched. Liked or annotated feeds are stored as passive anchors (protected from the purge, never fetched); their articles arrive via on-demand materialization (see 6.4)
- **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again
- **Error tracking**: `error_count` increments on failure, resets to 0 on success. Feeds with high error counts are surfaced as "dead feeds" to the user.

```sql
SELECT ... FROM feeds
WHERE EXISTS (SELECT 1 FROM subscriptions WHERE feed_url = feeds.feed_url)
  AND (last_fetched_at IS NULL OR last_fetched_at <= :cutoff)
ORDER BY last_fetched_at ASC NULLS FIRST
```

### 4.3 Feed Parsing

Go's `encoding/xml` for RSS and Atom. A simple `encoding/json` for JSON Feed.

Each parser returns a normalized `Feed` and a slice of `Article` structs:

```go
type Article struct {
    FeedURL   string
    GUID      string
    Title     string
    URL       string
    Author    string
    Content   string
    Summary   string
    Published time.Time
    Updated   time.Time
}
```

Articles are deduplicated by `(feed_url, guid)`. On upsert, only the article metadata changes; reading state is preserved.

### 4.4 Article Content

Glean stores article content locally so the reading experience is fast and consistent:

```sql
CREATE TABLE articles (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    feed_url    TEXT NOT NULL,
    guid        TEXT NOT NULL,
    title       TEXT NOT NULL DEFAULT '',
    url         TEXT,
    author      TEXT,
    summary     TEXT,
    content     TEXT,
    full_content TEXT,
    published   DATETIME,
    updated     DATETIME,
    fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    language    TEXT NOT NULL DEFAULT '',
    UNIQUE(feed_url, guid)
);

CREATE INDEX idx_articles_language ON articles(language);
```

### 4.5 Read State

Read/unread state is tracked per user per article:

```sql
CREATE TABLE read_state (
    user_did    TEXT NOT NULL,
    article_id  INTEGER NOT NULL,
    is_read     BOOLEAN NOT NULL DEFAULT 0,
    read_at     DATETIME,
    PRIMARY KEY (user_did, article_id)
);

CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
```

### 4.6 Reading Experience

The `/articles` page is the main reading view:

- **River of news**: Chronological list of all unread articles across all subscriptions
- **Feed filter**: Narrow to a single feed or category
- **Mark as read**: Individual toggle or "Mark all read" (optionally scoped to one feed)
- **Like**: Endorse an article (public, synced to Bluesky PDS)
- **Open original**: Title links to the source article
- **Share**: Share to Bluesky
- **Keyboard navigation**: Single-letter route shortcuts (`g` dashboard, `a` articles, `f` feeds, `t` trending, `l` library) via `<svelte:window>` in the layout; `Escape` closes dialogs
- **Expanded view**: User setting that shows full article content inline on the articles page. Articles are automatically marked as read via `IntersectionObserver` after being visible for 3 seconds. YouTube videos are embedded inline. A duplicate like button appears at the bottom of each article. Configurable in profile settings.
- **Daily digest**: An AI-generated summary of unread articles, shown on the dashboard when enabled in profile settings. The LLM receives the titles and summaries of up to 50 unread articles and produces a grouped overview with linked references. A "Read" button marks all digest articles as read. Digests are cached per user for 24 hours and generated via singleflight to avoid duplicate LLM calls.

### 4.7 Feed Discovery from Content

Beyond the clustering system, Glean also discovers new feeds from article content:

- **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds
- **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display
- **Dead feed detection**: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it.

## 5. System Architecture

Glean is a Worker-plus-container deployment. A **Cloudflare Worker** is the public origin: it server-renders the SvelteKit app and forwards `/api/*`, `/xrpc/*`, `/metrics`, and `/ready` to one named **Cloudflare Container** running the Go AppView. The Go process still fills two roles: **AppView** (indexing `at.glean.*` records from Jetstream, serving XRPC queries) and **RSS reader** (fetching and storing feed content).

The container is a singleton (`getContainer(env.GLEAN, "glean")`, `max_instances: 1`). The AppView is a shared index plus a Jetstream websocket, so it cannot be load-balanced across interchangeable instances. The process does not self-schedule: a single Worker cron heartbeat (every 15 minutes, see `wrangler.jsonc`) invokes the authenticated `/internal/tick` endpoint, which runs whatever is due (fetch every tick, cluster on the :00 tick hourly, sync on the :00 tick every 8 hours), and any request wakes a sleeping container (first request after a quiet period pays a few seconds of cold start). `sleepAfter` (5m) is the cost dial: the container sleeps when idle, Jetstream dies with the process, and the next wake resumes the stream from the cursor stored in D1.

The container's D1 and R2 traffic goes to the virtual host `glean.proxy` (`GLEAN_D1_PROXY_URL`): the platform intercepts requests to that host from inside the container's network namespace and runs them as Worker code next to the bindings, on the same machine. `wrangler dev` mirrors the arrangement with a sidecar in the same network namespace, so the same URL works locally.

```mermaid
flowchart TD
    B[Browser] -->|"GET pages, __data.json"| W
    B -->|"/api, /xrpc, /metrics, /ready"| W

    subgraph Edge["Cloudflare edge (any colo)"]
        W["Worker: glean, SvelteKit SSR + router"]
        A[(Static assets)]
        P["outbound host glean.proxy: auth'd /d1 + /r2 proxy"]
        D1[("D1: SQL + FTS5")]
        W --> A
        P --> D1
    end

    subgraph Singleton["Singleton placement (one datacenter)"]
        DO["Durable Object 'glean'
(wakes the container)"]
        C["Container: Go AppView
chi API · feed scheduler · jetstream consumer · cluster cron · materializer"]
        DO --> C
    end

    W -->|"backend paths + cron ticks, getContainer(env.GLEAN)"| DO
    W -->|"SSR server loads, handleFetch + cookies"| DO
    C -->|"SQL as JSON over /d1 (bearer token)"| P
    C -->|"article bodies over /r2 (bearer token)"| P
    P --> R2[("R2 archive")]
    C -->|"embeddings + KNN, REST"| V[(Vectorize)]
    C -->|"fetch RSS/Atom/JSON, scrape, sync PDS"| NET[Internet]
    NET -.->|"Jetstream websocket"| C
    C -->|"writes on user action"| PDS[User PDS]
```

AppView responsibilities:

- Subscribe to Jetstream network-wide (no wantedDids filter) for `at.glean.*`, `at.margin.note`, `app.skyreader.feed.subscription`, `site.standard.document`, and follow records; follow edges are storage-gated to graph-relevant pairs
- Index records into D1
- Convert `at.margin.note` records to annotations (displayed alongside glean annotations), skip if duplicate glean annotation exists
- Mirror glean annotations as `at.margin.note` records on user PDS for interoperability
- Import `app.skyreader.feed.subscription` records as glean subscriptions
- Serve XRPC query endpoints (`at.glean.listSubscriptions`, etc.)
- Write to user PDS on behalf of the user (when the user acts through the UI)
- Materialize liked/annotated articles whose row is missing by scraping the article URL

## 6. Database Schema

All SQL runs against one database with flat table names: Cloudflare D1, reached through the Worker binding proxy in production and through `wrangler dev`'s local D1 in development. D1 exposes a single `main` schema, so the exact same statements run everywhere. The schema initializes in one transaction so the D1 driver flushes it as a single proxy batch: cold starts pay one round trip, not one per statement. The Go test suites run the same schema and statements against in-process sqlite (modernc.org/sqlite); D1 is sqlite, so only the transport differs. Content KNN is Cloudflare Vectorize, not sqlite-vec.

Foreign key constraints are not used. Referential integrity is enforced by the application layer.

Transactions over D1 are emulated by the driver: writes buffer on the connection and flush as one D1 batch (a single transaction, chunked at 100 statements to respect D1's time budget). Reads inside a transaction run immediately against committed state; every such read in the codebase either precedes the transaction's writes or touches tables the writes do not affect.

### 6.1 Users

Profile data (handle, display name, avatar) is resolved on-the-fly via AT Protocol identity resolution rather than stored locally.

```sql
CREATE TABLE users (
    did            TEXT PRIMARY KEY,
    indexed_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    follows_dirty  BOOLEAN NOT NULL DEFAULT 1
);

CREATE TABLE user_settings (
    did            TEXT PRIMARY KEY,
    languages      TEXT,
    expanded_view  BOOLEAN NOT NULL DEFAULT 0,
    digest_enabled BOOLEAN NOT NULL DEFAULT 0,
    updated_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
```

### 6.2 Feed Subscriptions

Indexed from `at.glean.subscription` records on user PDS.

```sql
CREATE TABLE subscriptions (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    user_did    TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    title       TEXT,
    category    TEXT,
    added_at    DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    uri         TEXT,
    cid         TEXT,
    UNIQUE(user_did, feed_url)
);
```

### 6.3 Feeds

Known feeds. Subscribed feeds are fetched; liked or annotated feeds are stored as passive join anchors and never fetched (their articles arrive via materialization, see 6.4).

```sql
CREATE TABLE feeds (
    feed_url        TEXT PRIMARY KEY,
    title           TEXT,
    site_url        TEXT,
    description     TEXT,
    feed_type       TEXT CHECK(feed_type IN ('rss', 'atom', 'json', 'atproto')),
    last_fetched_at DATETIME,
    last_error      TEXT,
    subscriber_count INTEGER NOT NULL DEFAULT 0,
    consecutive_empty_fetches INTEGER NOT NULL DEFAULT 0,
    error_count     INTEGER NOT NULL DEFAULT 0,
    favicon_url     TEXT
);
```

### 6.4 Articles

Fetched from RSS and atproto feeds that have at least one subscription network-wide.

```sql
CREATE TABLE articles (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    feed_url    TEXT NOT NULL,
    guid        TEXT NOT NULL,
    title       TEXT NOT NULL DEFAULT '',
    url         TEXT,
    author      TEXT,
    summary     TEXT,
    content     TEXT,
    full_content TEXT,
    published   DATETIME,
    updated     DATETIME,
    fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    language    TEXT NOT NULL DEFAULT '',
    UNIQUE(feed_url, guid)
);

CREATE INDEX idx_articles_language ON articles(language);
```

Article bodies are stored in two tiers. Fresh articles carry `content`
(feed-provided body) and `full_content` (scraped body) in the row. Once a row
is older than `GLEAN_ARCHIVE_WINDOW` days (default 90; the deployment sets 3),
the maintenance cycle uploads each body to R2 (`articles/<id>/content`,
`articles/<id>/full`) and NULLs the columns. Article reads hydrate NULL bodies
from R2 transparently, so lists, article pages, search results, and digests
behave the same across the tiers. If a body exists nowhere (a pre-R2 row, or
an offload that failed before the upload), the article page re-fetches the
body from the source URL once, stores it through the normal path, and serves
it; the next read hydrates from R2. A failed re-fetch is retried at most once
per hour per article, so dead sources do not add scrape latency to every
view. The columns stay NULL-safe: a body that cannot be recovered reads as
empty and the UI falls back to the summary plus the fetch-content action.

The archive exists because D1 caps a database at 10 GB and feed bodies are the
bulk of the row; R2 is an order of magnitude cheaper per byte and article
content should be kept forever, exactly as fetched (annotations quote the
body, so a re-fetch of a since-edited page would break highlight alignment).
Bodies were never searchable (the FTS index covers title, summary, and author
only), so archiving an article changes nothing about search.

D1 size is bounded at every stage. A single fetch inserts at most the 50
newest items per feed (first fetches otherwise dump entire feed histories);
the article URL is the row identity (guid) whenever one exists (at:// record
URIs excepted), so feeds that regenerate their GUIDs cannot re-insert their
window every tick; articles already present are filtered out before any
INSERT is sent, and so are items whose publish date is already past the
retention window (maintenance purges those within the hour);
summaries are stored as a ~1000-character plain-text excerpt,
with the full text of description-only feeds moved into `content`; articles
older than `GLEAN_ARTICLE_RETENTION_DAYS` days (default 30, 0 disables) are
deleted unless a like or annotation references them, and undated articles
never age out (their publish date is unknown, so age-purging them resurrected
every item still in a feed's window as unread once per retention period; the
per-feed cap and the unreferenced-feed purge bound them instead); each feed
keeps at most 500 article rows; and unreferenced feeds are purged with their
articles.

Articles referenced by a like or annotation but absent locally (they already
rotated out of the feed's item window, or the feed is not subscribed) are
**materialized** on demand: the `article_url` is scraped and inserted through
the normal upsert path, using the article URL as GUID, the page title (or a
humanized URL slug) as title, and a plain-text excerpt as summary. Liked and
annotated feeds are passive, stored to anchor the joins and never fetched,
and their articles survive the retention purge as long as the like or
annotation exists. If the like is later deleted, the feed and articles become
unreferenced and the hourly purge removes them.

### 6.5 Read State

```sql
CREATE TABLE read_state (
    user_did    TEXT NOT NULL,
    article_id  INTEGER NOT NULL,
    is_read     BOOLEAN NOT NULL DEFAULT 0,
    read_at     DATETIME,
    PRIMARY KEY (user_did, article_id)
);

CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
```

### 6.6 Annotations, Likes

Local mirror of AT Protocol lexicon records for fast querying.

```sql
CREATE TABLE annotations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    uri         TEXT NOT NULL UNIQUE,
    author_did  TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    article_url TEXT NOT NULL,
    quote       TEXT,
    note        TEXT,
    tags        TEXT,
    rating      INTEGER,
    created_at  DATETIME NOT NULL,
    cid         TEXT
);

CREATE TABLE likes (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    uri         TEXT NOT NULL UNIQUE,
    author_did  TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    article_url TEXT NOT NULL,
    created_at  DATETIME NOT NULL,
    cid         TEXT,
    UNIQUE(author_did, feed_url, article_url)
);
```

### 6.7 Cluster Precomputation

Stores precomputed similarity data to avoid recalculating on every request.

```sql
CREATE TABLE user_similarity (
    user_a     TEXT NOT NULL,
    user_b     TEXT NOT NULL,
    jaccard    REAL NOT NULL,
    common_feeds INTEGER NOT NULL,
    common_likes INTEGER NOT NULL DEFAULT 0,
    common_tags INTEGER NOT NULL DEFAULT 0,
    computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_a, user_b),
    CHECK(user_a < user_b)
);
```

### 6.8 Follows

Tracks follow relationships between users (from `app.bsky.graph.follow` and `sh.tangled.graph.follow` records).

```sql
CREATE TABLE follows (
    user_did    TEXT NOT NULL,
    target_did  TEXT NOT NULL,
    uri         TEXT,
    cid         TEXT,
    followed_at DATETIME,
    PRIMARY KEY (user_did, target_did)
);

CREATE INDEX idx_follows_target ON follows(target_did);
CREATE INDEX idx_follows_uri ON follows(uri);
```

### 6.9 OAuth Storage

Login attempts store their handshake state until the callback consumes it;
sessions store the OAuth tokens per login. Maintenance deletes abandoned
requests after 7 days and sessions untouched for 90 days, so both tables stay
bounded.

```sql
CREATE TABLE oauth_auth_requests (
    state      TEXT PRIMARY KEY,
    data       TEXT NOT NULL,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE oauth_sessions (
    account_did TEXT NOT NULL,
    session_id  TEXT NOT NULL,
    data        TEXT NOT NULL,
    updated_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (account_did, session_id)
);
```

## 7. Recommendations

Glean uses a multi-signal recommendation system that combines subscription overlap, like patterns, social graph distance, and user behavior feedback.

### 7.1 Signals

| Signal       | Source                     | Weight (default) | Description                                             |
| ------------ | -------------------------- | ---------------- | ------------------------------------------------------- |
| Subscription | `subscriptions`            | 1.0              | Jaccard over subscriber sets between similar users      |
| Like         | `likes`                    | 0.5              | Time-decayed like co-occurrence (30-day half-life)      |
| Tag          | `annotations.tags`         | 0.3              | Jaccard over annotation tag sets                        |
| Social       | `follow_distances`         | 0.7              | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1        |
| Popularity   | `feeds.subscriber_count`   | 0.2              | `log(1 + subscribers) / log(1 + max)`                   |
| Category     | `subscriptions.category`   | 0.4              | Boost feeds matching user's existing categories         |
| Content      | Vectorize `glean-articles` | 0.4              | Cosine similarity via embedding KNN (requires embedder) |

### 7.2 Feed Co-occurrence (Jaccard Similarity)

For any two feeds, the similarity is the Jaccard index of their subscriber sets:

```
J(A, B) = |subscribers(A) ∩ subscribers(B)| / |subscribers(A) ∪ subscribers(B)|
```

Feed description similarity is also computed via embedding cosine similarity (requires embedder) and added as a boost.

### 7.3 User Similarity

For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:

```
J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost
```

Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life).

### 7.4 Precomputed + On-Demand Scoring

Recommendations are **precomputed** for all active users on every cluster cron cycle (hourly; see 7.10) and stored in the `precomputed_recommendations` table

**Feed recommendation score** (computed in SQL):

```
score = sub_signal * w_sub
      + like_signal * w_like
      + social_signal * w_social
      + pop_signal * w_pop
      + category_signal * w_category
```

Where:

- `sub_signal = SUM(jaccard(target, U))` for similar users U subscribed to feed
- `like_signal = SUM(jaccard(target, U) * time_decay)` for likes in that feed by similar users
- `social_signal = SUM(distance_weight)` from follow_distances
- `pop_signal = log(1 + subscriber_count) / log(1 + max_subscribers)`
- `category_signal = 1` if feed description matches user's top categories

**Article recommendation score**:

```
score = like_signal * w_like
      + social_signal * w_social
      + content_signal * w_content
      + recency_signal * 0.2
```

Content signal uses embedding vectors: the user's liked article embeddings are averaged into a single interest vector, then a KNN query against the `glean-articles` Vectorize index finds semantically similar articles. This requires an embedder to be configured; without it, the content signal is 0.

**Language filtering**: Users can set preferred languages on their profile (stored in the `user_settings` table as a JSON array of ISO 639-1 codes). When set, article recommendations are filtered to only include articles whose `language` column matches one of the selected codes **or** whose `language` is empty (not yet classified). This ensures users with language preferences still see all recommendations when the LLM hasn't run or hasn't classified certain articles yet. If no languages are set (empty/nil), all articles are shown regardless of language.

### 7.5 User Feedback (Dismiss)

Users can dismiss recommendations they don't want to see again:

- `POST /feeds/dismiss`: dismiss a feed recommendation
- `POST /articles/dismiss`: dismiss an article recommendation
- Dismissals are stored locally in `dismissed_recommendations` (not on PDS)
- Dismissed items are excluded from all future recommendation queries
- Auto-dismiss: items shown ≥5 times over >5 days without action are auto-dismissed

Recommended articles use the same card template as regular articles, with an additional "Hide" button that dismisses the recommendation. Dismissing only removes that specific article from recommendations; it does not affect the user's likes or signal weights, and similar articles may still be recommended.

### 7.6 Auto-Tuned Signal Weights

Each user has a row in `user_signal_weights` with per-signal weights. When a user acts on a recommendation (subscribes, likes), the dominant signal that produced that recommendation is rewarded:

```
new_weight = MAX(0.1, MIN(3.0, old_weight * (1 + learning_rate * delta)))
```

- `learning_rate = 0.1`, `delta = +1` for reward, `-1` for penalty
- Only activates after `minActionsTune = 5` positive actions
- Defaults are used when no row exists for a user

### 7.7 Social Graph

Follow distances (1-hop through 3-hop) are computed incrementally, for reader users only: a `users` row exists only when a DID has a reader record, and every query reads `user_a = <requesting user>` joined against `subscriptions` or `likes`. A `follows_dirty` column on `users` tracks whose follow graph changed since the last cron run. Only dirty users are reprocessed: their existing rows in `follow_distances` are deleted and recomputed via BFS, then the dirty flag is cleared. The `follows` table stores only the out-edges the BFS traverses: edges with a reader on either end (a reader's out-edges for 1-hop, edges into readers for 2-hop), so 3-hop coverage is partial. Maintenance purges distance rows whose `user_a` no longer has a `users` row.

- 1-hop: direct follows (weight 1.0)
- 2-hop: friends-of-friends (weight 0.3)
- 3-hop: third-degree connections (weight 0.1)

### 7.8 Diversity & Freshness

After scoring, diversity filtering is applied in Go (not SQL):

- **Domain diversity**: max 2 feeds from the same domain in results
- **Category diversity**: max 3 feeds from the same category in results
- This prevents recommendation clustering on a single source

### 7.9 Cold Start

New users with <5 subscriptions get a fallback strategy:

1. Feeds from 1-hop followed users (70% weight)
2. Globally popular feeds by subscriber count (30% weight)

### 7.10 Clustering Engine (Cron)

The heartbeat's :00 tick (hourly) runs the cluster pass. Maintenance runs **first**, so the computation steps below run on the rows that survive instead of embedding and scoring articles that are about to expire:

1. **DB maintenance**: Prune `recommendation_impressions` older than 14 days; prune OAuth login attempts older than 7 days and sessions untouched for 90 days; purge follow edges that fail the storage gate (neither endpoint is a reader user, answered from an in-process known-users cache the jetstream gate also uses); purge `follow_distances` rows whose `user_a` has no `users` row; delete articles older than `GLEAN_ARTICLE_RETENTION_DAYS` days unless a like or annotation references them (undated articles are exempt; the per-feed cap and the unreferenced-feed purge remove them); trim every feed to its 500 newest articles; delete feeds with no subscription, like, or annotation along with their articles; delete `read_state` rows pointing at removed articles; then run `PRAGMA incremental_vacuum` on the local file (D1 manages its own storage). All deletes run in rowid-keyed chunks so maintenance never holds a long write transaction.
2. **Offload article bodies**: Articles older than `GLEAN_ARCHIVE_WINDOW` days (default 90; the deployment sets 3) get `content`/`full_content` uploaded to R2 and the columns NULLed (see 6.4). Uploads precede the UPDATE so an interrupted pass is idempotent.
3. **Compute feed embeddings**: Embed new feed descriptions via embedding API into `feed_embeddings` table (skipped if no embedder configured)
4. **Compute user similarity**: Diff-update `user_similarity` (subscription Jaccard + time-decayed likes + tags + follow boost; only rows whose value actually changed are written)
5. **Compute article embeddings**: Embed new articles (`title + summary + content`, excluding `full_content` to stay within embedding model token limits) via embedding API into the `glean-articles` Vectorize index (skipped if no embedder configured)
6. **Detect article languages**: Batch-classify article languages via LLM, updating the `language` column (skipped if no LLM configured)
7. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table), committed one user per transaction so a long run never holds the recs writer lock for the whole job, and a failure keeps completed users; each recompute writes only new, changed, or removed pairs
8. **Compute signal profiles**: Per-user category/tag/like summaries (diff-updated the same way)
9. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action
10. **Precompute recommendations**: For each active user, compute feed, article, and people recommendations and store as JSON in the `precomputed_recommendations` table. This ensures instant load times for all recommendation sections.
11. **Log purge stats**: The cron logs how many follows, articles (unreferenced, expired, overage), feeds, and read_state rows maintenance removed, plus how many article bodies were offloaded

Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.

### 7.11 User Interaction Tables

Impressions feed the auto-dismiss pass and the per-user signal-weight tuning. Maintenance deletes impressions older than 14 days (the auto-dismiss window is 5 days).

```sql
CREATE TABLE dismissed_recommendations (
    user_did     TEXT NOT NULL,
    target_type  TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
    target_id    TEXT NOT NULL,
    reason       TEXT,
    dismissed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_did, target_type, target_id)
);

CREATE TABLE recommendation_impressions (
    user_did       TEXT NOT NULL,
    target_type    TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
    target_id      TEXT NOT NULL,
    first_shown_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    last_shown_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    shown_count    INTEGER NOT NULL DEFAULT 1,
    acted          BOOLEAN NOT NULL DEFAULT 0,
    PRIMARY KEY (user_did, target_type, target_id)
);
```

### 7.12 Computed Recommendation Tables

Written exclusively by the cron. Read during recommendation requests (precomputed results served first, on-demand fallback if missing). The similarity tables are defined in 6.7.

```sql
CREATE TABLE follow_distances (
    user_a   TEXT NOT NULL,
    user_b   TEXT NOT NULL,
    distance INTEGER NOT NULL CHECK(distance IN (1, 2, 3)),
    PRIMARY KEY (user_a, user_b)
);

CREATE TABLE user_signal_weights (
    user_did   TEXT PRIMARY KEY,
    w_sub      REAL NOT NULL DEFAULT 1.0,
    w_like     REAL NOT NULL DEFAULT 0.5,
    w_tag      REAL NOT NULL DEFAULT 0.3,
    w_social   REAL NOT NULL DEFAULT 0.7,
    w_pop      REAL NOT NULL DEFAULT 0.2,
    w_category REAL NOT NULL DEFAULT 0.4,
    w_content  REAL NOT NULL DEFAULT 0.4,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE user_signal_profiles (
    user_did       TEXT PRIMARY KEY,
    total_likes     INTEGER NOT NULL DEFAULT 0,
    total_tags      INTEGER NOT NULL DEFAULT 0,
    top_categories  TEXT,
    updated_at      DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE precomputed_recommendations (
    user_did    TEXT NOT NULL,
    rec_type    TEXT NOT NULL CHECK(rec_type IN ('feed', 'article', 'person')),
    data        TEXT NOT NULL,
    computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_did, rec_type)
);
```

### 7.13 Embeddings (recommended)

When `GLEAN_EMBED_BASE_URL` is configured, article text and feed descriptions are embedded and upserted into Vectorize (`glean-articles`, `glean-feeds`). SQL keeps `article_embedding_ids` and `feed_embedding_meta` so cron knows what is already indexed. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph.

The embedding model must output exactly the Vectorize index dimensions (1536 in the setup commands): nothing reshapes or truncates vectors, so a mismatched model fails the upsert loudly in the cron logs.

The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Gemini, Ollama, local inference servers).

### 7.14 LLM Client (optional)

When `GLEAN_LLM_BASE_URL` is configured, an LLM client is available for text classification and summarization tasks. It uses the same `github.com/openai/openai-go` SDK pointed at any OpenAI-compatible `/v1/chat/completions` endpoint.

**Language detection**: The cron job calls `DetectLanguages` in batches of up to 100 articles per request. For each article, the title and summary (truncated to 500 characters) are sent with a prompt asking for ISO 639-1 codes. Results are written to `articles.language`. Articles that are already classified (non-empty `language`) are skipped. An empty response from the LLM defaults to `'unknown'` rather than a specific language, ensuring unclassifiable articles aren't miscategorized.

Without an LLM, all articles remain at the default empty language value and language-based filtering is unavailable. The daily digest feature also requires an LLM to generate summaries; without it, the digest is unavailable.

SQL tracks which vectors exist:

```sql
CREATE TABLE feed_embedding_meta (
    feed_url TEXT PRIMARY KEY,
    source_text TEXT NOT NULL DEFAULT ''
);

CREATE TABLE article_embedding_ids (
    article_id INTEGER PRIMARY KEY
);
```

During cron, `ComputeArticleEmbeddings` embeds new articles in batches using `title + summary + content` and upserts them to Vectorize. `ComputeFeedEmbeddings` embeds feed descriptions and re-embeds when source text changes. On-demand article recs average liked-article vectors and query Vectorize. Cold-start feed recs do the same with subscribed feeds.

## 8. HTTP API

The Go server exposes a JSON API under `/api/*`. The SvelteKit frontend (see `web/`) owns all HTML page routes and consumes these endpoints via `web/src/lib/api.ts`. The Worker forwards browser `/api/*` to the container; SvelteKit server loads reach the container through `handleFetch` during server-side rendering (cookies forwarded), and subsequent client-side navigations fetch `__data.json` from the browser. All state-changing requests require a double-submit CSRF token (`X-CSRF-Token` header or `csrf_token` form field) matching the `glean_csrf` cookie.

### 8.1 JSON endpoints

All endpoints below return JSON. Those marked 🔒 require authentication (session cookie). Form fields are sent `application/x-www-form-urlencoded`; file uploads are `multipart/form-data`.

| Route                              | Method | Auth | Description                                                                    |
| ---------------------------------- | ------ | ---- | ------------------------------------------------------------------------------ |
| `/api/me`                          | GET    |      | Current user, CSRF token, feature flags (`has_llm`, `client_id`)               |
| `/api/dashboard`                   | GET    | 🔒   | Dashboard: article recs, unread articles, trending, digest flag                |
| `/api/feeds`                       | GET    | 🔒   | Subscriptions (paginated), categories, dead feeds                              |
| `/api/feeds/add`                   | POST   | 🔒   | Add a feed URL (fields: `feed_url`, `category`)                                |
| `/api/feeds/edit`                  | POST   | 🔒   | Edit a subscription category                                                   |
| `/api/feeds/opml/upload`           | POST   | 🔒   | Bulk-import subscriptions from OPML (field: `opml` file)                       |
| `/api/feeds/opml/download`         | GET    | 🔒   | Export subscriptions as OPML                                                   |
| `/api/feeds/refresh`               | POST   | 🔒   | Refresh all subscribed feeds                                                   |
| `/api/feeds/retry`                 | POST   | 🔒   | Retry a failed feed                                                            |
| `/api/feeds/list`                  | GET    | 🔒   | Flat subscription list (optional `?category=`)                                 |
| `/api/feeds/clear`                 | POST   | 🔒   | Remove all subscriptions                                                       |
| `/api/articles`                    | GET    | 🔒   | Read articles (paginated; `?feed=`, `?status=`, `?q=`, `?sort=`, `?category=`) |
| `/api/articles/new-count`          | GET    | 🔒   | Count of articles newer than `?since=` (unix seconds); banner polling          |
| `/api/articles/{id}`               | GET    | 🔒   | Article detail + annotations (marks read as a side effect)                     |
| `/api/articles/{id}/read`          | POST   | 🔒   | Mark article read                                                              |
| `/api/articles/{id}/unread`        | POST   | 🔒   | Mark article unread                                                            |
| `/api/articles/{id}/like`          | POST   | 🔒   | Toggle like (writes/deletes on PDS)                                            |
| `/api/articles/{id}/fetch-content` | POST   | 🔒   | Scrape full article content from the source URL                                |
| `/api/articles/mark-all-read`      | POST   | 🔒   | Mark all (or `?feed=`-scoped) articles read                                    |
| `/api/trending`                    | GET    |      | Trending articles (`?scope=for-me` requires auth)                              |
| `/api/profile/{did}`               | GET    | 🔒   | Public profile: feeds, annotations, languages (resolves handles)               |
| `/api/library`                     | GET    | 🔒   | Liked articles + annotations (paginated)                                       |
| `/api/library/create`              | POST   | 🔒   | Create annotation (writes at.glean.annotation + at.margin.note mirror)         |
| `/api/library/{id}/delete`         | POST   | 🔒   | Delete annotation + mirror                                                     |
| `/api/recs/articles`               | GET    | 🔒   | Article recommendations                                                        |
| `/api/recs/feeds`                  | GET    | 🔒   | Feed recommendations                                                           |
| `/api/recs/people`                 | GET    | 🔒   | People recommendations (followed + discover)                                   |
| `/api/recs/dismiss-feed`           | POST   | 🔒   | Dismiss a feed recommendation                                                  |
| `/api/recs/dismiss-article`        | POST   | 🔒   | Dismiss an article recommendation                                              |
| `/api/recs/dismiss-person`         | POST   | 🔒   | Dismiss a person recommendation                                                |
| `/api/settings/languages/{code}`   | POST   | 🔒   | Toggle a preferred recommendation language                                     |
| `/api/settings/expanded-view`      | POST   | 🔒   | Toggle expanded article view                                                   |
| `/api/settings/digest-enabled`     | POST   | 🔒   | Toggle daily digest                                                            |
| `/api/digest`                      | GET    | 🔒   | Daily digest (LLM summary of unread articles); 204 when none/unavailable       |
| `/api/digest/mark-read`            | POST   | 🔒   | Mark digest articles read (field: repeated `ids`)                              |
| `/api/auth/login`                  | GET    |      | Whether OAuth is enabled (`oauth_enabled`)                                     |
| `/api/auth/register`               | GET    |      | Start registration via Eurosky (`{redirect}`)                                  |
| `/api/auth/actors`                 | GET    |      | Handle typeahead (`?q=`) → `{actors}`                                          |
| `/api/auth/start`                  | POST   | 🔒   | Start OAuth flow (field: `handle`) → `{redirect}`                              |
| `/api/auth/callback`               | GET    |      | OAuth callback (browser redirect, not JSON)                                    |
| `/api/auth/logout`                 | POST   | 🔒   | End session → `{redirect}`                                                     |
| `/api/oauth/client-metadata`       | GET    |      | OAuth client metadata document                                                 |
| `/api/sitemap`                     | GET    |      | Sitemap entries (JSON)                                                         |
| `/api/stats`                       | GET    |      | Prometheus metrics parsed into `{metrics}`                                     |
| `/metrics`                         | GET    |      | Raw Prometheus exposition                                                      |
| `/ready`                           | GET    |      | Container health check (204 if SQLite is reachable)                            |

XRPC query endpoints (public, no `/api` prefix):

| Route                               | Description                 |
| ----------------------------------- | --------------------------- |
| `/xrpc/at.glean.listSubscriptions`  | List a user's subscriptions |
| `/xrpc/at.glean.listAnnotations`    | List annotations            |
| `/xrpc/at.glean.listLikes`          | List likes                  |
| `/xrpc/at.glean.getTrending`        | Trending articles           |
| `/xrpc/at.glean.getRecommendations` | Recommendations             |
| `/xrpc/at.glean.listFeedLists`      | Feed lists                  |

### 8.2 Frontend routes

All HTML page routes live in the SvelteKit app (`web/src/routes/`), not on the Go server. The app server-renders (`ssr = true` in the root `+layout.ts`): each route's `+page.server.ts` load runs in the Worker during SSR, calling the JSON API via `endpointsFor(event.fetch)` (routed to the container by `handleFetch`), and the browser hydrates the result; client-side navigations fetch `__data.json`. The layout (`+layout.svelte`) renders the shared chrome and mounts the new-articles banner.

| Path             | Load source          |
| ---------------- | -------------------- |
| `/`              | `+page` (landing)    |
| `/dashboard`     | `/api/dashboard`     |
| `/articles`      | `/api/articles`      |
| `/articles/[id]` | `/api/articles/{id}` |
| `/feeds`         | `/api/feeds`         |
| `/library`       | `/api/library`       |
| `/trending`      | `/api/trending`      |
| `/profile/[did]` | `/api/profile/{did}` |
| `/stats`         | `/api/stats`         |
| `/terms`         | (static content)     |
| `/auth/login`    | `/api/auth/login`    |
| `/robots.txt`    | (static, +server.ts) |
| `/sitemap.xml`   | `/api/sitemap`       |

## 9. Project Structure

```
glean/
├── main.go                        # Entry point, wire everything
├── go.mod / go.sum
├── wrangler.jsonc                 # Worker + Container + cron + assets
├── worker/index.ts                # Route /api /xrpc to the named Go container
├── Dockerfile                     # Go API, CGO-free
├── Makefile                       # Targets: dev, deploy, test, ...
├── lexicons/
│   └── at/
│       ├── glean/                  # Glean lexicon JSON schemas (subscription, annotation, like)
│       └── margin/                 # External: at.margin.note W3C Web Annotation schema
│   └── app/bsky/graph/             # External: app.bsky.graph.follow schema
├── internal/
│   ├── atproto/
│   │   ├── auth.go                # DID resolution, OAuth flow
│   │   ├── client.go              # XRPC client (write to user PDS)
│   │   ├── collectiondir.go       # Collection directory backfill (startup)
│   │   ├── jetstream.go           # Subscribe to Jetstream via official client
│   │   ├── stream_handler.go      # Stream event → DB handler
│   │   ├── lexicon.go             # Lexicon record types (at.glean.*, maintained by hand)
│   │   ├── lexicon_external.go    # External lexicon record types (FollowRecord, MarginNoteRecord, SkyreaderSubscriptionRecord)
│   │   ├── lexicon_test.go        # Test: Go structs match lexicon JSON schemas
│   │   ├── sync.go                # PDS record reconciliation
│   │   └── xrpc.go                # XRPC query handlers (AppView endpoints)
│   ├── blob/
│   │   └── blob.go                # R2 client via the outbound /r2 proxy
│   ├── db/
│   │   ├── db.go                  # Store over D1 via the worker proxy; flat table names
│   │   ├── sqlite.go             # In-process sqlite backend for the test suites (D1 is sqlite)
│   │   ├── d1.go                  # D1-over-HTTP driver (DSN from Cloudflare creds)
│   │   ├── migrations.go          # Schema migrations
│   │   ├── blobs.go               # Article-body R2 offload + hydration
│   │   ├── cursor_store.go        # Jetstream cursor persistence
│   │   ├── user.go                # User queries
│   │   ├── feed.go                # Feed + subscription queries
│   │   ├── article.go             # Article queries
│   │   ├── social.go              # Like, annotation queries
│   │   ├── follow.go              # Follow queries
│   │   ├── oauth_store.go         # OAuth session storage
│   │   ├── user_settings.go       # User settings queries
│   │   └── store.go               # FeedStore adapter for scheduler
│   ├── feed/
│   │   ├── parser.go              # RSS/Atom/RDF/JSON feed parser
│   │   ├── fetcher.go             # Scheduler with dedup + Fetcher
│   │   ├── discover.go            # Feed auto-discovery from URLs
│   │   └── opml.go                # OPML import/export
│   ├── httpclient/
│   │   └── httpclient.go          # Shared HTTP transport, retry logic, User-Agent
│   ├── scraper/
│   │   └── scraper.go             # Full article content scraper
│   ├── metrics/
│   │   └── metrics.go             # Prometheus metrics definitions
│   ├── ml/
│   │   ├── embed.go               # Embedder interface + OpenAI-compatible implementation + vector helpers
│   │   ├── langdetect.go          # Known-language table + detection
│   │   └── llm.go                 # TextModel interface + OpenAI-compatible LLM implementation
│   ├── cluster/
│   │   ├── jaccard.go             # Jaccard similarity computation
│   │   ├── article.go             # Article + feed embedding computation, Vectorize KNN content boost, language detection
│   │   ├── scoring.go             # Feed + people + article recommendation queries (precomputed + on-demand fallback)
│   │   ├── precompute.go          # Precompute all user recommendations (run by cron 4x/day)
│   │   ├── social.go              # Incremental follow-distance computation (1-3 hop, dirty-flag)
│   │   ├── weights.go             # Bandit-style signal weight auto-tuning
│   │   ├── diversity.go           # Post-query domain/category diversity filtering
│   │   └── cron.go                # Background recomputation scheduler
│   ├── feedback/
│   │   └── feedback.go            # Dismiss + impression tracking service
│   ├── server/
│   │   ├── server.go              # HTTP server, chi router setup (/api/* routes)
│   │   ├── api.go                 # JSON helpers (writeJSON), DTOs, null helpers
│   │   ├── api_helpers.go         # URL validation helpers
│   │   ├── auth_handler.go        # OAuth login/callback/register/logout
│   │   ├── feeds_handler.go       # Feed management handlers
│   │   ├── articles_handler.go    # Article reading handlers
│   │   ├── annotations_handler.go # Annotation + library handlers
│   │   ├── dashboard_handler.go   # Dashboard + recommendation handlers
│   │   ├── recs_handler.go        # Recommendation dismiss handlers
│   │   ├── trending_handler.go    # Trending handler
│   │   ├── profile_handler.go     # Public profile handler
│   │   ├── settings_handler.go    # User settings (language preferences, digest/expanded toggle)
│   │   ├── digest_handler.go      # Daily digest handler (LLM summary, mark-read)
│   │   ├── stats_handler.go       # Stats handler (Prometheus metrics → JSON)
│   │   ├── sitemap_handler.go     # Sitemap handler
│   │   ├── index_handler.go       # /api/me + auth-login meta handler
│   │   ├── sync_handlers.go       # Periodic sync + collection-dir backfill
│   │   ├── sanitize.go            # HTML sanitization for article content
│   │   ├── pagination.go          # Pagination helpers
│   │   ├── middleware.go          # Auth, logging, CSRF, CORS middleware
│   │   └── session.go             # Session management
├── web/                           # SvelteKit frontend (SSR), adapter-cloudflare
│   ├── src/
│   │   ├── app.css                # Tailwind v4 entry + design tokens (@theme, [data-theme], @utility)
│   │   ├── app.html               # HTML shell
│   │   ├── app.d.ts               # Locals types (user, csrfToken)
│   │   ├── hooks.server.ts        # handleFetch: route server-load /api fetches to the container
│   │   ├── lib/
│   │   │   ├── api.ts             # Typed endpoint client (endpoints / endpointsFor(fetch))
│   │   │   ├── types.ts           # API response types
│   │   │   ├── format.ts          # Date/HTML/youtube formatting helpers
│   │   │   ├── toast.svelte.ts    # Global toast notifications ($state runes)
│   │   │   └── components/        # Svelte components (ArticleCard, FeedItem, Icon, ...)
│   │   └── routes/               # File-based routes (+page.server.ts load + +page.svelte)
│   │       ├── +layout.ts        # ssr = true (server-rendered app)
│   │       ├── +layout.server.ts # /api/me load (user, CSRF token, feature flags)
│   │       ├── +layout.svelte     # Chrome (nav, footer, dialogs, new-articles banner)
│   │       ├── articles/[id]/     # Article detail
│   │       ├── dashboard/         # Dashboard
│   │       ├── feeds/             # Feed management
│   │       ├── library/           # Liked + annotations
│   │       ├── trending/          # Trending
│   │       ├── profile/[did]/     # Public profile
│   │       ├── stats/             # Metrics
│   │       ├── auth/login/        # Login
│   │       ├── terms/             # Terms (static)
│   │       ├── robots.txt/        # robots (+server.ts)
│   │       └── sitemap.xml/       # Sitemap (+server.ts)
│   ├── static/                    # Favicons, manifest, banner
│   ├── svelte.config.js           # adapter-cloudflare
│   ├── vite.config.ts
│   └── package.json               # bun; svelte, @sveltejs/kit, @tailwindcss/vite
└── docs/
    ├── specs.md                   # Technical specification (this document)
    └── design.md                  # Design system
```

## 10. Auth Flow

Session cookies are HMAC-signed using `GLEAN_SESSION_KEY` (required, must be set to a random string). The server refuses to start without it.

DID resolution uses a configurable PLC directory (`GLEAN_PLC_URL`, defaults to `https://plc.eurosky.network`). The identity directory is initialized once at startup via `InitIdentity()` with a caching layer (250k entries, 24h TTL).

1. User visits `/`, clicks "Sign in with Bluesky" (or any AT Proto PDS)
2. Server redirects to AT Protocol OAuth authorization endpoint
3. User authorizes on their PDS
4. PDS redirects back with authorization code
5. Server exchanges code for access token + refresh token
6. Server resolves DID and creates a session (cookie with HMAC-signed DID)
7. On each request, middleware verifies and decodes the session, loads user from DB (or creates if new)

The server never stores the user's AT Protocol password. It stores the session tokens for XRPC calls to the user's PDS (to read/write their lexicon records).

## 11. Data Flow

### 11.1 User Imports OPML

```
Browser ──POST /api/feeds/opml/upload──► Server
                                      │
                                      ├─► Parse OPML, extract feed URLs
                                      ├─► Fetch each feed, validate + store in `feeds` table
                                      ├─► For each feed, create an `at.glean.subscription` record
                                      │   via XRPC write to user's PDS
                                      ├─► Insert subscriptions in local `subscriptions` table
                                      └─◄ JSON response ({added}); client toasts and reloads
```

### 11.2 Reading the Feed

```
Browser ──GET /articles──► SvelteKit route (server-rendered)
                              │
                              ├─► Load: GET /api/articles (in the Worker via handleFetch)
                              │      └─► Query articles for user's subscriptions
                              └─◄ HTML with the list; browser hydrates
```

### 11.3 Recommendations

```
Cron (every 1h) ──► Cluster Engine
                           │
                           ├─► Compute user similarity
                           ├─► Compute article embeddings (if embedder configured)
                           ├─► Detect article languages (if LLM configured)
                           ├─► Compute follow distances
                           ├─► Compute signal profiles
                           └─► Auto-dismiss stale recommendations

Browser ──GET /dashboard──► Server
                                │
                                ├─► Read precomputed recommendations from DB (instant)
                                │   (fallback to on-demand if missing)
                                ├─► Fetch feed metadata
                                └─◄ JSON; client renders recommendation cards
```

## 12. Key Design Decisions

### 12.1 Why use lexicon records for feed subscriptions?

- **User sovereignty**: Each feed subscription lives as a record on the user's PDS. They can export them, move PDS, or revoke access at any time.
- **Interoperability**: Any AT Protocol app can read the lexicon and integrate with Glean data.
- **No lock-in**: If Glean shuts down, the user's data is intact on their PDS.
- **OPML as transport only**: OPML is a common interchange format used solely for import (onboarding from existing readers) and export (offboarding). The canonical representation is always the lexicon: individual `at.glean.subscription` records, one per feed.

### 12.2 Storage

- Relational recs and the AppView index live in D1 (FTS5 included). Content KNN lives in Vectorize. Article bodies archive to R2. The Go container is the Jetstream consumer and the cron, not the disk.
- D1 is the only storage backend, in production and local dev alike (`wrangler dev` serves the binding locally); the Go test suites run the same statements against in-process sqlite, since D1 is sqlite.

### 12.3 Prometheus Metrics

Glean exposes a `/metrics` endpoint for monitoring. Key metrics:

- `glean_feeds_fetched_total`: Total feed fetch attempts (counter)
- `glean_feeds_fetched_last_timestamp_seconds`: Unix timestamp of last feed fetch (gauge)
- `glean_feed_fetch_duration_seconds`: Histogram of feed fetch latency
- `glean_articles_upserted_total`: Counter of articles stored from feeds
- `glean_jetstream_events_total`: Jetstream events labeled by collection and action
- `glean_jetstream_errors_total`: Jetstream handler errors
- `glean_jetstream_reconnects_total`: Jetstream reconnection count
- `glean_http_requests_total`: HTTP request counts labeled by method, path, and status
- `glean_http_request_duration_seconds`: HTTP request duration labeled by method and path
- `glean_pds_sync_runs_total` / `glean_pds_sync_errors_total`: PDS sync runs and errors
- `glean_cluster_runs_total` / `glean_cluster_duration_seconds`: Recommendation engine runs and timing

### 12.4 Why a Worker in front of one Container?

- Cloudflare Containers are reached only through a Worker. The Worker is the public origin, TLS, and asset host; the container never has a public IP.
- The Worker serves the SvelteKit app at the edge, server-rendered. The Go API cannot live at the edge: Jetstream and feed fetching need a long-lived Linux process.
- One named container instance is required because the Jetstream consumer and SQLite writer are process-global. `getRandom` / multi-instance routing would split the index.
- Workers cannot keep a websocket open for hours, so Jetstream stays inside the container. The Worker cron only prevents idle sleep.

### 12.5 AppView Architecture

Glean operates as an AT Protocol AppView. This means:

- **Read path**: All `at.glean.*` data is consumed from Jetstream, not by polling individual PDS instances. The Jetstream consumer runs as a persistent goroutine, upserting records into D1 as they arrive.
- **Write path**: Users write records to their own PDS (via standard AT Protocol `com.atproto.repo.createRecord` / `deleteRecord`). Glean never stores user data directly; it only indexes what Jetstream delivers.
- **Query path**: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.
- **Trade-off**: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (`at.glean.subscription`) live on the PDS.

### 12.6 Privacy Model

All PDS records are public. There is no notion of private data on the AT Protocol: everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.

### 12.7 Cloudflare constraints this design accepts

- **D1 is the store, not the process.** Jetstream still needs a long-lived Go container, and containers cannot use Worker bindings. The bridge is an outbound handler: the container POSTs statements to `http://glean.proxy/d1`, the platform intercepts that host inside the container's network namespace, and the statements run through the D1 binding on the same machine. No request-count limit (the D1 REST API is capped account-wide at 1,200 requests / 5 min) and no public endpoint.
- **R2 through the same proxy.** Article bodies read and write through `/r2` on the same host via the R2 binding. The container holds no S3 credentials and does no request signing.
- **Vectors are Vectorize**, not sqlite-vec. Recs KNN is `Index.Query` via the cloudflare-go SDK.
- **Singleton process.** `max_instances: 1`. Two containers would each consume Jetstream.
- **Cold start.** Cron pokes `/ready` so sleepAfter does not drop the websocket.

### 12.8 Proxy security model

`/d1` and `/r2` run as outbound handlers on the virtual host `glean.proxy`:
the runtime intercepts requests to that host from inside the container's
network namespace and runs the handler next to the bindings, so only the
container can reach them. `/internal/tick` is reachable only through the
Durable Object binding the Worker cron uses. Bearer tokens still gate every
endpoint as a second layer:

- Each endpoint derives its own token (`db.ProxyToken`, SHA-256 of
  `GLEAN_SESSION_KEY` + purpose), so one leaked token grants only that
  endpoint's scope.
- The tokens reveal nothing about `GLEAN_SESSION_KEY` itself (one-way hash),
  which alone signs session cookies, so a leaked proxy token cannot forge
  sessions.
- Bindings scope the blast radius: one D1 database, one R2 bucket, no
  account-level API. `/r2` exposes no listing.
- Comparison is constant-time on both sides, and the tokens never leave the
  machine or the logs.
- Rotating `GLEAN_SESSION_KEY` rotates every token and invalidates all user
  sessions in one step.

The precondition is a high-entropy `GLEAN_SESSION_KEY` (`openssl rand -hex
32`); a weak key makes every derived token brute-forceable.

## 13. Configuration

All `GLEAN_*` vars are set as Worker vars (`wrangler.jsonc`, which carries the
deployment values) or Worker secrets; the Worker forwards them to the
container. The Go binary owns the defaults below (`main.go`).

| Variable                       | Default                                      | What it does                                                                                                     |
| ------------------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `GLEAN_SESSION_KEY`            | _(required)_                                 | High-entropy secret (`openssl rand -hex 32`); signs session cookies and derives the /d1, /r2, /tick proxy tokens |
| `GLEAN_FRONTEND_URL`           | _(required)_                                 | Public origin; the OAuth callback is `$GLEAN_FRONTEND_URL/api/auth/callback`                                     |
| `GLEAN_OAUTH_CLIENT_ID`        | `https://glean.at/api/oauth/client-metadata` | OAuth client-metadata URL; empty runs the localhost dev flow                                                     |
| `GLEAN_JETSTREAM`              | `wss://jetstream1.eurosky.network`           | Jetstream WebSocket URL                                                                                          |
| `GLEAN_PLC_URL`                | `https://plc.eurosky.network`                | PLC directory URL for DID resolution                                                                             |
| `GLEAN_ARCHIVE_WINDOW`         | `90`                                         | Days article bodies stay in D1 before the offload pass moves them to R2 (0 disables)                             |
| `GLEAN_ARTICLE_RETENTION_DAYS` | `30`                                         | Days article rows live before maintenance deletes them; liked, annotated, and undated articles survive           |
| `GLEAN_COLLECTION_DIR_URL`     | _(empty)_                                    | Collection directory URL; a fresh database bootstraps from it once on boot                                       |
| `GLEAN_BACKFILL_CONCURRENCY`   | `5`                                          | Max concurrent backfill workers                                                                                  |
| `GLEAN_EMBED_BASE_URL`         | _(empty)_                                    | Embeddings API base URL (recommended)                                                                            |
| `GLEAN_EMBED_API_KEY`          | _(empty)_                                    | API key for the embeddings endpoint                                                                              |
| `GLEAN_EMBED_MODEL`            | `text-embedding-3-small`                     | Embedding model name; must output exactly the Vectorize index dimensions (a mismatch fails upserts loudly)       |
| `GLEAN_LLM_BASE_URL`           | _(empty)_                                    | LLM API base URL for language detection and digest summaries                                                     |
| `GLEAN_LLM_API_KEY`            | _(empty)_                                    | API key for the LLM endpoint                                                                                     |
| `GLEAN_LLM_MODEL`              | `gpt-4o-mini`                                | LLM model name                                                                                                   |
| `GLEAN_CLOUDFLARE_ACCOUNT_ID`  | _(secret)_                                   | Account for Vectorize                                                                                            |
| `GLEAN_CLOUDFLARE_API_TOKEN`   | _(secret)_                                   | API token with Vectorize permissions                                                                             |
| `GLEAN_D1_PROXY_URL`           | `http://glean.proxy` (var)                   | Outbound host the container POSTs SQL (`/d1`) and article bodies (`/r2`) to; also enables the body archive       |
| `GLEAN_VECTORIZE_ARTICLES`     | `glean-articles`                             | Vectorize index for article embeddings                                                                           |
| `GLEAN_VECTORIZE_FEEDS`        | `glean-feeds`                                | Vectorize index for feed embeddings                                                                              |
| `GLEAN_ADDR`                   | `:8080`                                      | Container listen address (set by the Worker)                                                                     |

## 14. Future Considerations

- **Email digest**: Periodic email with top articles from subscribed feeds
- **Digest personalization**: Feed the user's liked topics and reading patterns into the digest prompt for more targeted summaries