glean
The social RSS reader. Built on AT Protocol. glean.at
app glean atproto atmosphere rss feed social
Star
Fork
Watch
glean docs specs.md
79 kB
Markdown
at main

Glean - Design Document #

1. Overview #

Glean is a social RSS reader built on the AT Protocol. It operates as an AppView for the at.glean.* lexicon namespace: it indexes records from Jetstream, serves XRPC query endpoints, and provides the web UI at glean.at.

Users store their RSS feed subscriptions as individual lexicon records on their PDS (one record per feed). Glean's AppView consumes Jetstream, indexes those records, fetches the referenced RSS feeds, and serves both the reader UI and public XRPC APIs for the at.glean.* namespace.

The core idea: your RSS subscriptions are a strong signal about your interests. When enough people expose theirs, you can discover both people (who reads the same things) and content (what similar readers follow that you don't).

2. Stack #

Layer Technology
Edge Cloudflare Worker: SvelteKit SSR + routing to the container
Backend Go, running as a single Cloudflare Container
Database Cloudflare D1 (SQL + FTS5, via Worker binding proxy) + Vectorize
Frontend SvelteKit (adapter-cloudflare, SSR) + TailwindCSS v4
Auth AT Protocol OAuth / DID resolution (configurable PLC directory)
AT Protocol role AppView for at.glean.* lexicons
Data source AT Protocol Jetstream → SQLite index

3. AT Protocol Lexicons #

All user data lives on their PDS. The server does not own user data; it indexes and aggregates it.

3.1 at.glean.subscription #

A single RSS feed subscription. One record per feed per user. Created automatically when a user subscribes to a feed. During onboarding, existing subscriptions can be bulk-imported from an OPML file: the OPML is parsed and individual subscription records are created.

{
  "lexicon": 1,
  "id": "at.glean.subscription",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "A single RSS feed subscription.",
      "record": {
        "type": "object",
        "required": ["feedUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "title": { "type": "string" },
          "category": { "type": "string" }
        }
      }
    }
  }
}

OPML Import/Export #

OPML is not part of the lexicon. It is only used as a transport format:

  • Import (onboarding): User uploads an OPML file. Glean parses it, validates each feed URL, creates individual at.glean.subscription records on the user's PDS, and indexes them locally.
  • Export (offboarding): Glean reads the user's at.glean.subscription records from their PDS and generates an OPML file for download. The user's repository remains the canonical source.

3.2 at.glean.annotation #

Reading notes on an article: quote a passage, tag it, rate it, or write a note. A user can have many annotations per article. These are public records on the PDS; users can always share an annotation on Bluesky if they want discussion.

{
  "lexicon": 1,
  "id": "at.glean.annotation",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "Reading note on a specific RSS article.",
      "record": {
        "type": "object",
        "required": ["feedUrl", "articleUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "articleUrl": { "type": "string" },
          "quote": { "type": "string", "maxGraphemes": 5000 },
          "note": { "type": "string", "maxGraphemes": 500 },
          "tags": {
            "type": "array",
            "items": { "type": "string", "maxGraphemes": 50 },
            "maxLength": 10
          },
          "rating": { "type": "integer", "minimum": 1, "maximum": 5 }
        }
      }
    }
  }
}

3.3 at.glean.like #

A user likes an article. The liked feed surfaces popular articles and feeds into discovery. Likes also feed into the recommendation system.

{
  "lexicon": 1,
  "id": "at.glean.like",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "description": "Like an RSS article.",
      "record": {
        "type": "object",
        "required": ["feedUrl", "articleUrl"],
        "properties": {
          "createdAt": { "type": "string", "format": "datetime" },
          "feedUrl": { "type": "string" },
          "articleUrl": { "type": "string" }
        }
      }
    }
  }
}

3.4 at.margin.note (External) #

Glean also indexes records from the at.margin.note lexicon (owned by margin.at). These are displayed in the UI as if they were at.glean.annotation records: margin notes appear alongside glean annotations on article detail pages.

Ingestion (margin.at → Glean) #

Margin notes are indexed from both Jetstream and PDS sync, same as glean records. The mapping from margin note to glean annotation:

Margin note field Annotation field Notes
target.source article_url W3C SpecificResource source URL
body.value note Text content of the annotation
target.selector.exact quote TextQuoteSelector exact text
tags tags Direct mapping
createdAt created_at Direct mapping
(looked up from articles DB) feed_url Resolved by matching target.source against known article URLs

When no matching article exists in the local DB, the annotation is stored with an empty feed_url.

Mirroring (Glean → margin.at) #

When a user creates an annotation through Glean, two records are written to the user's PDS:

  1. at.glean.annotation: the primary record (canonical URI for the annotation)
  2. at.margin.note: mirror for interoperability with margin.at clients

The conversion from glean annotation to margin note uses NewMarginNoteRecord:

Glean annotation field Margin note field Notes
articleUrl target.source Direct mapping
quote target.selector.exact Wrapped in a TextQuoteSelector
note body.value body.format = "text/plain"
tags tags Direct mapping
(constant) motivation Always "commenting"

The margin note mirror is fire-and-forget: if it fails, the glean annotation still succeeds. The Jetstream consumer will pick up the margin note create event, but handleMarginNote skips it via AnnotationExistsByContent (checks author_did + article_url + quote + note) to prevent duplicate annotations.

3.5 app.skyreader.feed.subscription (External) #

Glean also indexes records from the Skyreader lexicon (app.skyreader.feed.subscription). When a user has subscriptions in Skyreader, they are imported as Glean subscriptions during PDS sync and via Jetstream events. This lets users who previously used Skyreader seamlessly transition to Glean without re-subscribing to their feeds.

The mapping from Skyreader subscription to Glean subscription:

Skyreader field Glean field Notes
feedUrl feed_url Direct mapping
title title Direct mapping
siteUrl site_url Stored on the feed record
createdAt added_at Direct mapping
(none) category Empty (Skyreader has no categories)

If a Glean subscription already exists for the same feed_url, the existing one is kept. If the existing subscription has no URI (was created locally without PDS sync), the Skyreader URI/CID is backfilled.

3.6 app.bsky.graph.follow (External) #

Follow relationships are tracked from Bluesky and Tangled follow records. The FollowRecord struct is validated against the lexicon at lexicons/app/bsky/graph/follow.json. The optional via field (a strong ref) is preserved as raw JSON but not used by Glean.

3.7 Lexicon Constants #

All collection NSIDs are defined as constants in lexicon.go and used throughout the codebase:

const (
    CollectionSubscription          = "at.glean.subscription"
    CollectionAnnotation            = "at.glean.annotation"
    CollectionLike                  = "at.glean.like"
    CollectionMarginNote            = "at.margin.note"
    CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
    CollectionBskyFollow            = "app.bsky.graph.follow"
    CollectionTangledFollow         = "sh.tangled.graph.follow"
)

3.8 AppView Query Lexicons #

As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed at.glean.* data without implementing their own indexer.

at.glean.listSubscriptions #

List subscriptions from a repo, with optional filtering.

Input:
  repo: string (DID of the user)
  category?: string
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  subscriptions: [{ uri, cid, value: at.glean.subscription#main, indexedAt }]

at.glean.listFeedLists #

List subscription lists from multiple repos, with optional filtering.

Input:
  actors?: string[] (filter by DIDs)
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  feeds: [{ did, subscriptionCount, subscriptions: [{ feedUrl, title, category }] }]

at.glean.listAnnotations #

List annotations for an article, a feed, or by a user.

Input:
  feedUrl?: string
  articleUrl?: string
  author?: string (DID)
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  annotations: [{ uri, cid, author: { did, handle }, value, indexedAt }]

at.glean.listLikes #

List liked articles, optionally filtered by user or feed.

Input:
  author?: string (DID)
  feedUrl?: string
  limit?: integer (default 50, max 100)
  cursor?: string

Output:
  cursor?: string
  likes: [{ uri, cid, author: { did, handle }, value: at.glean.like#main, indexedAt }]

at.glean.getTrending #

Articles with the most likes, forming the community feed.

Input:
  limit?: integer (default 50, max 100)
  cursor?: string
  since?: string (datetime)

Output:
  cursor?: string
  articles: [{ feedUrl, articleUrl, title, likeCount, annotations: [...] }]

at.glean.getRecommendations #

Get feed recommendations for a user based on clustering.

Input:
  repo: string (DID of the user, query parameter)
  limit?: integer (default 20, max 50)

Output:
  feeds: [{ feedUrl, title, siteUrl, description, subscriberCount, score }]
  people: [{ did, handle, displayName, avatar, jaccard, commonFeeds, isFollowed }]

3.9 AppView Jetstream Consumption #

Glean subscribes to a Jetstream endpoint (GLEAN_JETSTREAM, default wss://jetstream1.eurosky.network):

SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note", "app.skyreader.feed.subscription", "site.standard.document"]

Every collection the handler processes is subscribed network-wide, including events by Glean's own users, which arrive through those same collections. No wantedDids filter is used: the users table holds every follow target (see 7.7), so a DID list would grow into megabytes of query string and a rejected WebSocket handshake. This keeps live sync with Skyreader subscriptions, margin.at notes, and the social graph.

The follow firehose is mainnet-scale, so follow edges are storage-gated to the slice the distance BFS actually traverses: edges with a reader user on either end. A users row exists only when a DID has a reader record (at.glean.*, skyreader.*, or margin.*: subscriptions, likes, and annotations), so a reader's out-edges resolve their 1-hop recommendations and edges into readers resolve 2-hop paths for whoever follows the follower. Every distance query joins follow_distances against subscriptions or likes, so distances to non-readers are never read. Everything else would mirror the network's graph without influencing any output; 3-hop coverage is partial, which its 0.1 signal weight absorbs. Articles for site.standard.publication feeds arrive via the feed scheduler (PDS polling through StandardSiteFetcher, atproto feeds, not RSS); the site.standard.document stream adds real-time create/update/delete for already-known publication feeds. Documents for unknown publications are dropped so the network-wide stream cannot mirror every publication into the article index. (site.standard.publication events are not subscribed: no handler ever consumed them.)

Records created while the consumer is offline are covered by PDS sync. The maintenance pass enforces the storage gates on already-stored rows.

On each event:

  • create: Insert record into local SQLite, update materialized counts
  • delete: Tombstone the record (soft delete to preserve foreign key integrity)
  • update: Replace the record's CID and value

The AppView does not handle writes. Users write records to their own PDS. Glean only reads them from Jetstream.

4. RSS Reader #

Glean is first and foremost an RSS reader. It fetches, parses, and stores articles from RSS, Atom, and JSON feeds so users can read them in a clean interface.

4.1 Feed Fetching #

A background scheduler polls subscribed feeds on a configurable tick. Feeds are fetched at most once per cycle regardless of how many users share them.

4.2 Fetch Schedule #

The scheduler uses a configurable tick interval with in-flight deduplication:

  • Tick interval: A Worker cron heartbeat (*/15 * * * * in wrangler.jsonc) invokes POST /internal/tick; feed fetching runs on every tick
  • Staleness threshold: Feeds not fetched in the last 30 minutes are eligible
  • Subscriber filter: Only feeds with at least one subscription are fetched. Liked or annotated feeds are stored as passive anchors (protected from the purge, never fetched); their articles arrive via on-demand materialization (see 6.4)
  • In-flight dedup: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again
  • Error tracking: error_count increments on failure, resets to 0 on success. Feeds with high error counts are surfaced as "dead feeds" to the user.
SELECT ... FROM feeds
WHERE EXISTS (SELECT 1 FROM subscriptions WHERE feed_url = feeds.feed_url)
  AND (last_fetched_at IS NULL OR last_fetched_at <= :cutoff)
ORDER BY last_fetched_at ASC NULLS FIRST

4.3 Feed Parsing #

Go's encoding/xml for RSS and Atom. A simple encoding/json for JSON Feed.

Each parser returns a normalized Feed and a slice of Article structs:

type Article struct {
    FeedURL   string
    GUID      string
    Title     string
    URL       string
    Author    string
    Content   string
    Summary   string
    Published time.Time
    Updated   time.Time
}

Articles are deduplicated by (feed_url, guid). On upsert, only the article metadata changes; reading state is preserved.

4.4 Article Content #

Glean stores article content locally so the reading experience is fast and consistent:

CREATE TABLE articles (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    feed_url    TEXT NOT NULL,
    guid        TEXT NOT NULL,
    title       TEXT NOT NULL DEFAULT '',
    url         TEXT,
    author      TEXT,
    summary     TEXT,
    content     TEXT,
    full_content TEXT,
    published   DATETIME,
    updated     DATETIME,
    fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    language    TEXT NOT NULL DEFAULT '',
    UNIQUE(feed_url, guid)
);

CREATE INDEX idx_articles_language ON articles(language);

4.5 Read State #

Read/unread state is tracked per user per article:

CREATE TABLE read_state (
    user_did    TEXT NOT NULL,
    article_id  INTEGER NOT NULL,
    is_read     BOOLEAN NOT NULL DEFAULT 0,
    read_at     DATETIME,
    PRIMARY KEY (user_did, article_id)
);

CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;

4.6 Reading Experience #

The /articles page is the main reading view:

  • River of news: Chronological list of all unread articles across all subscriptions
  • Feed filter: Narrow to a single feed or category
  • Mark as read: Individual toggle or "Mark all read" (optionally scoped to one feed)
  • Like: Endorse an article (public, synced to Bluesky PDS)
  • Open original: Title links to the source article
  • Share: Share to Bluesky
  • Keyboard navigation: Single-letter route shortcuts (g dashboard, a articles, f feeds, t trending, l library) via <svelte:window> in the layout; Escape closes dialogs
  • Expanded view: User setting that shows full article content inline on the articles page. Articles are automatically marked as read via IntersectionObserver after being visible for 3 seconds. YouTube videos are embedded inline. A duplicate like button appears at the bottom of each article. Configurable in profile settings.
  • Daily digest: An AI-generated summary of unread articles, shown on the dashboard when enabled in profile settings. The LLM receives the titles and summaries of up to 50 unread articles and produces a grouped overview with linked references. A "Read" button marks all digest articles as read. Digests are cached per user for 24 hours and generated via singleflight to avoid duplicate LLM calls.

4.7 Feed Discovery from Content #

Beyond the clustering system, Glean also discovers new feeds from article content:

  • Auto-discovery: When fetching a feed, parse <link rel="alternate" type="application/rss+xml"> from the feed's site URL to discover related feeds
  • Feedfavicon: Fetch favicon.ico or /apple-touch-icon.png from the feed's site URL for display
  • Dead feed detection: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it.

5. System Architecture #

Glean is a Worker-plus-container deployment. A Cloudflare Worker is the public origin: it server-renders the SvelteKit app and forwards /api/*, /xrpc/*, /metrics, and /ready to one named Cloudflare Container running the Go AppView. The Go process still fills two roles: AppView (indexing at.glean.* records from Jetstream, serving XRPC queries) and RSS reader (fetching and storing feed content).

The container is a singleton (getContainer(env.GLEAN, "glean"), max_instances: 1). The AppView is a shared index plus a Jetstream websocket, so it cannot be load-balanced across interchangeable instances. The process does not self-schedule: a single Worker cron heartbeat (every 15 minutes, see wrangler.jsonc) invokes the authenticated /internal/tick endpoint, which runs whatever is due (fetch every tick, cluster on the :00 tick hourly, sync on the :00 tick every 8 hours), and any request wakes a sleeping container (first request after a quiet period pays a few seconds of cold start). sleepAfter (5m) is the cost dial: the container sleeps when idle, Jetstream dies with the process, and the next wake resumes the stream from the cursor stored in D1.

The container's D1 and R2 traffic goes to the virtual host glean.proxy (GLEAN_D1_PROXY_URL): the platform intercepts requests to that host from inside the container's network namespace and runs them as Worker code next to the bindings, on the same machine. wrangler dev mirrors the arrangement with a sidecar in the same network namespace, so the same URL works locally.

flowchart TD
    B[Browser] -->|"GET pages, __data.json"| W
    B -->|"/api, /xrpc, /metrics, /ready"| W

    subgraph Edge["Cloudflare edge (any colo)"]
        W["Worker: glean, SvelteKit SSR + router"]
        A[(Static assets)]
        P["outbound host glean.proxy: auth'd /d1 + /r2 proxy"]
        D1[("D1: SQL + FTS5")]
        W --> A
        P --> D1
    end

    subgraph Singleton["Singleton placement (one datacenter)"]
        DO["Durable Object 'glean'
(wakes the container)"]
        C["Container: Go AppView
chi API · feed scheduler · jetstream consumer · cluster cron · materializer"]
        DO --> C
    end

    W -->|"backend paths + cron ticks, getContainer(env.GLEAN)"| DO
    W -->|"SSR server loads, handleFetch + cookies"| DO
    C -->|"SQL as JSON over /d1 (bearer token)"| P
    C -->|"article bodies over /r2 (bearer token)"| P
    P --> R2[("R2 archive")]
    C -->|"embeddings + KNN, REST"| V[(Vectorize)]
    C -->|"fetch RSS/Atom/JSON, scrape, sync PDS"| NET[Internet]
    NET -.->|"Jetstream websocket"| C
    C -->|"writes on user action"| PDS[User PDS]

AppView responsibilities:

  • Subscribe to Jetstream network-wide (no wantedDids filter) for at.glean.*, at.margin.note, app.skyreader.feed.subscription, site.standard.document, and follow records; follow edges are storage-gated to graph-relevant pairs
  • Index records into D1
  • Convert at.margin.note records to annotations (displayed alongside glean annotations), skip if duplicate glean annotation exists
  • Mirror glean annotations as at.margin.note records on user PDS for interoperability
  • Import app.skyreader.feed.subscription records as glean subscriptions
  • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
  • Write to user PDS on behalf of the user (when the user acts through the UI)
  • Materialize liked/annotated articles whose row is missing by scraping the article URL

6. Database Schema #

All SQL runs against one database with flat table names: Cloudflare D1, reached through the Worker binding proxy in production and through wrangler dev's local D1 in development. D1 exposes a single main schema, so the exact same statements run everywhere. The schema initializes in one transaction so the D1 driver flushes it as a single proxy batch: cold starts pay one round trip, not one per statement. The Go test suites run the same schema and statements against in-process sqlite (modernc.org/sqlite); D1 is sqlite, so only the transport differs. Content KNN is Cloudflare Vectorize, not sqlite-vec.

Foreign key constraints are not used. Referential integrity is enforced by the application layer.

Transactions over D1 are emulated by the driver: writes buffer on the connection and flush as one D1 batch (a single transaction, chunked at 100 statements to respect D1's time budget). Reads inside a transaction run immediately against committed state; every such read in the codebase either precedes the transaction's writes or touches tables the writes do not affect.

6.1 Users #

Profile data (handle, display name, avatar) is resolved on-the-fly via AT Protocol identity resolution rather than stored locally.

CREATE TABLE users (
    did            TEXT PRIMARY KEY,
    indexed_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    follows_dirty  BOOLEAN NOT NULL DEFAULT 1
);

CREATE TABLE user_settings (
    did            TEXT PRIMARY KEY,
    languages      TEXT,
    expanded_view  BOOLEAN NOT NULL DEFAULT 0,
    digest_enabled BOOLEAN NOT NULL DEFAULT 0,
    updated_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

6.2 Feed Subscriptions #

Indexed from at.glean.subscription records on user PDS.

CREATE TABLE subscriptions (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    user_did    TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    title       TEXT,
    category    TEXT,
    added_at    DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    uri         TEXT,
    cid         TEXT,
    UNIQUE(user_did, feed_url)
);

6.3 Feeds #

Known feeds. Subscribed feeds are fetched; liked or annotated feeds are stored as passive join anchors and never fetched (their articles arrive via materialization, see 6.4).

CREATE TABLE feeds (
    feed_url        TEXT PRIMARY KEY,
    title           TEXT,
    site_url        TEXT,
    description     TEXT,
    feed_type       TEXT CHECK(feed_type IN ('rss', 'atom', 'json', 'atproto')),
    last_fetched_at DATETIME,
    last_error      TEXT,
    subscriber_count INTEGER NOT NULL DEFAULT 0,
    consecutive_empty_fetches INTEGER NOT NULL DEFAULT 0,
    error_count     INTEGER NOT NULL DEFAULT 0,
    favicon_url     TEXT
);

6.4 Articles #

Fetched from RSS and atproto feeds that have at least one subscription network-wide.

CREATE TABLE articles (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    feed_url    TEXT NOT NULL,
    guid        TEXT NOT NULL,
    title       TEXT NOT NULL DEFAULT '',
    url         TEXT,
    author      TEXT,
    summary     TEXT,
    content     TEXT,
    full_content TEXT,
    published   DATETIME,
    updated     DATETIME,
    fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    language    TEXT NOT NULL DEFAULT '',
    UNIQUE(feed_url, guid)
);

CREATE INDEX idx_articles_language ON articles(language);

Article bodies are stored in two tiers. Fresh articles carry content (feed-provided body) and full_content (scraped body) in the row. Once a row is older than GLEAN_ARCHIVE_WINDOW days (default 90; the deployment sets 3), the maintenance cycle uploads each body to R2 (articles/<id>/content, articles/<id>/full) and NULLs the columns. Article reads hydrate NULL bodies from R2 transparently, so lists, article pages, search results, and digests behave the same across the tiers. If a body exists nowhere (a pre-R2 row, or an offload that failed before the upload), the article page re-fetches the body from the source URL once, stores it through the normal path, and serves it; the next read hydrates from R2. A failed re-fetch is retried at most once per hour per article, so dead sources do not add scrape latency to every view. The columns stay NULL-safe: a body that cannot be recovered reads as empty and the UI falls back to the summary plus the fetch-content action.

The archive exists because D1 caps a database at 10 GB and feed bodies are the bulk of the row; R2 is an order of magnitude cheaper per byte and article content should be kept forever, exactly as fetched (annotations quote the body, so a re-fetch of a since-edited page would break highlight alignment). Bodies were never searchable (the FTS index covers title, summary, and author only), so archiving an article changes nothing about search.

D1 size is bounded at every stage. A single fetch inserts at most the 50 newest items per feed (first fetches otherwise dump entire feed histories); the article URL is the row identity (guid) whenever one exists (at:// record URIs excepted), so feeds that regenerate their GUIDs cannot re-insert their window every tick; articles already present are filtered out before any INSERT is sent, and so are items whose publish date is already past the retention window (maintenance purges those within the hour); summaries are stored as a ~1000-character plain-text excerpt, with the full text of description-only feeds moved into content; articles older than GLEAN_ARTICLE_RETENTION_DAYS days (default 30, 0 disables) are deleted unless a like or annotation references them, and undated articles never age out (their publish date is unknown, so age-purging them resurrected every item still in a feed's window as unread once per retention period; the per-feed cap and the unreferenced-feed purge bound them instead); each feed keeps at most 500 article rows; and unreferenced feeds are purged with their articles.

Articles referenced by a like or annotation but absent locally (they already rotated out of the feed's item window, or the feed is not subscribed) are materialized on demand: the article_url is scraped and inserted through the normal upsert path, using the article URL as GUID, the page title (or a humanized URL slug) as title, and a plain-text excerpt as summary. Liked and annotated feeds are passive, stored to anchor the joins and never fetched, and their articles survive the retention purge as long as the like or annotation exists. If the like is later deleted, the feed and articles become unreferenced and the hourly purge removes them.

6.5 Read State #

CREATE TABLE read_state (
    user_did    TEXT NOT NULL,
    article_id  INTEGER NOT NULL,
    is_read     BOOLEAN NOT NULL DEFAULT 0,
    read_at     DATETIME,
    PRIMARY KEY (user_did, article_id)
);

CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;

6.6 Annotations, Likes #

Local mirror of AT Protocol lexicon records for fast querying.

CREATE TABLE annotations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    uri         TEXT NOT NULL UNIQUE,
    author_did  TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    article_url TEXT NOT NULL,
    quote       TEXT,
    note        TEXT,
    tags        TEXT,
    rating      INTEGER,
    created_at  DATETIME NOT NULL,
    cid         TEXT
);

CREATE TABLE likes (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    uri         TEXT NOT NULL UNIQUE,
    author_did  TEXT NOT NULL,
    feed_url    TEXT NOT NULL,
    article_url TEXT NOT NULL,
    created_at  DATETIME NOT NULL,
    cid         TEXT,
    UNIQUE(author_did, feed_url, article_url)
);

6.7 Cluster Precomputation #

Stores precomputed similarity data to avoid recalculating on every request.

CREATE TABLE user_similarity (
    user_a     TEXT NOT NULL,
    user_b     TEXT NOT NULL,
    jaccard    REAL NOT NULL,
    common_feeds INTEGER NOT NULL,
    common_likes INTEGER NOT NULL DEFAULT 0,
    common_tags INTEGER NOT NULL DEFAULT 0,
    computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_a, user_b),
    CHECK(user_a < user_b)
);

6.8 Follows #

Tracks follow relationships between users (from app.bsky.graph.follow and sh.tangled.graph.follow records).

CREATE TABLE follows (
    user_did    TEXT NOT NULL,
    target_did  TEXT NOT NULL,
    uri         TEXT,
    cid         TEXT,
    followed_at DATETIME,
    PRIMARY KEY (user_did, target_did)
);

CREATE INDEX idx_follows_target ON follows(target_did);
CREATE INDEX idx_follows_uri ON follows(uri);

6.9 OAuth Storage #

Login attempts store their handshake state until the callback consumes it; sessions store the OAuth tokens per login. Maintenance deletes abandoned requests after 7 days and sessions untouched for 90 days, so both tables stay bounded.

CREATE TABLE oauth_auth_requests (
    state      TEXT PRIMARY KEY,
    data       TEXT NOT NULL,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE oauth_sessions (
    account_did TEXT NOT NULL,
    session_id  TEXT NOT NULL,
    data        TEXT NOT NULL,
    updated_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (account_did, session_id)
);

7. Recommendations #

Glean uses a multi-signal recommendation system that combines subscription overlap, like patterns, social graph distance, and user behavior feedback.

7.1 Signals #

Signal Source Weight (default) Description
Subscription subscriptions 1.0 Jaccard over subscriber sets between similar users
Like likes 0.5 Time-decayed like co-occurrence (30-day half-life)
Tag annotations.tags 0.3 Jaccard over annotation tag sets
Social follow_distances 0.7 Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1
Popularity feeds.subscriber_count 0.2 log(1 + subscribers) / log(1 + max)
Category subscriptions.category 0.4 Boost feeds matching user's existing categories
Content Vectorize glean-articles 0.4 Cosine similarity via embedding KNN (requires embedder)

7.2 Feed Co-occurrence (Jaccard Similarity) #

For any two feeds, the similarity is the Jaccard index of their subscriber sets:

J(A, B) = |subscribers(A) ∩ subscribers(B)| / |subscribers(A) ∪ subscribers(B)|

Feed description similarity is also computed via embedding cosine similarity (requires embedder) and added as a boost.

7.3 User Similarity #

For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:

J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost

Like overlap uses exponential time decay: EXP(-0.023 * age_days) (30-day half-life).

7.4 Precomputed + On-Demand Scoring #

Recommendations are precomputed for all active users on every cluster cron cycle (hourly; see 7.10) and stored in the precomputed_recommendations table

Feed recommendation score (computed in SQL):

score = sub_signal * w_sub
      + like_signal * w_like
      + social_signal * w_social
      + pop_signal * w_pop
      + category_signal * w_category

Where:

  • sub_signal = SUM(jaccard(target, U)) for similar users U subscribed to feed
  • like_signal = SUM(jaccard(target, U) * time_decay) for likes in that feed by similar users
  • social_signal = SUM(distance_weight) from follow_distances
  • pop_signal = log(1 + subscriber_count) / log(1 + max_subscribers)
  • category_signal = 1 if feed description matches user's top categories

Article recommendation score:

score = like_signal * w_like
      + social_signal * w_social
      + content_signal * w_content
      + recency_signal * 0.2

Content signal uses embedding vectors: the user's liked article embeddings are averaged into a single interest vector, then a KNN query against the glean-articles Vectorize index finds semantically similar articles. This requires an embedder to be configured; without it, the content signal is 0.

Language filtering: Users can set preferred languages on their profile (stored in the user_settings table as a JSON array of ISO 639-1 codes). When set, article recommendations are filtered to only include articles whose language column matches one of the selected codes or whose language is empty (not yet classified). This ensures users with language preferences still see all recommendations when the LLM hasn't run or hasn't classified certain articles yet. If no languages are set (empty/nil), all articles are shown regardless of language.

7.5 User Feedback (Dismiss) #

Users can dismiss recommendations they don't want to see again:

  • POST /feeds/dismiss: dismiss a feed recommendation
  • POST /articles/dismiss: dismiss an article recommendation
  • Dismissals are stored locally in dismissed_recommendations (not on PDS)
  • Dismissed items are excluded from all future recommendation queries
  • Auto-dismiss: items shown ≥5 times over >5 days without action are auto-dismissed

Recommended articles use the same card template as regular articles, with an additional "Hide" button that dismisses the recommendation. Dismissing only removes that specific article from recommendations; it does not affect the user's likes or signal weights, and similar articles may still be recommended.

7.6 Auto-Tuned Signal Weights #

Each user has a row in user_signal_weights with per-signal weights. When a user acts on a recommendation (subscribes, likes), the dominant signal that produced that recommendation is rewarded:

new_weight = MAX(0.1, MIN(3.0, old_weight * (1 + learning_rate * delta)))
  • learning_rate = 0.1, delta = +1 for reward, -1 for penalty
  • Only activates after minActionsTune = 5 positive actions
  • Defaults are used when no row exists for a user

7.7 Social Graph #

Follow distances (1-hop through 3-hop) are computed incrementally, for reader users only: a users row exists only when a DID has a reader record, and every query reads user_a = <requesting user> joined against subscriptions or likes. A follows_dirty column on users tracks whose follow graph changed since the last cron run. Only dirty users are reprocessed: their existing rows in follow_distances are deleted and recomputed via BFS, then the dirty flag is cleared. The follows table stores only the out-edges the BFS traverses: edges with a reader on either end (a reader's out-edges for 1-hop, edges into readers for 2-hop), so 3-hop coverage is partial. Maintenance purges distance rows whose user_a no longer has a users row.

  • 1-hop: direct follows (weight 1.0)
  • 2-hop: friends-of-friends (weight 0.3)
  • 3-hop: third-degree connections (weight 0.1)

7.8 Diversity & Freshness #

After scoring, diversity filtering is applied in Go (not SQL):

  • Domain diversity: max 2 feeds from the same domain in results
  • Category diversity: max 3 feeds from the same category in results
  • This prevents recommendation clustering on a single source

7.9 Cold Start #

New users with <5 subscriptions get a fallback strategy:

  1. Feeds from 1-hop followed users (70% weight)
  2. Globally popular feeds by subscriber count (30% weight)

7.10 Clustering Engine (Cron) #

The heartbeat's :00 tick (hourly) runs the cluster pass. Maintenance runs first, so the computation steps below run on the rows that survive instead of embedding and scoring articles that are about to expire:

  1. DB maintenance: Prune recommendation_impressions older than 14 days; prune OAuth login attempts older than 7 days and sessions untouched for 90 days; purge follow edges that fail the storage gate (neither endpoint is a reader user, answered from an in-process known-users cache the jetstream gate also uses); purge follow_distances rows whose user_a has no users row; delete articles older than GLEAN_ARTICLE_RETENTION_DAYS days unless a like or annotation references them (undated articles are exempt; the per-feed cap and the unreferenced-feed purge remove them); trim every feed to its 500 newest articles; delete feeds with no subscription, like, or annotation along with their articles; delete read_state rows pointing at removed articles; then run PRAGMA incremental_vacuum on the local file (D1 manages its own storage). All deletes run in rowid-keyed chunks so maintenance never holds a long write transaction.
  2. Offload article bodies: Articles older than GLEAN_ARCHIVE_WINDOW days (default 90; the deployment sets 3) get content/full_content uploaded to R2 and the columns NULLed (see 6.4). Uploads precede the UPDATE so an interrupted pass is idempotent.
  3. Compute feed embeddings: Embed new feed descriptions via embedding API into feed_embeddings table (skipped if no embedder configured)
  4. Compute user similarity: Diff-update user_similarity (subscription Jaccard + time-decayed likes + tags + follow boost; only rows whose value actually changed are written)
  5. Compute article embeddings: Embed new articles (title + summary + content, excluding full_content to stay within embedding model token limits) via embedding API into the glean-articles Vectorize index (skipped if no embedder configured)
  6. Detect article languages: Batch-classify article languages via LLM, updating the language column (skipped if no LLM configured)
  7. Compute follow distances: Incremental BFS for dirty users (1-hop through 3-hop from follows table), committed one user per transaction so a long run never holds the recs writer lock for the whole job, and a failure keeps completed users; each recompute writes only new, changed, or removed pairs
  8. Compute signal profiles: Per-user category/tag/like summaries (diff-updated the same way)
  9. Auto-dismiss stale: Dismiss items shown >=5 times over >5 days without action
  10. Precompute recommendations: For each active user, compute feed, article, and people recommendations and store as JSON in the precomputed_recommendations table. This ensures instant load times for all recommendation sections.
  11. Log purge stats: The cron logs how many follows, articles (unreferenced, expired, overage), feeds, and read_state rows maintenance removed, plus how many article bodies were offloaded

Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.

7.11 User Interaction Tables #

Impressions feed the auto-dismiss pass and the per-user signal-weight tuning. Maintenance deletes impressions older than 14 days (the auto-dismiss window is 5 days).

CREATE TABLE dismissed_recommendations (
    user_did     TEXT NOT NULL,
    target_type  TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
    target_id    TEXT NOT NULL,
    reason       TEXT,
    dismissed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_did, target_type, target_id)
);

CREATE TABLE recommendation_impressions (
    user_did       TEXT NOT NULL,
    target_type    TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
    target_id      TEXT NOT NULL,
    first_shown_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    last_shown_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    shown_count    INTEGER NOT NULL DEFAULT 1,
    acted          BOOLEAN NOT NULL DEFAULT 0,
    PRIMARY KEY (user_did, target_type, target_id)
);

7.12 Computed Recommendation Tables #

Written exclusively by the cron. Read during recommendation requests (precomputed results served first, on-demand fallback if missing). The similarity tables are defined in 6.7.

CREATE TABLE follow_distances (
    user_a   TEXT NOT NULL,
    user_b   TEXT NOT NULL,
    distance INTEGER NOT NULL CHECK(distance IN (1, 2, 3)),
    PRIMARY KEY (user_a, user_b)
);

CREATE TABLE user_signal_weights (
    user_did   TEXT PRIMARY KEY,
    w_sub      REAL NOT NULL DEFAULT 1.0,
    w_like     REAL NOT NULL DEFAULT 0.5,
    w_tag      REAL NOT NULL DEFAULT 0.3,
    w_social   REAL NOT NULL DEFAULT 0.7,
    w_pop      REAL NOT NULL DEFAULT 0.2,
    w_category REAL NOT NULL DEFAULT 0.4,
    w_content  REAL NOT NULL DEFAULT 0.4,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE user_signal_profiles (
    user_did       TEXT PRIMARY KEY,
    total_likes     INTEGER NOT NULL DEFAULT 0,
    total_tags      INTEGER NOT NULL DEFAULT 0,
    top_categories  TEXT,
    updated_at      DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE precomputed_recommendations (
    user_did    TEXT NOT NULL,
    rec_type    TEXT NOT NULL CHECK(rec_type IN ('feed', 'article', 'person')),
    data        TEXT NOT NULL,
    computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_did, rec_type)
);

When GLEAN_EMBED_BASE_URL is configured, article text and feed descriptions are embedded and upserted into Vectorize (glean-articles, glean-feeds). SQL keeps article_embedding_ids and feed_embedding_meta so cron knows what is already indexed. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph.

The embedding model must output exactly the Vectorize index dimensions (1536 in the setup commands): nothing reshapes or truncates vectors, so a mismatched model fails the upsert loudly in the cron logs.

The embedder uses the official github.com/openai/openai-go SDK with option.WithBaseURL(), so any OpenAI-compatible /v1/embeddings endpoint works (OpenAI, Gemini, Ollama, local inference servers).

7.14 LLM Client (optional) #

When GLEAN_LLM_BASE_URL is configured, an LLM client is available for text classification and summarization tasks. It uses the same github.com/openai/openai-go SDK pointed at any OpenAI-compatible /v1/chat/completions endpoint.

Language detection: The cron job calls DetectLanguages in batches of up to 100 articles per request. For each article, the title and summary (truncated to 500 characters) are sent with a prompt asking for ISO 639-1 codes. Results are written to articles.language. Articles that are already classified (non-empty language) are skipped. An empty response from the LLM defaults to 'unknown' rather than a specific language, ensuring unclassifiable articles aren't miscategorized.

Without an LLM, all articles remain at the default empty language value and language-based filtering is unavailable. The daily digest feature also requires an LLM to generate summaries; without it, the digest is unavailable.

SQL tracks which vectors exist:

CREATE TABLE feed_embedding_meta (
    feed_url TEXT PRIMARY KEY,
    source_text TEXT NOT NULL DEFAULT ''
);

CREATE TABLE article_embedding_ids (
    article_id INTEGER PRIMARY KEY
);

During cron, ComputeArticleEmbeddings embeds new articles in batches using title + summary + content and upserts them to Vectorize. ComputeFeedEmbeddings embeds feed descriptions and re-embeds when source text changes. On-demand article recs average liked-article vectors and query Vectorize. Cold-start feed recs do the same with subscribed feeds.

8. HTTP API #

The Go server exposes a JSON API under /api/*. The SvelteKit frontend (see web/) owns all HTML page routes and consumes these endpoints via web/src/lib/api.ts. The Worker forwards browser /api/* to the container; SvelteKit server loads reach the container through handleFetch during server-side rendering (cookies forwarded), and subsequent client-side navigations fetch __data.json from the browser. All state-changing requests require a double-submit CSRF token (X-CSRF-Token header or csrf_token form field) matching the glean_csrf cookie.

8.1 JSON endpoints #

All endpoints below return JSON. Those marked 🔒 require authentication (session cookie). Form fields are sent application/x-www-form-urlencoded; file uploads are multipart/form-data.

Route Method Auth Description
/api/me GET Current user, CSRF token, feature flags (has_llm, client_id)
/api/dashboard GET 🔒 Dashboard: article recs, unread articles, trending, digest flag
/api/feeds GET 🔒 Subscriptions (paginated), categories, dead feeds
/api/feeds/add POST 🔒 Add a feed URL (fields: feed_url, category)
/api/feeds/edit POST 🔒 Edit a subscription category
/api/feeds/opml/upload POST 🔒 Bulk-import subscriptions from OPML (field: opml file)
/api/feeds/opml/download GET 🔒 Export subscriptions as OPML
/api/feeds/refresh POST 🔒 Refresh all subscribed feeds
/api/feeds/retry POST 🔒 Retry a failed feed
/api/feeds/list GET 🔒 Flat subscription list (optional ?category=)
/api/feeds/clear POST 🔒 Remove all subscriptions
/api/articles GET 🔒 Read articles (paginated; ?feed=, ?status=, ?q=, ?sort=, ?category=)
/api/articles/new-count GET 🔒 Count of articles newer than ?since= (unix seconds); banner polling
/api/articles/{id} GET 🔒 Article detail + annotations (marks read as a side effect)
/api/articles/{id}/read POST 🔒 Mark article read
/api/articles/{id}/unread POST 🔒 Mark article unread
/api/articles/{id}/like POST 🔒 Toggle like (writes/deletes on PDS)
/api/articles/{id}/fetch-content POST 🔒 Scrape full article content from the source URL
/api/articles/mark-all-read POST 🔒 Mark all (or ?feed=-scoped) articles read
/api/trending GET Trending articles (?scope=for-me requires auth)
/api/profile/{did} GET 🔒 Public profile: feeds, annotations, languages (resolves handles)
/api/library GET 🔒 Liked articles + annotations (paginated)
/api/library/create POST 🔒 Create annotation (writes at.glean.annotation + at.margin.note mirror)
/api/library/{id}/delete POST 🔒 Delete annotation + mirror
/api/recs/articles GET 🔒 Article recommendations
/api/recs/feeds GET 🔒 Feed recommendations
/api/recs/people GET 🔒 People recommendations (followed + discover)
/api/recs/dismiss-feed POST 🔒 Dismiss a feed recommendation
/api/recs/dismiss-article POST 🔒 Dismiss an article recommendation
/api/recs/dismiss-person POST 🔒 Dismiss a person recommendation
/api/settings/languages/{code} POST 🔒 Toggle a preferred recommendation language
/api/settings/expanded-view POST 🔒 Toggle expanded article view
/api/settings/digest-enabled POST 🔒 Toggle daily digest
/api/digest GET 🔒 Daily digest (LLM summary of unread articles); 204 when none/unavailable
/api/digest/mark-read POST 🔒 Mark digest articles read (field: repeated ids)
/api/auth/login GET Whether OAuth is enabled (oauth_enabled)
/api/auth/register GET Start registration via Eurosky ({redirect})
/api/auth/actors GET Handle typeahead (?q=) → {actors}
/api/auth/start POST 🔒 Start OAuth flow (field: handle) → {redirect}
/api/auth/callback GET OAuth callback (browser redirect, not JSON)
/api/auth/logout POST 🔒 End session → {redirect}
/api/oauth/client-metadata GET OAuth client metadata document
/api/sitemap GET Sitemap entries (JSON)
/api/stats GET Prometheus metrics parsed into {metrics}
/metrics GET Raw Prometheus exposition
/ready GET Container health check (204 if SQLite is reachable)

XRPC query endpoints (public, no /api prefix):

Route Description
/xrpc/at.glean.listSubscriptions List a user's subscriptions
/xrpc/at.glean.listAnnotations List annotations
/xrpc/at.glean.listLikes List likes
/xrpc/at.glean.getTrending Trending articles
/xrpc/at.glean.getRecommendations Recommendations
/xrpc/at.glean.listFeedLists Feed lists

8.2 Frontend routes #

All HTML page routes live in the SvelteKit app (web/src/routes/), not on the Go server. The app server-renders (ssr = true in the root +layout.ts): each route's +page.server.ts load runs in the Worker during SSR, calling the JSON API via endpointsFor(event.fetch) (routed to the container by handleFetch), and the browser hydrates the result; client-side navigations fetch __data.json. The layout (+layout.svelte) renders the shared chrome and mounts the new-articles banner.

Path Load source
/ +page (landing)
/dashboard /api/dashboard
/articles /api/articles
/articles/[id] /api/articles/{id}
/feeds /api/feeds
/library /api/library
/trending /api/trending
/profile/[did] /api/profile/{did}
/stats /api/stats
/terms (static content)
/auth/login /api/auth/login
/robots.txt (static, +server.ts)
/sitemap.xml /api/sitemap

9. Project Structure #

glean/
├── main.go                        # Entry point, wire everything
├── go.mod / go.sum
├── wrangler.jsonc                 # Worker + Container + cron + assets
├── worker/index.ts                # Route /api /xrpc to the named Go container
├── Dockerfile                     # Go API, CGO-free
├── Makefile                       # Targets: dev, deploy, test, ...
├── lexicons/
│   └── at/
│       ├── glean/                  # Glean lexicon JSON schemas (subscription, annotation, like)
│       └── margin/                 # External: at.margin.note W3C Web Annotation schema
│   └── app/bsky/graph/             # External: app.bsky.graph.follow schema
├── internal/
│   ├── atproto/
│   │   ├── auth.go                # DID resolution, OAuth flow
│   │   ├── client.go              # XRPC client (write to user PDS)
│   │   ├── collectiondir.go       # Collection directory backfill (startup)
│   │   ├── jetstream.go           # Subscribe to Jetstream via official client
│   │   ├── stream_handler.go      # Stream event → DB handler
│   │   ├── lexicon.go             # Lexicon record types (at.glean.*, maintained by hand)
│   │   ├── lexicon_external.go    # External lexicon record types (FollowRecord, MarginNoteRecord, SkyreaderSubscriptionRecord)
│   │   ├── lexicon_test.go        # Test: Go structs match lexicon JSON schemas
│   │   ├── sync.go                # PDS record reconciliation
│   │   └── xrpc.go                # XRPC query handlers (AppView endpoints)
│   ├── blob/
│   │   └── blob.go                # R2 client via the outbound /r2 proxy
│   ├── db/
│   │   ├── db.go                  # Store over D1 via the worker proxy; flat table names
│   │   ├── sqlite.go             # In-process sqlite backend for the test suites (D1 is sqlite)
│   │   ├── d1.go                  # D1-over-HTTP driver (DSN from Cloudflare creds)
│   │   ├── migrations.go          # Schema migrations
│   │   ├── blobs.go               # Article-body R2 offload + hydration
│   │   ├── cursor_store.go        # Jetstream cursor persistence
│   │   ├── user.go                # User queries
│   │   ├── feed.go                # Feed + subscription queries
│   │   ├── article.go             # Article queries
│   │   ├── social.go              # Like, annotation queries
│   │   ├── follow.go              # Follow queries
│   │   ├── oauth_store.go         # OAuth session storage
│   │   ├── user_settings.go       # User settings queries
│   │   └── store.go               # FeedStore adapter for scheduler
│   ├── feed/
│   │   ├── parser.go              # RSS/Atom/RDF/JSON feed parser
│   │   ├── fetcher.go             # Scheduler with dedup + Fetcher
│   │   ├── discover.go            # Feed auto-discovery from URLs
│   │   └── opml.go                # OPML import/export
│   ├── httpclient/
│   │   └── httpclient.go          # Shared HTTP transport, retry logic, User-Agent
│   ├── scraper/
│   │   └── scraper.go             # Full article content scraper
│   ├── metrics/
│   │   └── metrics.go             # Prometheus metrics definitions
│   ├── ml/
│   │   ├── embed.go               # Embedder interface + OpenAI-compatible implementation + vector helpers
│   │   ├── langdetect.go          # Known-language table + detection
│   │   └── llm.go                 # TextModel interface + OpenAI-compatible LLM implementation
│   ├── cluster/
│   │   ├── jaccard.go             # Jaccard similarity computation
│   │   ├── article.go             # Article + feed embedding computation, Vectorize KNN content boost, language detection
│   │   ├── scoring.go             # Feed + people + article recommendation queries (precomputed + on-demand fallback)
│   │   ├── precompute.go          # Precompute all user recommendations (run by cron 4x/day)
│   │   ├── social.go              # Incremental follow-distance computation (1-3 hop, dirty-flag)
│   │   ├── weights.go             # Bandit-style signal weight auto-tuning
│   │   ├── diversity.go           # Post-query domain/category diversity filtering
│   │   └── cron.go                # Background recomputation scheduler
│   ├── feedback/
│   │   └── feedback.go            # Dismiss + impression tracking service
│   ├── server/
│   │   ├── server.go              # HTTP server, chi router setup (/api/* routes)
│   │   ├── api.go                 # JSON helpers (writeJSON), DTOs, null helpers
│   │   ├── api_helpers.go         # URL validation helpers
│   │   ├── auth_handler.go        # OAuth login/callback/register/logout
│   │   ├── feeds_handler.go       # Feed management handlers
│   │   ├── articles_handler.go    # Article reading handlers
│   │   ├── annotations_handler.go # Annotation + library handlers
│   │   ├── dashboard_handler.go   # Dashboard + recommendation handlers
│   │   ├── recs_handler.go        # Recommendation dismiss handlers
│   │   ├── trending_handler.go    # Trending handler
│   │   ├── profile_handler.go     # Public profile handler
│   │   ├── settings_handler.go    # User settings (language preferences, digest/expanded toggle)
│   │   ├── digest_handler.go      # Daily digest handler (LLM summary, mark-read)
│   │   ├── stats_handler.go       # Stats handler (Prometheus metrics → JSON)
│   │   ├── sitemap_handler.go     # Sitemap handler
│   │   ├── index_handler.go       # /api/me + auth-login meta handler
│   │   ├── sync_handlers.go       # Periodic sync + collection-dir backfill
│   │   ├── sanitize.go            # HTML sanitization for article content
│   │   ├── pagination.go          # Pagination helpers
│   │   ├── middleware.go          # Auth, logging, CSRF, CORS middleware
│   │   └── session.go             # Session management
├── web/                           # SvelteKit frontend (SSR), adapter-cloudflare
│   ├── src/
│   │   ├── app.css                # Tailwind v4 entry + design tokens (@theme, [data-theme], @utility)
│   │   ├── app.html               # HTML shell
│   │   ├── app.d.ts               # Locals types (user, csrfToken)
│   │   ├── hooks.server.ts        # handleFetch: route server-load /api fetches to the container
│   │   ├── lib/
│   │   │   ├── api.ts             # Typed endpoint client (endpoints / endpointsFor(fetch))
│   │   │   ├── types.ts           # API response types
│   │   │   ├── format.ts          # Date/HTML/youtube formatting helpers
│   │   │   ├── toast.svelte.ts    # Global toast notifications ($state runes)
│   │   │   └── components/        # Svelte components (ArticleCard, FeedItem, Icon, ...)
│   │   └── routes/               # File-based routes (+page.server.ts load + +page.svelte)
│   │       ├── +layout.ts        # ssr = true (server-rendered app)
│   │       ├── +layout.server.ts # /api/me load (user, CSRF token, feature flags)
│   │       ├── +layout.svelte     # Chrome (nav, footer, dialogs, new-articles banner)
│   │       ├── articles/[id]/     # Article detail
│   │       ├── dashboard/         # Dashboard
│   │       ├── feeds/             # Feed management
│   │       ├── library/           # Liked + annotations
│   │       ├── trending/          # Trending
│   │       ├── profile/[did]/     # Public profile
│   │       ├── stats/             # Metrics
│   │       ├── auth/login/        # Login
│   │       ├── terms/             # Terms (static)
│   │       ├── robots.txt/        # robots (+server.ts)
│   │       └── sitemap.xml/       # Sitemap (+server.ts)
│   ├── static/                    # Favicons, manifest, banner
│   ├── svelte.config.js           # adapter-cloudflare
│   ├── vite.config.ts
│   └── package.json               # bun; svelte, @sveltejs/kit, @tailwindcss/vite
└── docs/
    ├── specs.md                   # Technical specification (this document)
    └── design.md                  # Design system

10. Auth Flow #

Session cookies are HMAC-signed using GLEAN_SESSION_KEY (required, must be set to a random string). The server refuses to start without it.

DID resolution uses a configurable PLC directory (GLEAN_PLC_URL, defaults to https://plc.eurosky.network). The identity directory is initialized once at startup via InitIdentity() with a caching layer (250k entries, 24h TTL).

  1. User visits /, clicks "Sign in with Bluesky" (or any AT Proto PDS)
  2. Server redirects to AT Protocol OAuth authorization endpoint
  3. User authorizes on their PDS
  4. PDS redirects back with authorization code
  5. Server exchanges code for access token + refresh token
  6. Server resolves DID and creates a session (cookie with HMAC-signed DID)
  7. On each request, middleware verifies and decodes the session, loads user from DB (or creates if new)

The server never stores the user's AT Protocol password. It stores the session tokens for XRPC calls to the user's PDS (to read/write their lexicon records).

11. Data Flow #

11.1 User Imports OPML #

Browser ──POST /api/feeds/opml/upload──► Server
                                      │
                                      ├─► Parse OPML, extract feed URLs
                                      ├─► Fetch each feed, validate + store in `feeds` table
                                      ├─► For each feed, create an `at.glean.subscription` record
                                      │   via XRPC write to user's PDS
                                      ├─► Insert subscriptions in local `subscriptions` table
                                      └─◄ JSON response ({added}); client toasts and reloads

11.2 Reading the Feed #

Browser ──GET /articles──► SvelteKit route (server-rendered)
                              │
                              ├─► Load: GET /api/articles (in the Worker via handleFetch)
                              │      └─► Query articles for user's subscriptions
                              └─◄ HTML with the list; browser hydrates

11.3 Recommendations #

Cron (every 1h) ──► Cluster Engine
                           │
                           ├─► Compute user similarity
                           ├─► Compute article embeddings (if embedder configured)
                           ├─► Detect article languages (if LLM configured)
                           ├─► Compute follow distances
                           ├─► Compute signal profiles
                           └─► Auto-dismiss stale recommendations

Browser ──GET /dashboard──► Server
                                │
                                ├─► Read precomputed recommendations from DB (instant)
                                │   (fallback to on-demand if missing)
                                ├─► Fetch feed metadata
                                └─◄ JSON; client renders recommendation cards

12. Key Design Decisions #

12.1 Why use lexicon records for feed subscriptions? #

  • User sovereignty: Each feed subscription lives as a record on the user's PDS. They can export them, move PDS, or revoke access at any time.
  • Interoperability: Any AT Protocol app can read the lexicon and integrate with Glean data.
  • No lock-in: If Glean shuts down, the user's data is intact on their PDS.
  • OPML as transport only: OPML is a common interchange format used solely for import (onboarding from existing readers) and export (offboarding). The canonical representation is always the lexicon: individual at.glean.subscription records, one per feed.

12.2 Storage #

  • Relational recs and the AppView index live in D1 (FTS5 included). Content KNN lives in Vectorize. Article bodies archive to R2. The Go container is the Jetstream consumer and the cron, not the disk.
  • D1 is the only storage backend, in production and local dev alike (wrangler dev serves the binding locally); the Go test suites run the same statements against in-process sqlite, since D1 is sqlite.

12.3 Prometheus Metrics #

Glean exposes a /metrics endpoint for monitoring. Key metrics:

  • glean_feeds_fetched_total: Total feed fetch attempts (counter)
  • glean_feeds_fetched_last_timestamp_seconds: Unix timestamp of last feed fetch (gauge)
  • glean_feed_fetch_duration_seconds: Histogram of feed fetch latency
  • glean_articles_upserted_total: Counter of articles stored from feeds
  • glean_jetstream_events_total: Jetstream events labeled by collection and action
  • glean_jetstream_errors_total: Jetstream handler errors
  • glean_jetstream_reconnects_total: Jetstream reconnection count
  • glean_http_requests_total: HTTP request counts labeled by method, path, and status
  • glean_http_request_duration_seconds: HTTP request duration labeled by method and path
  • glean_pds_sync_runs_total / glean_pds_sync_errors_total: PDS sync runs and errors
  • glean_cluster_runs_total / glean_cluster_duration_seconds: Recommendation engine runs and timing

12.4 Why a Worker in front of one Container? #

  • Cloudflare Containers are reached only through a Worker. The Worker is the public origin, TLS, and asset host; the container never has a public IP.
  • The Worker serves the SvelteKit app at the edge, server-rendered. The Go API cannot live at the edge: Jetstream and feed fetching need a long-lived Linux process.
  • One named container instance is required because the Jetstream consumer and SQLite writer are process-global. getRandom / multi-instance routing would split the index.
  • Workers cannot keep a websocket open for hours, so Jetstream stays inside the container. The Worker cron only prevents idle sleep.

12.5 AppView Architecture #

Glean operates as an AT Protocol AppView. This means:

  • Read path: All at.glean.* data is consumed from Jetstream, not by polling individual PDS instances. The Jetstream consumer runs as a persistent goroutine, upserting records into D1 as they arrive.
  • Write path: Users write records to their own PDS (via standard AT Protocol com.atproto.repo.createRecord / deleteRecord). Glean never stores user data directly; it only indexes what Jetstream delivers.
  • Query path: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.
  • Trade-off: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (at.glean.subscription) live on the PDS.

12.6 Privacy Model #

All PDS records are public. There is no notion of private data on the AT Protocol: everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.

12.7 Cloudflare constraints this design accepts #

  • D1 is the store, not the process. Jetstream still needs a long-lived Go container, and containers cannot use Worker bindings. The bridge is an outbound handler: the container POSTs statements to http://glean.proxy/d1, the platform intercepts that host inside the container's network namespace, and the statements run through the D1 binding on the same machine. No request-count limit (the D1 REST API is capped account-wide at 1,200 requests / 5 min) and no public endpoint.
  • R2 through the same proxy. Article bodies read and write through /r2 on the same host via the R2 binding. The container holds no S3 credentials and does no request signing.
  • Vectors are Vectorize, not sqlite-vec. Recs KNN is Index.Query via the cloudflare-go SDK.
  • Singleton process. max_instances: 1. Two containers would each consume Jetstream.
  • Cold start. Cron pokes /ready so sleepAfter does not drop the websocket.

12.8 Proxy security model #

/d1 and /r2 run as outbound handlers on the virtual host glean.proxy: the runtime intercepts requests to that host from inside the container's network namespace and runs the handler next to the bindings, so only the container can reach them. /internal/tick is reachable only through the Durable Object binding the Worker cron uses. Bearer tokens still gate every endpoint as a second layer:

  • Each endpoint derives its own token (db.ProxyToken, SHA-256 of GLEAN_SESSION_KEY + purpose), so one leaked token grants only that endpoint's scope.
  • The tokens reveal nothing about GLEAN_SESSION_KEY itself (one-way hash), which alone signs session cookies, so a leaked proxy token cannot forge sessions.
  • Bindings scope the blast radius: one D1 database, one R2 bucket, no account-level API. /r2 exposes no listing.
  • Comparison is constant-time on both sides, and the tokens never leave the machine or the logs.
  • Rotating GLEAN_SESSION_KEY rotates every token and invalidates all user sessions in one step.

The precondition is a high-entropy GLEAN_SESSION_KEY (openssl rand -hex 32); a weak key makes every derived token brute-forceable.

13. Configuration #

All GLEAN_* vars are set as Worker vars (wrangler.jsonc, which carries the deployment values) or Worker secrets; the Worker forwards them to the container. The Go binary owns the defaults below (main.go).

Variable Default What it does
GLEAN_SESSION_KEY (required) High-entropy secret (openssl rand -hex 32); signs session cookies and derives the /d1, /r2, /tick proxy tokens
GLEAN_FRONTEND_URL (required) Public origin; the OAuth callback is $GLEAN_FRONTEND_URL/api/auth/callback
GLEAN_OAUTH_CLIENT_ID https://glean.at/api/oauth/client-metadata OAuth client-metadata URL; empty runs the localhost dev flow
GLEAN_JETSTREAM wss://jetstream1.eurosky.network Jetstream WebSocket URL
GLEAN_PLC_URL https://plc.eurosky.network PLC directory URL for DID resolution
GLEAN_ARCHIVE_WINDOW 90 Days article bodies stay in D1 before the offload pass moves them to R2 (0 disables)
GLEAN_ARTICLE_RETENTION_DAYS 30 Days article rows live before maintenance deletes them; liked, annotated, and undated articles survive
GLEAN_COLLECTION_DIR_URL (empty) Collection directory URL; a fresh database bootstraps from it once on boot
GLEAN_BACKFILL_CONCURRENCY 5 Max concurrent backfill workers
GLEAN_EMBED_BASE_URL (empty) Embeddings API base URL (recommended)
GLEAN_EMBED_API_KEY (empty) API key for the embeddings endpoint
GLEAN_EMBED_MODEL text-embedding-3-small Embedding model name; must output exactly the Vectorize index dimensions (a mismatch fails upserts loudly)
GLEAN_LLM_BASE_URL (empty) LLM API base URL for language detection and digest summaries
GLEAN_LLM_API_KEY (empty) API key for the LLM endpoint
GLEAN_LLM_MODEL gpt-4o-mini LLM model name
GLEAN_CLOUDFLARE_ACCOUNT_ID (secret) Account for Vectorize
GLEAN_CLOUDFLARE_API_TOKEN (secret) API token with Vectorize permissions
GLEAN_D1_PROXY_URL http://glean.proxy (var) Outbound host the container POSTs SQL (/d1) and article bodies (/r2) to; also enables the body archive
GLEAN_VECTORIZE_ARTICLES glean-articles Vectorize index for article embeddings
GLEAN_VECTORIZE_FEEDS glean-feeds Vectorize index for feed embeddings
GLEAN_ADDR :8080 Container listen address (set by the Worker)

14. Future Considerations #

  • Email digest: Periodic email with top articles from subscribed feeds
  • Digest personalization: Feed the user's liked topics and reading patterns into the digest prompt for more targeted summaries