diff --git a/docs/README.md b/docs/README.md index 4e0293b..c0a204a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,13 +1,21 @@ # Twisted Documentation -Documentation is organized by project: +## Reference -- [`app/`](app/) for the Ionic/Vue client -- [`api/`](api/) for the Go Tap/index/search service +Completed work — architecture, APIs, and data models as built. -## Quick Links +- [`reference/api.md`](reference/api.md) — Go search API service +- [`reference/app.md`](reference/app.md) — Ionic Vue mobile app +- [`reference/lexicons.md`](reference/lexicons.md) — Tangled AT Protocol record types -- App spec index: [`app/specs/README.md`](app/specs/README.md) -- App task index: [`app/tasks/phase-6.md`](app/tasks/phase-6.md) -- API spec index: [`api/specs/README.md`](api/specs/README.md) -- API task index: [`api/tasks/README.md`](api/tasks/README.md) +## Specs + +Forward-looking designs for remaining work. + +- [`specs/data-sources.md`](specs/data-sources.md) — Constellation, Tangled XRPC, Tap, AT Protocol, Bluesky OAuth +- [`specs/search.md`](specs/search.md) — Keyword, semantic, and hybrid search +- [`specs/app-features.md`](specs/app-features.md) — Remaining mobile app features + +## Roadmap + +- [`roadmap.md`](roadmap.md) — All remaining milestones and tasks diff --git a/docs/TODO.md b/docs/TODO.md deleted file mode 100644 index 259cade..0000000 --- a/docs/TODO.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: To-Dos -updated: 2026-03-23 ---- - -A catch-all for ideas, issues/bugs, and future work that doesn't fit into the current specs or tasks. This is a "parking lot." - -## App - -- Repo stars, forks, etc. are not properly parsed from JSON. -- ATOM/RSS feed link for repos: (`tangled.org/{did}/{repo}/feed.atom`) - -## API diff --git a/docs/api/deploy.md b/docs/api/deploy.md deleted file mode 100644 index 713cb20..0000000 --- a/docs/api/deploy.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -title: "Deployment Guide" -updated: 2026-03-23 ---- - -# Railway Deployment Guide - -Deploy the Twister API and indexer as Railway services alongside the existing Tap instance. - -## Prerequisites - -- Railway project with Tap already deployed -- Turso database created with auth token -- GitHub repository connected to Railway - -## Service Layout - -| Service | Start Command | Health Check | Public | Port | -| ------- | ----------------- | -------------- | ------ | ---- | -| tap | (pre-existing) | `GET /health` | no | — | -| api | `twister api` | `GET /healthz` | yes | 8080 | -| indexer | `twister indexer` | `GET /health` | no | 9090 | - -All services use the same Docker image. Railway overrides `CMD` with the per-service start command. - -## Step 1 — Create Services - -In the Railway dashboard, create two new services from the same GitHub repo: - -1. **api** — set start command to `twister api` -2. **indexer** — set start command to `twister indexer` - -Both services build from `packages/api/Dockerfile`. - -## Step 2 — Set Environment Variables - -### Shared (set on both services) - -```sh -TURSO_DATABASE_URL=libsql://twister-prod-.turso.io -TURSO_AUTH_TOKEN= -LOG_LEVEL=info -LOG_FORMAT=json -``` - -### API only - -```sh -HTTP_BIND_ADDR=:8080 -SEARCH_DEFAULT_LIMIT=20 -SEARCH_MAX_LIMIT=100 -``` - -### Indexer only - -```sh -TAP_URL=wss://${{tap.RAILWAY_PRIVATE_DOMAIN}}/channel -TAP_AUTH_PASSWORD= -INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,sh.tangled.string,sh.tangled.actor.profile,sh.tangled.repo.issue.comment,sh.tangled.repo.pull.comment,sh.tangled.repo.issue.state,sh.tangled.repo.pull.status,sh.tangled.feed.star -INDEXER_HEALTH_ADDR=:9090 -``` - -Use `${{tap.RAILWAY_PRIVATE_DOMAIN}}` to reference Tap's internal hostname. This keeps traffic on Railway's private network. - -## Step 3 — Configure Health Checks - -In the Railway dashboard, configure per-service: - -- **api**: HTTP health check on path `/healthz`, port `8080` -- **indexer**: HTTP health check on path `/health`, port `9090` - -Railway uses these to gate deployment rollouts and restart unhealthy containers. - -## Step 4 — Configure Autodeploy - -Connect the GitHub repository in the Railway dashboard. Railway will build and deploy on every push to the configured branch. - -The Dockerfile uses multi-stage builds with `CGO_ENABLED=0` for a static binary on Alpine. - -## Step 5 — Deploy and Verify - -After the first deploy: - -1. Confirm API is healthy: `curl https:///healthz` -2. Confirm API readiness: `curl https:///readyz` -3. Check indexer health in Railway logs (health check on `:9090/health`) - -## Step 6 — Bootstrap Content - -Run graph backfill to populate initial content from seed users: - -```bash -twister backfill --seeds=docs/api/seeds.txt --max-hops=2 -``` - -Wait for Tap to finish historical sync, then verify search returns results: - -```bash -curl "https:///search?q=tangled" -``` diff --git a/docs/api/seeds.txt b/docs/api/seeds.txt deleted file mode 100644 index a8f8488..0000000 --- a/docs/api/seeds.txt +++ /dev/null @@ -1,9 +0,0 @@ -# Example seed handles for Twister graph backfill -# One DID or handle per line. Comments and blank lines are ignored. - -anirudh.fi -atprotocol.dev -zzstoatzz.io -oppi.li -desertthunder.dev -tangled.org diff --git a/docs/api/specs/01-architecture.md b/docs/api/specs/01-architecture.md deleted file mode 100644 index 2dbc76c..0000000 --- a/docs/api/specs/01-architecture.md +++ /dev/null @@ -1,313 +0,0 @@ ---- -title: "Spec 01 — Architecture" -updated: 2026-03-22 ---- - -## 1. Purpose - -Build a Go-based search service for Tangled content on AT Protocol that: - -- ingests Tangled records through **Tap** (already deployed on Railway) -- denormalizes them into internal search documents -- indexes them in **Turso/libSQL** -- exposes a search API with **keyword**, **semantic**, and **hybrid** retrieval modes -- exposes index-backed summary APIs for data the public Tangled APIs do not answer efficiently, such as followers - -## 2. Functional Goals - -The system shall: - -- index Tangled-specific ATProto collections under the `sh.tangled.*` namespace -- support initial backfill and continuous incremental sync via Tap -- support lexical retrieval using Turso's Tantivy-backed FTS -- support semantic retrieval using vector embeddings -- support hybrid ranking combining lexical and semantic signals -- expose stable HTTP APIs for search, document lookup, and graph/profile summaries -- support deployment on **Railway** - -## 3. Non-Functional Goals - -The system shall prioritize: - -- **correctness of sync** — cursors never advance ahead of committed data -- **operational simplicity** — single binary, subcommand-driven -- **incremental delivery** — keyword search ships before embeddings -- **small deployable services** — process groups, not microservices -- **reindexability** — any document or collection can be re-normalized and re-indexed -- **low coupling** — sync, indexing, and serving are independent concerns - -## 4. Out of Scope (v1) - -- code-aware symbol search -- sourcegraph-style structural search -- personalized ranking -- access control beyond public/private visibility flags in indexed records -- full analytics pipeline -- custom ANN infrastructure outside Turso/libSQL - -## 5. Design Principles - -1. **Tap owns synchronization correctness.** The application does not consume the raw firehose. Tap handles connection, cryptographic verification, backfill, and filtering. - -2. **The indexer owns denormalization.** Raw ATProto records are never queried directly by the public API. - -3. **The public API serves denormalized projections.** Search ranking and graph summaries depend on the indexed document model, not transport. - -4. **Keyword search is the baseline.** Semantic and hybrid search are layered on top. - -5. **Embeddings are asynchronous.** Ingestion is never blocked on vector generation unless explicitly configured. - -6. **Twister complements public Tangled APIs.** Repo detail stays on knots/PDSes; the index adds discovery and cross-network summaries. - -## 6. External Systems - -- **AT Protocol network** — source of all Tangled content -- **Tap** — filtered event delivery from the AT Protocol firehose (deployed on Railway) -- **Turso/libSQL** — relational storage, Tantivy-backed FTS, and native vector search -- **Ollama** — local embedding model server (nomic-embed-text or EmbeddingGemma); deployed as a Railway sidecar service -- **Railway** — deployment platform for Twister services, Tap, and Ollama - -## 7. Architecture Summary - -```text -ATProto Firehose / PDS - │ - ▼ - Tap (Railway) - │ WebSocket / webhook JSON events - ▼ - Go Indexer Service - ├─ decode Tap events - ├─ normalize records → documents - ├─ upsert documents - ├─ schedule embeddings - └─ persist sync cursor - │ - ▼ - Turso/libSQL - ├─ documents table - ├─ document_embeddings table - ├─ FTS index (Tantivy-backed) - ├─ vector index (DiskANN) - └─ sync_state table - │ - ▼ - Go Search API - ├─ keyword search (fts_match / fts_score) - ├─ semantic search (vector_top_k) - ├─ hybrid search (weighted merge) - ├─ profile and graph summaries - └─ document fetch -``` - -## 8. Runtime Units - -| Unit | Role | Deployment | -| -------------- | -------------------------------------------- | -------------------------- | -| `api` | HTTP search, graph summary, and document API | Railway service (public) | -| `indexer` | Tap consumer, normalizer, DB writer | Railway service (internal) | -| `embed-worker` | Async embedding generation via Ollama | Optional Railway service | -| `ollama` | Local embedding model server | Railway service (internal) | -| `tap` | ATProto sync | Railway (already deployed) | - -## 9. Repository Structure - -```text -main.go - -internal/ - api/ # HTTP handlers, middleware, routes - config/ # Config struct, env parsing - embed/ # Embedding provider abstraction, worker - index/ # FTS and vector index management - ingest/ # Tap event consumer, ingestion loop - normalize/ # Per-collection record → document adapters - observability/# Structured logging, metrics - ranking/ # Score normalization, hybrid merge - search/ # Search orchestration (keyword, semantic, hybrid) - store/ # DB access layer, migrations, domain types - tapclient/ # Tap WebSocket/webhook client -``` - -## 10. Binary Subcommands - -```bash -twister api # Start HTTP search API -twister indexer # Start Tap consumer / indexer -twister embed-worker # Start async embedding worker -twister reindex # Re-normalize and upsert documents -twister reembed # Re-generate embeddings -twister backfill # Bootstrap index from seed users -twister healthcheck # One-shot health probe -``` - -## 11. Technology Choices - -### Embedding: Ollama (self-hosted) - -Embeddings are generated locally via Ollama rather than an external API service. This eliminates per-token costs, external service dependencies, and data egress concerns. - -**Recommended models (in order of preference):** - -| Model | Parameters | Dimensions | Quantized Size | Notes | -|-------|-----------|------------|----------------|-------| -| nomic-embed-text-v1.5 | 137M | 768 (Matryoshka: 64–768) | ~262 MB (F16) | 8192 context, battle-tested, Railway template exists | -| EmbeddingGemma | 308M | 768 | <200 MB (quantized) | Best-in-class MTEB for size, released Sept 2025 | -| all-minilm | 23M | 384 | ~46 MB | Budget option, lower quality | - -**Go integration:** Use the official Ollama Go client (`github.com/ollama/ollama/api`) with the `Embed()` method. The embed-worker calls Ollama over Railway's internal network (`ollama.railway.internal:11434`). - -**Railway deployment:** Ollama runs as a separate Railway service (~1–2 GB RAM, 1–2 vCPU, ~$10–30/mo). The nomic-embed Railway template provides a proven starting point. No cold starts on always-on services; model loads in 2–10 seconds on first request after deploy. - -### Language: Go - -Go is the implementation language for the API server, indexer, embedding worker, and CLI commands. Rationale: straightforward long-running services, excellent HTTP support, good concurrency model, small container footprint. - -### Sync Layer: Tap - -Tap is the only supported sync source in v1. It handles firehose connection, cryptographic verification, backfill, and filtering, then delivers simple JSON events via WebSocket or webhook. - -**Tap is already deployed on Railway.** Twister connects to it as a WebSocket client. - -#### Tap Capabilities - -- Validates repo structure, MST integrity, and identity signatures -- Automatic backfill fetches full repo history from PDS when repos are added -- Filtered output by DID list, collection, or full network mode -- Ordering guarantees: historical events (`live: false`) delivered before live events (`live: true`) - -#### Tap Delivery Modes - -| Mode | Config | Behavior | -| -------------------------- | ----------------------- | ------------------------------------------------- | -| WebSocket + acks (default) | — | Client acks each event; no data loss | -| Fire-and-forget | `TAP_DISABLE_ACKS=true` | Events marked acked on receipt; simpler but lossy | -| Webhook | `TAP_WEBHOOK_URL=...` | Events POSTed as JSON; acked on HTTP 200 | - -#### Tap API Endpoints (reference) - -| Endpoint | Method | Purpose | -| --------------------- | ------ | ------------------------------------- | -| `/health` | GET | Health check | -| `/channel` | WS | WebSocket event stream | -| `/repos/add` | POST | Add DIDs to track | -| `/repos/remove` | POST | Stop tracking a repo | -| `/info/:did` | GET | Repo state, rev, record count, errors | -| `/stats/repo-count` | GET | Total tracked repos | -| `/stats/record-count` | GET | Total tracked records | -| `/stats/cursors` | GET | Firehose and list repos cursors | - -#### Key Tap Configuration - -| Variable | Default | Purpose | -| ------------------------ | ------- | ---------------------------------------------------------------------------------- | -| `TAP_SIGNAL_COLLECTION` | — | Auto-track repos with records in this collection | -| `TAP_COLLECTION_FILTERS` | — | Comma-separated collection filters (e.g., `sh.tangled.repo,sh.tangled.repo.issue`) | -| `TAP_ADMIN_PASSWORD` | — | Basic auth for API access | -| `TAP_DISABLE_ACKS` | `false` | Fire-and-forget mode | -| `TAP_WEBHOOK_URL` | — | Webhook delivery URL | - -### Storage and Search: Turso/libSQL - -Turso/libSQL is used for relational metadata storage, Tantivy-backed full-text search, and native vector search. - -#### Go SDK Options - -| Package | CGo | Embedded Replicas | Remote | -| -------------------------------------------------- | --- | ----------------- | ------ | -| `github.com/tursodatabase/go-libsql` | Yes | Yes | Yes | -| `github.com/tursodatabase/libsql-client-go/libsql` | No | No | Yes | - -Both register as `database/sql` drivers under `"libsql"`. They cannot be imported in the same binary. - -**Recommendation:** Use `libsql-client-go` (pure Go, remote-only) unless embedded replicas are needed for local read performance. - -#### Connection Patterns - -```go -// Remote only (pure Go, no CGo) -import _ "github.com/tursodatabase/libsql-client-go/libsql" -db, err := sql.Open("libsql", "libsql://your-db.turso.io?authToken=TOKEN") - -// Embedded replica (CGo required) -import "github.com/tursodatabase/go-libsql" -connector, err := libsql.NewEmbeddedReplicaConnector( - "local.db", "libsql://your-db.turso.io", - libsql.WithAuthToken("TOKEN"), - libsql.WithSyncInterval(time.Minute), -) -db := sql.OpenDB(connector) -``` - -#### Full-Text Search (Tantivy-backed) - -Turso FTS is **not** standard SQLite FTS5. It uses Tantivy under the hood. - -```sql --- Create FTS index with per-column tokenizers and weights -CREATE INDEX idx_docs_fts ON documents USING fts ( - title WITH tokenizer=default, - body WITH tokenizer=default, - summary WITH tokenizer=default, - repo_name WITH tokenizer=simple, - author_handle WITH tokenizer=raw -) WITH (weights='title=3.0,repo_name=2.5,author_handle=2.0,summary=1.5,body=1.0'); - --- Filter by match -SELECT id, title FROM documents -WHERE fts_match(title, body, summary, repo_name, author_handle, 'search query'); - --- BM25 scoring -SELECT id, title, fts_score(title, body, summary, repo_name, author_handle, 'search query') AS score -FROM documents -ORDER BY score DESC; - --- Highlighting -SELECT fts_highlight(title, '', '', 'search query') AS highlighted -FROM documents; -``` - -**Available tokenizers:** `default` (Unicode-aware), `raw` (exact match), `simple` (whitespace+punctuation), `whitespace`, `ngram` (2-3 char n-grams). - -**Query syntax (Tantivy):** `database AND search`, `database NOT nosql`, `"exact phrase"`, `data*` (prefix), `title:database` (field-specific), `title:database^2` (boosting). - -**Limitations:** No snippet function (use highlighting). No automatic segment merging (manual `OPTIMIZE INDEX` required). -No read-your-writes within a transaction. No MATCH operator (use `fts_match()` function). - -#### Vector Search - -```sql --- Vector column type -embedding F32_BLOB(768) - --- Insert -INSERT INTO document_embeddings (document_id, embedding, ...) -VALUES (?, vector32(?), ...); -- ? is JSON array '[0.1, 0.2, ...]' - --- Brute-force similarity search -SELECT d.id, vector_distance_cos(e.embedding, vector32(?)) AS distance -FROM documents d -JOIN document_embeddings e ON d.id = e.document_id -ORDER BY distance ASC LIMIT 20; - --- Create ANN index (DiskANN) -CREATE INDEX idx_embeddings ON document_embeddings( - libsql_vector_idx(embedding, 'metric=cosine') -); - --- ANN search via index -SELECT d.id, d.title -FROM vector_top_k('idx_embeddings', vector32(?), 20) AS v -JOIN document_embeddings e ON e.rowid = v.id -JOIN documents d ON d.id = e.document_id; -``` - -**Vector types:** `F32_BLOB` (recommended), `F16_BLOB`, `F64_BLOB`, `F8_BLOB`, `F1BIT_BLOB`. - -**Distance functions:** `vector_distance_cos` (cosine), `vector_distance_l2` (Euclidean). - -**Max dimensions:** 65,536. Dimension is fixed at table creation. - -### Deployment: Railway - -Railway is the deployment platform. It supports health checks, autodeploy, per-service scaling, and internal networking. Tap is already deployed here. Twister deploys as separate Railway services (api, indexer, embed-worker) within the same project. diff --git a/docs/api/specs/02-tangled-lexicons.md b/docs/api/specs/02-tangled-lexicons.md deleted file mode 100644 index 8e8e2a1..0000000 --- a/docs/api/specs/02-tangled-lexicons.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: "Spec 02 — Tangled Lexicons" -updated: 2026-03-22 -source: https://github.com/mary-ext/atcute/tree/trunk/packages/definitions/tangled/lexicons/sh/tangled ---- - -All Tangled records use the `sh.tangled.*` namespace. Records use TID keys unless noted otherwise. - -## 1. Searchable Record Types - -These are the primary records Twister indexes for search. - -### sh.tangled.repo - -Repository metadata. Key: `tid`. - -| Field | Type | Required | Description | -| ------------- | -------- | -------- | ---------------------------------------------- | -| `name` | string | yes | Repository name | -| `knot` | string | yes | Knot (hosting node) where the repo was created | -| `spindle` | string | no | CI runner for jobs | -| `description` | string | no | 1–140 graphemes | -| `website` | uri | no | Related URI | -| `topics` | string[] | no | Up to 50 topic tags, each 1–50 chars | -| `source` | uri | no | Upstream source | -| `labels` | at-uri[] | no | Label definitions this repo subscribes to | -| `createdAt` | datetime | yes | | - -### sh.tangled.repo.issue - -Issue on a repository. Key: `tid`. - -| Field | Type | Required | Description | -| ------------ | -------- | -------- | -------------------------------- | -| `repo` | at-uri | yes | AT-URI of the parent repo record | -| `title` | string | yes | Issue title | -| `body` | string | no | Issue body (markdown) | -| `createdAt` | datetime | yes | | -| `mentions` | did[] | no | Mentioned users | -| `references` | at-uri[] | no | Referenced records | - -### sh.tangled.repo.pull - -Pull request. Key: `tid`. - -| Field | Type | Required | Description | -| ------------ | -------- | -------- | -------------------------------------------------- | -| `target` | object | yes | `{repo: at-uri, branch: string}` | -| `title` | string | yes | PR title | -| `body` | string | no | PR description (markdown) | -| `patchBlob` | blob | yes | Patch content (`text/x-patch`) | -| `source` | object | no | `{branch: string, sha: string(40), repo?: at-uri}` | -| `createdAt` | datetime | yes | | -| `mentions` | did[] | no | Mentioned users | -| `references` | at-uri[] | no | Referenced records | - -### sh.tangled.string - -Code snippet / gist. Key: `tid`. - -| Field | Type | Required | Description | -| ------------- | -------- | -------- | ------------------- | -| `filename` | string | yes | 1–140 graphemes | -| `description` | string | yes | Up to 280 graphemes | -| `createdAt` | datetime | yes | | -| `contents` | string | yes | Snippet content | - -### sh.tangled.actor.profile - -User profile. Key: `literal:self` (singleton per account). - -| Field | Type | Required | Description | -| -------------------- | -------- | -------- | ---------------------------- | -| `avatar` | blob | no | PNG/JPEG, max 1MB | -| `description` | string | no | Bio, up to 256 graphemes | -| `links` | uri[] | no | Up to 5 social/website links | -| `stats` | string[] | no | Up to 2 vanity stat types | -| `bluesky` | boolean | yes | Show Bluesky link | -| `location` | string | no | Up to 40 graphemes | -| `pinnedRepositories` | at-uri[] | no | Up to 6 pinned repos | -| `pronouns` | string | no | Up to 40 chars | - -## 2. Interaction Record Types - -These records represent social interactions. They may be indexed for counts/signals but are lower priority for text search. - -### sh.tangled.feed.star - -Star/favorite on a record. Key: `tid`. - -| Field | Type | Required | -| ----------- | -------- | -------- | -| `subject` | at-uri | yes | -| `createdAt` | datetime | yes | - -### sh.tangled.feed.reaction - -Emoji reaction on a record. Key: `tid`. - -| Field | Type | Required | Description | -| ----------- | -------- | -------- | ------------------------------- | -| `subject` | at-uri | yes | | -| `reaction` | string | yes | One of: 👍 👎 😆 🎉 🫤 ❤️ 🚀 👀 | -| `createdAt` | datetime | yes | | - -### sh.tangled.graph.follow - -Follow a user. Key: `tid`. - -| Field | Type | Required | -| ----------- | -------- | -------- | -| `subject` | did | yes | -| `createdAt` | datetime | yes | - -## 3. State Record Types - -These records track mutable state of issues and PRs. - -### sh.tangled.repo.issue.state - -| Field | Type | Required | Description | -| ------- | ------ | -------- | -------------------------------------------------------------------------- | -| `issue` | at-uri | yes | | -| `state` | string | yes | `sh.tangled.repo.issue.state.open` or `sh.tangled.repo.issue.state.closed` | - -### sh.tangled.repo.pull.status - -| Field | Type | Required | Description | -| -------- | ------ | -------- | ----------------------------------------------------------- | -| `pull` | at-uri | yes | | -| `status` | string | yes | `sh.tangled.repo.pull.status.open`, `.closed`, or `.merged` | - -## 4. Comment Record Types - -### sh.tangled.repo.issue.comment - -| Field | Type | Required | Description | -| ------------ | -------- | -------- | ------------------------------ | -| `issue` | at-uri | yes | Parent issue | -| `body` | string | yes | Comment body | -| `createdAt` | datetime | yes | | -| `replyTo` | at-uri | no | Parent comment (for threading) | -| `mentions` | did[] | no | | -| `references` | at-uri[] | no | | - -### sh.tangled.repo.pull.comment - -| Field | Type | Required | Description | -| ------------ | -------- | -------- | ------------ | -| `pull` | at-uri | yes | Parent PR | -| `body` | string | yes | Comment body | -| `createdAt` | datetime | yes | | -| `mentions` | did[] | no | | -| `references` | at-uri[] | no | | - -## 5. Infrastructure Record Types - -These are not indexed for search but may be consumed for operational context. - -| Collection | Description | -| ----------------------------- | ---------------------------------------------------- | -| `sh.tangled.label.definition` | Label definitions with name, valueType, scope, color | -| `sh.tangled.label.op` | Label application operations | -| `sh.tangled.git.refUpdate` | Git reference update events | -| `sh.tangled.knot.member` | Knot membership | -| `sh.tangled.spindle.member` | Spindle (CI runner) membership | -| `sh.tangled.pipeline.status` | CI pipeline status | - -## 6. Collection Priority for v1 Indexing - -| Priority | Collection | Rationale | -| -------- | ------------------------------- | ------------------------------------ | -| P0 | `sh.tangled.repo` | Core searchable content | -| P0 | `sh.tangled.repo.issue` | High-signal text content | -| P0 | `sh.tangled.repo.pull` | High-signal text content | -| P1 | `sh.tangled.string` | Searchable code snippets | -| P1 | `sh.tangled.actor.profile` | User/org discovery | -| P2 | `sh.tangled.repo.issue.comment` | Body text, high volume | -| P2 | `sh.tangled.repo.pull.comment` | Body text, high volume | -| P2 | `sh.tangled.repo.issue.state` | State for filtering, not text search | -| P2 | `sh.tangled.repo.pull.status` | State for filtering, not text search | -| P3 | `sh.tangled.feed.star` | Ranking signal (star count) | -| P3 | `sh.tangled.feed.reaction` | Ranking signal | -| P3 | `sh.tangled.graph.follow` | Ranking signal | - -### Tap Collection Filter for v1 - -```sh -TAP_COLLECTION_FILTERS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.issue.comment,sh.tangled.repo.issue.state,sh.tangled.repo.pull,sh.tangled.repo.pull.comment,sh.tangled.repo.pull.status,sh.tangled.string,sh.tangled.actor.profile,sh.tangled.feed.star - -# or sh.tangled.* -``` diff --git a/docs/api/specs/03-data-model.md b/docs/api/specs/03-data-model.md deleted file mode 100644 index 714de4f..0000000 --- a/docs/api/specs/03-data-model.md +++ /dev/null @@ -1,184 +0,0 @@ ---- -title: "Spec 03 — Data Model" -updated: 2026-03-22 ---- - -## 1. Search Document - -A **search document** is the internal denormalized representation used for retrieval. It is derived from one or more ATProto records via normalization. - -### Stable Identifier - -```sh -id = did + "|" + collection + "|" + rkey -``` - -Example: `did:plc:abc123|sh.tangled.repo|3kb3fge5lm32x` - -### Required Fields - -| Field | Type | Description | -| --------------- | ------- | -------------------------------------------------------------------------- | -| `id` | TEXT PK | Stable composite identifier | -| `did` | TEXT | Author DID | -| `collection` | TEXT | ATProto collection NSID | -| `rkey` | TEXT | Record key (TID) | -| `at_uri` | TEXT | Full AT-URI | -| `cid` | TEXT | Content identifier (hash) | -| `record_type` | TEXT | Normalized type label (e.g., `repo`, `issue`, `pull`, `string`, `profile`) | -| `title` | TEXT | Normalized title | -| `body` | TEXT | Normalized body text | -| `summary` | TEXT | Short summary / description | -| `repo_did` | TEXT | DID of the repo owner (resolved from at-uri for issues/PRs) | -| `repo_name` | TEXT | Repository name (resolved) | -| `author_handle` | TEXT | Author handle (resolved via identity) | -| `tags_json` | TEXT | JSON array of tags/topics | -| `language` | TEXT | Detected or declared language | -| `created_at` | TEXT | Record creation timestamp (ISO 8601) | -| `updated_at` | TEXT | Last record update timestamp | -| `indexed_at` | TEXT | When this document was last indexed | -| `deleted_at` | TEXT | Soft-delete timestamp (tombstone) | - -### Derived Fields (not stored in documents table) - -| Field | Location | Description | -| ---------------- | -------------------------------------- | ------------------------------ | -| Embedding vector | `document_embeddings` table | F32_BLOB(N) | -| FTS index | Turso FTS index | Tantivy-backed full-text index | -| Star count | Aggregated from `sh.tangled.feed.star` | Ranking signal | - -## 2. Core Documents Table - -```sql -CREATE TABLE documents ( - id TEXT PRIMARY KEY, - did TEXT NOT NULL, - collection TEXT NOT NULL, - rkey TEXT NOT NULL, - at_uri TEXT NOT NULL, - cid TEXT NOT NULL, - record_type TEXT NOT NULL, - title TEXT, - body TEXT, - summary TEXT, - repo_did TEXT, - repo_name TEXT, - author_handle TEXT, - tags_json TEXT, - language TEXT, - created_at TEXT, - updated_at TEXT, - indexed_at TEXT NOT NULL, - deleted_at TEXT -); - -CREATE INDEX idx_documents_did ON documents(did); -CREATE INDEX idx_documents_collection ON documents(collection); -CREATE INDEX idx_documents_record_type ON documents(record_type); -CREATE INDEX idx_documents_repo_did ON documents(repo_did); -CREATE INDEX idx_documents_created_at ON documents(created_at); -CREATE INDEX idx_documents_deleted_at ON documents(deleted_at); -``` - -## 3. FTS Index - -```sql -CREATE INDEX idx_documents_fts ON documents USING fts ( - title WITH tokenizer=default, - body WITH tokenizer=default, - summary WITH tokenizer=default, - repo_name WITH tokenizer=simple, - author_handle WITH tokenizer=raw, - tags_json WITH tokenizer=simple -) WITH (weights='title=3.0,repo_name=2.5,author_handle=2.0,summary=1.5,tags_json=1.2,body=1.0'); -``` - -### FTS Maintenance - -Turso's Tantivy-backed FTS uses `NoMergePolicy` — segment count grows with writes and is never automatically compacted. This increases query fan-out over time. - -**Required maintenance:** Run `OPTIMIZE INDEX idx_documents_fts;` periodically (e.g., daily cron or after bulk backfill). This merges segments and reclaims space. - -**Known limitations:** -- No read-your-writes within a transaction — FTS queries see a pre-commit snapshot -- No snippet function (use `fts_highlight()` for highlighting) -- FTS is experimental in Turso; requires the `fts` feature flag - -## 4. Embeddings Table - -```sql -CREATE TABLE document_embeddings ( - document_id TEXT PRIMARY KEY REFERENCES documents(id), - embedding F32_BLOB(768), - embedding_model TEXT NOT NULL, - embedded_at TEXT NOT NULL -); - -CREATE INDEX idx_embeddings_vec ON document_embeddings( - libsql_vector_idx(embedding, 'metric=cosine') -); -``` - -The vector dimension (768) matches nomic-embed-text-v1.5 and EmbeddingGemma defaults. Changing models may require a new column or table migration if the dimension changes. - -### Vector Index Tuning - -The DiskANN index accepts tuning parameters at creation time: - -```sql -CREATE INDEX idx_embeddings_vec ON document_embeddings( - libsql_vector_idx(embedding, 'metric=cosine', 'max_neighbors=50', 'search_l=200') -); -``` - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `max_neighbors` | 3*sqrt(D) | Graph connectivity; higher = better recall, more storage | -| `search_l` | 200 | Neighbors visited during search; higher = better recall, slower | -| `insert_l` | 70 | Neighbors visited during insert | -| `alpha` | 1.2 | Graph sparsity factor | -| `compress_neighbors` | — | Quantize neighbor vectors for storage savings | - -Start with defaults and tune after measuring recall on representative queries. - -## 5. Sync State Table - -```sql -CREATE TABLE sync_state ( - consumer_name TEXT PRIMARY KEY, - cursor TEXT NOT NULL, - high_water_mark TEXT, - updated_at TEXT NOT NULL -); -``` - -Stores the Tap event ID that has been successfully committed. On restart, the indexer resumes from this cursor. - -## 6. Embedding Jobs Table - -```sql -CREATE TABLE embedding_jobs ( - document_id TEXT PRIMARY KEY REFERENCES documents(id), - status TEXT NOT NULL, -- 'pending', 'processing', 'completed', 'failed' - attempts INTEGER NOT NULL DEFAULT 0, - last_error TEXT, - scheduled_at TEXT NOT NULL, - updated_at TEXT NOT NULL -); - -CREATE INDEX idx_embedding_jobs_status ON embedding_jobs(status); -``` - -## 7. Issue/PR State Cache (optional) - -To support filtering search results by issue state or PR status without joining back to the raw records: - -```sql -CREATE TABLE record_state ( - subject_uri TEXT PRIMARY KEY, -- at-uri of the issue or PR - state TEXT NOT NULL, -- 'open', 'closed', 'merged' - updated_at TEXT NOT NULL -); -``` - -Updated when `sh.tangled.repo.issue.state` or `sh.tangled.repo.pull.status` events are ingested. diff --git a/docs/api/specs/04-data-pipeline.md b/docs/api/specs/04-data-pipeline.md deleted file mode 100644 index e2f5447..0000000 --- a/docs/api/specs/04-data-pipeline.md +++ /dev/null @@ -1,364 +0,0 @@ ---- -title: "Spec 04 — Data Pipeline" -updated: 2026-03-22 ---- - -Covers the full data path: Tap event ingestion, record normalization, and failure handling. - -## 1. Tap Event Format - -### Record Events - -```json -{ - "id": 12345, - "type": "record", - "record": { - "live": true, - "rev": "3kb3fge5lm32x", - "did": "did:plc:abc123", - "collection": "sh.tangled.repo", - "rkey": "3kb3fge5lm32x", - "action": "create", - "cid": "bafyreig...", - "record": { - "$type": "sh.tangled.repo", - "name": "my-project", - "knot": "knot.tangled.org", - "description": "A cool project", - "topics": ["go", "search"], - "createdAt": "2026-03-22T12:00:00.000Z" - } - } -} -``` - -Key fields: - -- `id` — monotonic event ID, used as cursor -- `type` — `"record"` or `"identity"` -- `record.live` — `true` for real-time events, `false` for backfill -- `record.action` — `"create"`, `"update"`, or `"delete"` -- `record.did` — author DID -- `record.collection` — ATProto collection NSID -- `record.rkey` — record key -- `record.cid` — content identifier -- `record.record` — the full ATProto record payload (absent on delete) - -### Identity Events - -```json -{ - "id": 12346, - "type": "identity", - "identity": { - "did": "did:plc:abc123", - "handle": "alice.tangled.org", - "isActive": true, - "status": "active" - } -} -``` - -Identity events are always delivered for tracked repos, regardless of collection filters. - -## 2. WebSocket Protocol - -### Connection - -Connect to `wss:///channel` (or `ws://` for local dev). - -If `TAP_ADMIN_PASSWORD` is set, authenticate with HTTP Basic auth (`admin:`). - -### Acknowledgment Protocol - -Default mode requires the client to ack each event by sending the event `id` back over the WebSocket. Events are retried after `TAP_RETRY_TIMEOUT` (default 60s) if unacked. - -For simpler development, set `TAP_DISABLE_ACKS=true` on Tap for fire-and-forget delivery. - -### Ordering Guarantees - -Events are ordered **per-repo** (per-DID), not globally: - -- **Historical events** (`live: false`) may be sent concurrently within a repo -- **Live events** (`live: true`) are synchronization barriers — all prior events for that repo must complete before a live event is sent -- No ordering guarantee across different repos - -Example sequence for one repo: `H1, H2, L1, H3, H4, L2` - -- H1 and H2 sent concurrently -- Wait for completion, send L1 alone -- Wait for L1, send H3 and H4 concurrently -- Wait for completion, send L2 alone - -### Delivery Guarantee - -Events are delivered **at least once**. Duplicates may occur on crashes or ack timeouts. The indexer must handle idempotent upserts. - -## 3. Ingestion Contract - -For each event, the indexer: - -1. Validates `type` is `"record"` (identity events are handled separately) -2. Checks `record.collection` against the allowlist -3. Maps `record.action` to an operation: - - `create` → upsert document - - `update` → upsert document - - `delete` → tombstone document (`deleted_at = now`) -4. Decodes `record.record` into the collection-specific struct -5. Normalizes to internal `Document` -6. Upserts into the documents table -7. Schedules embedding job if eligible -8. Persists cursor (`event.id`) **only after successful DB commit** - -### Cursor Persistence Rules - -- If DB commit fails → cursor does not advance → event will be retried -- After successful DB writes, ack Tap first, then persist cursor for operator-visible resume -- If ack fails → cursor does not advance -- If ack succeeds but cursor persistence fails → retry cursor persistence until successful or process exit -- If normalization fails → log error, optionally dead-letter, skip → cursor advances -- If embedding scheduling fails → document remains keyword-searchable → cursor advances - -## 4. Backfill Behavior - -When a repo is added to Tap (via `/repos/add`, signal collection, or full network mode): - -1. Tap fetches full repo history from PDS via `com.atproto.sync.getRepo` -2. Firehose events for that repo are buffered during backfill -3. Historical events (`live: false`) are delivered first -4. After backfill completes, buffered live events drain -5. New firehose events stream normally (`live: true`) - -### Application-Level Backfill Support - -The indexer also supports: - -- Full reindex from existing corpus (re-normalize all stored documents) -- Targeted reindex by collection -- Targeted reindex by DID - -These do not involve Tap — they re-process documents already in the database. - -## 5. Normalization - -Normalization converts heterogeneous `sh.tangled.*` records into the common `Document` shape defined in [03-data-model.md](03-data-model.md). - -### Adapter Interface - -Each indexed collection provides an adapter: - -```go -type RecordAdapter interface { - Collection() string - RecordType() string - Normalize(event TapRecordEvent) (*Document, error) - Searchable(record map[string]any) bool -} -``` - -### Per-Collection Normalization - -#### sh.tangled.repo → `repo` - -| Document Field | Source | -| -------------- | -------------------------------- | -| `title` | `record.name` | -| `body` | `record.description` | -| `summary` | `record.description` (truncated) | -| `repo_name` | `record.name` | -| `repo_did` | `event.did` | -| `tags_json` | `json(record.topics)` | -| `created_at` | `record.createdAt` | - -**Searchable:** Always (unless empty name). - -#### sh.tangled.repo.issue → `issue` - -| Document Field | Source | -| -------------- | ------------------------------------------- | -| `title` | `record.title` | -| `body` | `record.body` | -| `summary` | First ~200 chars of `record.body` | -| `repo_did` | Extracted from `record.repo` AT-URI | -| `repo_name` | Resolved from repo AT-URI | -| `tags_json` | `[]` (labels resolved separately if needed) | -| `created_at` | `record.createdAt` | - -**Searchable:** Always. - -#### sh.tangled.repo.pull → `pull` - -| Document Field | Source | -| -------------- | ------------------------------------------ | -| `title` | `record.title` | -| `body` | `record.body` | -| `summary` | First ~200 chars of `record.body` | -| `repo_did` | Extracted from `record.target.repo` AT-URI | -| `repo_name` | Resolved from target repo AT-URI | -| `tags_json` | `[]` | -| `created_at` | `record.createdAt` | - -**Searchable:** Always. - -#### sh.tangled.string → `string` - -| Document Field | Source | -| -------------- | -------------------- | -| `title` | `record.filename` | -| `body` | `record.contents` | -| `summary` | `record.description` | -| `repo_name` | — | -| `repo_did` | — | -| `tags_json` | `[]` | -| `created_at` | `record.createdAt` | - -**Searchable:** Always (content is required). - -#### sh.tangled.actor.profile → `profile` - -| Document Field | Source | -| -------------- | ---------------------------------------------------- | -| `title` | Author handle (resolved from DID) | -| `body` | `record.description` | -| `summary` | `record.description` (truncated) + `record.location` | -| `repo_name` | — | -| `repo_did` | — | -| `tags_json` | `[]` | -| `created_at` | — (profiles don't have createdAt) | - -**Searchable:** If `description` is non-empty. - -#### sh.tangled.repo.issue.comment → `issue_comment` - -| Document Field | Source | -| -------------- | ----------------------------------------- | -| `title` | — (derived: "Comment on {issue title}") | -| `body` | `record.body` | -| `summary` | First ~200 chars of `record.body` | -| `repo_did` | Resolved from `record.issue` AT-URI chain | -| `repo_name` | Resolved | -| `created_at` | `record.createdAt` | - -**Searchable:** If body is non-empty. - -#### sh.tangled.repo.pull.comment → `pull_comment` - -Same pattern as issue comments, using `record.pull` instead of `record.issue`. - -### State Event Handling - -State and status records (`sh.tangled.repo.issue.state`, `sh.tangled.repo.pull.status`) do **not** produce new search documents. Instead, they update the `record_state` cache table (see [03-data-model.md](03-data-model.md)). - -### Interaction Event Handling - -Stars (`sh.tangled.feed.star`) and reactions (`sh.tangled.feed.reaction`) do not produce search documents. They may be aggregated for ranking signals in later phases. - -### Embedding Input Text - -For documents eligible for embedding, compose the input as: - -```sh -{title}\n{repo_name}\n{author_handle}\n{tags}\n{summary}\n{body} -``` - -Fields are joined with newlines. Empty fields are omitted. - -### Repo Name Resolution - -Issues, PRs, and comments reference their parent repo via AT-URI (e.g., `at://did:plc:abc/sh.tangled.repo/tid`). Resolving the repo name requires either: - -1. Looking up the repo document in the local `documents` table -2. Caching repo metadata in a lightweight lookup table - -Option 1 is preferred for v1. If the repo document hasn't been indexed yet, `repo_name` is left empty and backfilled on the next reindex pass. - -## 6. Identity Event Handling - -Identity events should be used to maintain an author handle cache: - -```sh -did → handle mapping -``` - -When an identity event arrives with a new handle, update `author_handle` on all documents with that DID. This ensures search by handle returns current results. - -## 7. Repo Management - -To add repos for tracking, POST to Tap's `/repos/add` endpoint: - -```bash -curl -u admin:PASSWORD -X POST https://tap-host/repos/add \ - -H "Content-Type: application/json" \ - -d '{"dids": ["did:plc:abc123", "did:plc:def456"]}' -``` - -Alternatively, use `TAP_SIGNAL_COLLECTION=sh.tangled.repo` to auto-track any repo that has Tangled repo records. - -## 8. Failure Handling - -### Ingestion Failures - -If Tap event processing fails before DB commit: - -- Log the failure with event ID, DID, collection, rkey, and error class -- Retry with exponential backoff (for transient errors like DB timeouts) -- Do **not** advance cursor — the event will be re-delivered by Tap -- After max retries for a persistent error, log and skip (cursor advances) - -### Normalization Failures - -If a record cannot be normalized: - -- Log collection, DID, rkey, CID, and error class -- Do not crash the process -- Skip the event and advance cursor -- Optionally insert into a `dead_letter` table for manual inspection - -### Embedding Failures - -If embedding generation fails: - -- The document remains keyword-searchable -- The embedding job is marked `failed` with `last_error` and incremented `attempts` -- Jobs are retried with exponential backoff up to a max attempt count -- After max attempts, the job enters `dead` state -- The embed-worker exposes failed job count as a metric -- If Ollama is unreachable (sidecar down), all pending jobs pause until connectivity is restored - -### DB Failures - -If Turso/libSQL is unreachable: - -- **API** returns `503` for search endpoints; `/healthz` still returns 200 (liveness), `/readyz` returns 503 -- **Indexer** pauses event processing and retries DB connection with backoff; cursor does not advance -- **Embed-worker** pauses job processing and retries - -### Tap Connection Failures - -If the WebSocket connection to Tap drops: - -- Reconnect with exponential backoff -- Resume from the last persisted cursor -- Log reconnection attempts and success - -Tap itself handles firehose reconnection independently — a Tap restart does not require indexer intervention beyond reconnecting the WebSocket. - -### Duplicate Event Handling - -Tap delivers events **at least once**. Duplicates are handled by: - -- Using `id = did|collection|rkey` as the primary key -- All writes are upserts (`INSERT OR REPLACE` / `ON CONFLICT ... DO UPDATE`) -- CID comparison can detect true no-ops (same content) vs. actual updates - -### Startup Recovery - -On indexer startup: - -1. Read `cursor` from `sync_state` table -2. Connect to Tap WebSocket -3. Tap replays events from the stored cursor position -4. Processing resumes normally - -If no cursor exists (first run), Tap delivers all historical events from backfill. diff --git a/docs/api/specs/05-search.md b/docs/api/specs/05-search.md deleted file mode 100644 index 6915416..0000000 --- a/docs/api/specs/05-search.md +++ /dev/null @@ -1,306 +0,0 @@ ---- -title: "Spec 05 — Search" -updated: 2026-03-22 ---- - -Covers all search modes, the public search API contract, scoring, and filtering. - -## 1. Search Modes - -| Mode | Backing | Available | -| ---------- | ------------------------------------ | --------- | -| `keyword` | Turso Tantivy-backed FTS | MVP | -| `semantic` | Vector similarity (DiskANN index) | Phase 2 | -| `hybrid` | Weighted merge of keyword + semantic | Phase 3 | - -## 2. Keyword Search - -### Implementation - -Uses Turso's `fts_score()` function for BM25 ranking: - -```sql -SELECT - d.id, d.title, d.summary, d.repo_name, d.author_handle, - d.collection, d.record_type, d.updated_at, - fts_score(d.title, d.body, d.summary, d.repo_name, d.author_handle, d.tags_json, ?) AS score -FROM documents d -WHERE fts_match(d.title, d.body, d.summary, d.repo_name, d.author_handle, d.tags_json, ?) - AND d.deleted_at IS NULL -ORDER BY score DESC -LIMIT ? OFFSET ?; -``` - -### Field Weights - -Configured in the FTS index definition: - -| Field | Weight | Rationale | -| --------------- | ------ | ------------------------------------ | -| `title` | 3.0 | Highest signal for relevance | -| `repo_name` | 2.5 | Exact repo lookups should rank first | -| `author_handle` | 2.0 | Author search is common | -| `summary` | 1.5 | More focused than body | -| `tags_json` | 1.2 | Topic matching | -| `body` | 1.0 | Baseline | - -### Query Features - -Tantivy query syntax is exposed to users: - -- Boolean: `go AND search`, `rust NOT unsafe` -- Phrase: `"pull request"` -- Prefix: `tang*` -- Field-specific: `title:parser` - -### Snippets - -Use `fts_highlight()` to generate highlighted snippets: - -```sql -fts_highlight(d.body, '', '', ?) AS body_snippet -``` - -### FTS Operational Notes - -- **Segment merging:** Turso FTS uses Tantivy's `NoMergePolicy`. Run `OPTIMIZE INDEX idx_documents_fts;` after bulk writes (backfill) and periodically in production to keep query performance stable. -- **Read-your-writes:** FTS queries within the same transaction see a pre-commit snapshot. If a document is written and immediately searched in the same transaction, FTS will not find it. The indexer and API are separate processes, so this is not a concern in normal operation. -- **Feature flag:** Turso FTS requires the `fts` feature flag to be enabled on the database. - -## 3. Semantic Search - -### Query Flow - -1. Convert user query text to embedding via Ollama (self-hosted) -2. Query `vector_top_k` for nearest neighbors -3. Join back to `documents` to get metadata -4. Filter out deleted/hidden documents -5. Return results with distance as score - -```sql -SELECT d.id, d.title, d.summary, d.repo_name, d.author_handle, - d.collection, d.record_type, d.updated_at -FROM vector_top_k('idx_embeddings_vec', vector32(?), ?) AS v -JOIN document_embeddings e ON e.rowid = v.id -JOIN documents d ON d.id = e.document_id -WHERE d.deleted_at IS NULL; -``` - -### Score Normalization - -Cosine distance ranges from 0 (identical) to 2 (opposite). Normalize to a 0–1 relevance score: - -```text -semantic_score = 1.0 - (distance / 2.0) -``` - -## 4. Hybrid Search - -### v1: Weighted Score Blending - -```text -hybrid_score = 0.65 * keyword_score_normalized + 0.35 * semantic_score_normalized -``` - -### Score Normalization for Blending - -Keyword (BM25) scores are unbounded. Normalize using min-max within the result set: - -```text -keyword_normalized = (score - min_score) / (max_score - min_score) -``` - -Semantic scores are already bounded after the distance-to-relevance conversion. - -### Merge Strategy - -1. Fetch top N keyword results (e.g., N=50) -2. Fetch top N semantic results -3. Merge on `document_id` -4. For documents appearing in both sets, combine scores -5. For documents in only one set, use that score (with 0 for the missing signal) -6. Sort by `hybrid_score` descending -7. Deduplicate -8. Apply limit/offset - -### v2: Reciprocal Rank Fusion (future) - -If keyword and semantic score scales prove unstable under weighted blending, replace with RRF: - -```text -rrf_score = Σ 1 / (k + rank_i) -``` - -where `k` is a constant (typically 60) and `rank_i` is the document's rank in each result list. - -## 5. Filtering - -All search modes support these filters, applied as SQL WHERE clauses: - -| Filter | Parameter | SQL | -| ----------- | ------------ | ------------------------------------------- | -| Collection | `collection` | `d.collection = ?` | -| Author | `author` | `d.author_handle = ?` or `d.did = ?` | -| Repo | `repo` | `d.repo_name = ?` or `d.repo_did = ?` | -| Record type | `type` | `d.record_type = ?` | -| Language | `language` | `d.language = ?` | -| Date range | `from`, `to` | `d.created_at >= ?` and `d.created_at <= ?` | -| State | `state` | Join to `record_state` table | - -## 6. Embedding Eligibility - -A document is eligible for embedding if: - -- `deleted_at IS NULL` -- `record_type` is one of: `repo`, `issue`, `pull`, `string`, `profile` -- At least one of `title`, `body`, or `summary` is non-empty -- Total text length exceeds a minimum threshold (e.g., 20 characters) - -## 7. API Endpoints - -### Health - -| Method | Path | Description | -| ------ | ---------- | -------------------------------- | -| GET | `/healthz` | Liveness — process is responsive | -| GET | `/readyz` | Readiness — DB is reachable | - -### Search - -| Method | Path | Description | -| ------ | ------------------ | ------------------------------------------------ | -| GET | `/search` | Search with configurable mode (default: keyword) | -| GET | `/search/keyword` | Keyword-only search | -| GET | `/search/semantic` | Semantic-only search | -| GET | `/search/hybrid` | Hybrid search | - -### Documents - -| Method | Path | Description | -| ------ | ----------------- | ----------------------------- | -| GET | `/documents/{id}` | Fetch a single document by ID | - -### Admin - -| Method | Path | Description | -| ------ | ---------------- | -------------------- | -| POST | `/admin/reindex` | Trigger reindex | -| POST | `/admin/reembed` | Trigger re-embedding | - -Admin endpoints are disabled by default. Enable with `ENABLE_ADMIN_ENDPOINTS=true`. - -## 8. Query Parameters - -| Parameter | Type | Default | Description | -| ------------ | ------ | --------- | -------------------------------------------------------------------- | -| `q` | string | required | Search query | -| `mode` | string | `keyword` | `keyword`, `semantic`, or `hybrid` | -| `limit` | int | 20 | Results per page (max: `SEARCH_MAX_LIMIT`) | -| `offset` | int | 0 | Pagination offset | -| `collection` | string | — | Filter by `sh.tangled.*` collection | -| `type` | string | — | Filter by record type (`repo`, `issue`, `pull`, `string`, `profile`) | -| `author` | string | — | Filter by author handle or DID | -| `repo` | string | — | Filter by repo name or repo DID | -| `language` | string | — | Filter by language | -| `from` | string | — | Created after (ISO 8601) | -| `to` | string | — | Created before (ISO 8601) | -| `state` | string | — | Filter by state (`open`, `closed`, `merged`) | - -## 9. Search Response - -```json -{ - "query": "rust markdown tui", - "mode": "hybrid", - "total": 142, - "limit": 20, - "offset": 0, - "results": [ - { - "id": "did:plc:abc|sh.tangled.repo|3kb3fge5lm32x", - "collection": "sh.tangled.repo", - "record_type": "repo", - "title": "glow-rs", - "body_snippet": "A TUI markdown viewer inspired by Glow...", - "summary": "Rust TUI markdown viewer", - "repo_name": "glow-rs", - "author_handle": "desertthunder.dev", - "score": 0.842, - "matched_by": ["keyword", "semantic"], - "created_at": "2026-03-20T10:00:00Z", - "updated_at": "2026-03-22T15:03:11Z" - } - ] -} -``` - -### Result Fields - -| Field | Type | Description | -| ------------------ | -------- | ------------------------------------------- | -| `id` | string | Document stable ID | -| `collection` | string | ATProto collection NSID | -| `record_type` | string | Normalized type label | -| `title` | string | Document title | -| `body_snippet` | string | Highlighted body excerpt | -| `summary` | string | Short description | -| `repo_name` | string | Repository name (if applicable) | -| `author_handle` | string | Author handle | -| `did` | string | Author DID when available | -| `at_uri` | string | Canonical AT URI when available | -| `primary_language` | string | Primary language for repo results | -| `stars` | number | Indexed star count for repo results | -| `follower_count` | number | Indexed follower count for profile results | -| `following_count` | number | Indexed following count for profile results | -| `score` | float | Relevance score (0–1) | -| `matched_by` | string[] | Which search modes produced this result | -| `created_at` | string | ISO 8601 creation timestamp | -| `updated_at` | string | ISO 8601 last update timestamp | - -## 10. Document Response - -`GET /documents/{id}` returns the full document: - -```json -{ - "id": "did:plc:abc|sh.tangled.repo|3kb3fge5lm32x", - "did": "did:plc:abc", - "collection": "sh.tangled.repo", - "rkey": "3kb3fge5lm32x", - "at_uri": "at://did:plc:abc/sh.tangled.repo/3kb3fge5lm32x", - "cid": "bafyreig...", - "record_type": "repo", - "title": "glow-rs", - "body": "A TUI markdown viewer inspired by Glow, written in Rust.", - "summary": "Rust TUI markdown viewer", - "repo_name": "glow-rs", - "author_handle": "desertthunder.dev", - "tags_json": "[\"rust\", \"tui\", \"markdown\"]", - "language": "en", - "created_at": "2026-03-20T10:00:00Z", - "updated_at": "2026-03-22T15:03:11Z", - "indexed_at": "2026-03-22T15:05:00Z", - "has_embedding": true -} -``` - -## 11. Error Responses - -| Status | Condition | -| ------ | ------------------------------------------------------------------ | -| 400 | Missing `q` parameter, invalid `limit`/`offset`, malformed filters | -| 404 | Document not found | -| 503 | DB unreachable (readiness failure) | - -```json -{ "error": "invalid_parameter", "message": "limit must be between 1 and 100" } -``` - -## 12. API Behavior - -- `keyword` returns only lexical matches via `fts_match`/`fts_score` -- `semantic` returns only embedding-backed matches via `vector_top_k` -- `hybrid` merges both result sets and reranks -- All modes exclude documents with `deleted_at IS NOT NULL` by default -- Pagination uses `limit`/`offset` (cursor-based pagination deferred) -- Mobile clients may use `type=repo` and `type=profile` to render repo/profile search directly diff --git a/docs/api/specs/06-operations.md b/docs/api/specs/06-operations.md deleted file mode 100644 index cfbab69..0000000 --- a/docs/api/specs/06-operations.md +++ /dev/null @@ -1,434 +0,0 @@ ---- -title: "Spec 06 — Operations" -updated: 2026-03-23 ---- - -Covers configuration, observability, security, and deployment. - -## 0. Quick Setup - -Tap is already deployed. For a new environment, the minimum operator work is: - -1. Create or choose a Turso database for that environment -2. Generate a Turso auth token for that database -3. Point `TURSO_DATABASE_URL` and `TURSO_AUTH_TOKEN` at that database -4. Create Railway services for `api` and `indexer` -5. Point `TAP_URL` at the existing Tap deployment -6. Run migrations/start the services -7. Run `twister backfill` before treating the environment as search-ready - -No separate `*_DEV` or `*_PROD` variables are required. Each environment keeps using the same variable names and simply points them at the appropriate Turso database. - -## 1. Configuration - -All configuration is via environment variables. - -### Required - -| Variable | Description | -| --------------------- | ----------------------------------------------------------- | -| `TAP_URL` | Tap WebSocket URL (e.g., `wss://tap.example.com/channel`) | -| `TAP_AUTH_PASSWORD` | Tap admin password for Basic auth (if set on Tap) | -| `TURSO_DATABASE_URL` | Turso connection URL (e.g., `libsql://db-name.turso.io`) | -| `TURSO_AUTH_TOKEN` | Turso JWT auth token | -| `INDEXED_COLLECTIONS` | Comma-separated list of `sh.tangled.*` collections to index | - -### Search - -| Variable | Default | Description | -| ---------------------- | --------- | ------------------------ | -| `SEARCH_DEFAULT_LIMIT` | `20` | Default results per page | -| `SEARCH_MAX_LIMIT` | `100` | Maximum results per page | -| `SEARCH_DEFAULT_MODE` | `keyword` | Default search mode | - -### Embedding (Ollama — self-hosted) - -| Variable | Default | Description | -| ---------------------- | ------------------------------------------ | ---------------------------------------------- | -| `OLLAMA_URL` | `http://ollama.railway.internal:11434` | Ollama server URL | -| `EMBEDDING_MODEL` | `nomic-embed-text` | Ollama model name | -| `EMBEDDING_DIM` | `768` | Vector dimensionality (must match model) | -| `EMBEDDING_BATCH_SIZE` | `32` | Documents per embedding batch | - -### Hybrid Search - -| Variable | Default | Description | -| ------------------------ | ------- | --------------------------------------- | -| `HYBRID_KEYWORD_WEIGHT` | `0.65` | Keyword score weight in hybrid ranking | -| `HYBRID_SEMANTIC_WEIGHT` | `0.35` | Semantic score weight in hybrid ranking | - -### Server - -| Variable | Default | Description | -| ------------------------ | ------- | ------------------------------------------- | -| `HTTP_BIND_ADDR` | `:8080` | API server bind address | -| `LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error` | -| `LOG_FORMAT` | `json` | Log format: `json` or `text` | -| `ENABLE_ADMIN_ENDPOINTS` | `false` | Enable `/admin/*` endpoints | -| `ADMIN_AUTH_TOKEN` | — | Bearer token for admin endpoints | - -### Example `.env` - -```bash -# Tap (deployed on Railway) -TAP_URL=wss://tap-instance.up.railway.app/channel -TAP_AUTH_PASSWORD=your-tap-admin-password - -# Turso -TURSO_DATABASE_URL=libsql://twister-db.turso.io -TURSO_AUTH_TOKEN=eyJhbGci... - -# Collections -INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,sh.tangled.string,sh.tangled.actor.profile,sh.tangled.repo.issue.comment,sh.tangled.repo.pull.comment,sh.tangled.repo.issue.state,sh.tangled.repo.pull.status,sh.tangled.feed.star - -# Search -SEARCH_DEFAULT_LIMIT=20 -SEARCH_MAX_LIMIT=100 - -# Embedding — Ollama (Phase 2) -# OLLAMA_URL=http://ollama.railway.internal:11434 -# EMBEDDING_MODEL=nomic-embed-text -# EMBEDDING_DIM=768 - -# Server -HTTP_BIND_ADDR=:8080 -LOG_LEVEL=info -ENABLE_ADMIN_ENDPOINTS=false -``` - -### Environment Selection - -Use the same variable names in every environment: - -- local development can point `TURSO_DATABASE_URL` and `TURSO_AUTH_TOKEN` at `twister-dev` -- production can point those same variables at `twister-prod` - -The application should not care which database it is talking to; only the environment wiring changes. - -## 1.5. Turso Setup - -### Recommended Databases - -Use one Turso database per environment, for example: - -- `twister-dev` -- `twister-prod` - -Keep the app config identical across environments and swap only these values: - -- `TURSO_DATABASE_URL` -- `TURSO_AUTH_TOKEN` - -### Basic Flow - -Using the Turso dashboard or CLI: - -1. Create the database for the target environment -2. Capture its libSQL URL -3. Create an auth token for the service -4. Set `TURSO_DATABASE_URL` and `TURSO_AUTH_TOKEN` in that environment - -Example values: - -```bash -# Development environment -TURSO_DATABASE_URL=libsql://twister-dev-your-org.turso.io -TURSO_AUTH_TOKEN=... - -# Production environment -TURSO_DATABASE_URL=libsql://twister-prod-your-org.turso.io -TURSO_AUTH_TOKEN=... -``` - -### Practical Rule - -Do not introduce `TURSO_DATABASE_URL_DEV`, `TURSO_DATABASE_URL_PROD`, or similar split variables. Railway environments, local shells, and CI should all set the same names with environment-specific values. - -## 1.6. Railway Setup - -### Project Layout - -Create or reuse one Railway project containing: - -- existing `tap` service -- `api` service running `twister api` -- `indexer` service running `twister indexer` - -### Basic Steps - -1. Connect the monorepo to Railway -2. Create the `api` and `indexer` services from the same source repo/Docker image -3. Set shared variables on both services: - - `TURSO_DATABASE_URL` - - `TURSO_AUTH_TOKEN` - - `LOG_LEVEL` - - `LOG_FORMAT` -4. Set API-specific variables: - - `HTTP_BIND_ADDR` - - `SEARCH_DEFAULT_LIMIT` - - `SEARCH_MAX_LIMIT` -5. Set indexer-specific variables: - - `TAP_URL` - - `TAP_AUTH_PASSWORD` - - `INDEXED_COLLECTIONS` -6. Configure health checks -7. Deploy -8. Run backfill against the environment before public validation - -### Dev vs Production on Railway - -If you use multiple Railway environments, keep the same service definitions and variable names in each one. Only the values change: - -- dev Railway environment -> `TURSO_DATABASE_URL=...twister-dev...` -- prod Railway environment -> `TURSO_DATABASE_URL=...twister-prod...` - -This keeps deployment logic simple and avoids conditional application config. - -## 2. Observability - -### Structured Logging - -Use Go's `slog` with JSON output. Every log entry includes: - -| Field | Description | -| --------- | ----------------------------------- | -| `ts` | Timestamp (RFC 3339) | -| `level` | Log level | -| `service` | `api`, `indexer`, or `embed-worker` | -| `msg` | Human-readable message | - -#### Context Fields (where applicable) - -| Field | When | -| ------------- | ------------------------ | -| `event_name` | Tap event processing | -| `event_id` | Tap event ID | -| `document_id` | Document operations | -| `did` | Any DID-scoped operation | -| `collection` | Record processing | -| `rkey` | Record processing | -| `cursor` | Cursor persistence | -| `error_class` | Error handling | -| `duration_ms` | Timed operations | - -### Metrics - -Recommended counters and gauges (via logs, Prometheus, or platform metrics): - -#### Ingestion - -| Metric | Type | Description | -| ------------------------------ | --------- | ---------------------------------- | -| `events_processed_total` | counter | Total Tap events processed | -| `events_failed_total` | counter | Events that failed processing | -| `normalization_failures_total` | counter | Normalization errors by collection | -| `upsert_duration_ms` | histogram | DB upsert latency | -| `cursor_position` | gauge | Current Tap cursor position | - -#### Embedding - -| Metric | Type | Description | -| -------------------------- | --------- | ------------------------------ | -| `embedding_queue_depth` | gauge | Pending embedding jobs | -| `embedding_failures_total` | counter | Failed embedding attempts | -| `embedding_duration_ms` | histogram | Per-document embedding latency | - -#### Search - -| Metric | Type | Description | -| ----------------------- | --------- | -------------------------- | -| `search_requests_total` | counter | Requests by mode | -| `search_duration_ms` | histogram | Query latency by mode | -| `search_results_count` | histogram | Results returned per query | - -### Health Checks - -#### API Process - -| Endpoint | Check | Healthy | -| -------------- | --------------------- | ------------------- | -| `GET /healthz` | Process is responsive | Always (liveness) | -| `GET /readyz` | DB connection works | `SELECT 1` succeeds | - -#### Indexer Process - -The indexer exposes a top-level health probe (not HTTP-routed): - -- Tap WebSocket connected or reconnecting -- Cursor advancing or intentionally idle -- DB reachable - -On Railway, this is a health check endpoint on a separate port (9090). - -#### Embed Worker - -- DB reachable -- Embedding provider reachable (periodic test call) -- Job queue not stalled (jobs processing within expected timeframe) - -## 3. Security - -### Secrets Management - -Secrets are injected through platform secret management: - -- **Railway:** Environment variables in the dashboard or `railway variables` - -Secrets are never stored in code, config files, or Docker images. - -Required secrets: - -| Secret | Purpose | -| ------------------- | --------------------------------- | -| `TURSO_AUTH_TOKEN` | Turso database authentication | -| `TAP_AUTH_PASSWORD` | Tap admin API authentication | -| `OLLAMA_URL` | Ollama sidecar connection (no secret if internal networking) | -| `ADMIN_AUTH_TOKEN` | Admin endpoint authentication | - -### Admin Endpoints - -Admin endpoints (`/admin/reindex`, `/admin/reembed`) are: - -- Disabled by default (`ENABLE_ADMIN_ENDPOINTS=false`) -- When enabled, protected by bearer token (`ADMIN_AUTH_TOKEN`) -- Alternatively, exposed only on internal networking (Railway private networking) - -### Input Validation - -The search API shall: - -- Validate `limit` is between 1 and `SEARCH_MAX_LIMIT` -- Validate `offset` is non-negative -- Reject unknown or malformed filter parameters with 400 -- Sanitize query strings before passing to FTS (Tantivy query parser handles this, but validate basic structure) -- Bound hybrid requests (limit concurrent vector searches) - -### Tap Authentication - -The indexer authenticates to Tap using HTTP Basic auth (`admin:`). The WebSocket upgrade request includes the auth header. - -### Data Privacy - -- All indexed content is public ATProto data -- No private or authenticated content is ingested -- Deleted records are tombstoned (`deleted_at` set) and excluded from search results -- Tombstoned documents are periodically purged (configurable retention) - -## 4. Deployment - -### Railway (Primary) - -All Twister services deploy as separate Railway services within the same project. Tap is already deployed here. - -#### Service Layout - -| Service | Start Command | Health Check | Public | -| ------------ | ---------------------- | ------------------ | ------ | -| tap | (already deployed) | `GET /health` | no | -| api | `twister api` | `GET /healthz` | yes | -| indexer | `twister indexer` | `GET :9090/health` | no | -| embed-worker | `twister embed-worker` | `GET :9091/health` | no | -| ollama | (Railway template) | `GET /api/tags` | no | - -All services share the same Docker image. Railway uses the start command to select the subcommand. - -#### Environment Variables - -Set per-service in the Railway dashboard or via `railway variables`: - -```bash -# Shared across services -TURSO_DATABASE_URL=libsql://twister-db.turso.io -TURSO_AUTH_TOKEN=eyJ... -LOG_LEVEL=info -LOG_FORMAT=json - -# API service -HTTP_BIND_ADDR=:8080 -SEARCH_DEFAULT_LIMIT=20 -SEARCH_MAX_LIMIT=100 -ENABLE_ADMIN_ENDPOINTS=false - -# Indexer service -TAP_URL=wss://${{tap.RAILWAY_PUBLIC_DOMAIN}}/channel # Railway service reference -TAP_AUTH_PASSWORD=... -INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,sh.tangled.string,sh.tangled.actor.profile - -# Embed-worker + Ollama (Phase 2) -# OLLAMA_URL=http://ollama.railway.internal:11434 -# EMBEDDING_MODEL=nomic-embed-text -``` - -Railway supports referencing other services' variables with `${{service.VAR}}` syntax, which is useful for linking the indexer to Tap's domain. - -#### First-Time Bootstrap Checklist - -After the first successful deploy of a new environment: - -1. Confirm API readiness on `/readyz` -2. Confirm indexer health and Tap connectivity -3. Run graph backfill with the environment's seed file -4. Wait for Tap historical sync to settle -5. Verify that search returns known historical repos/profiles - -#### Health Checks - -Railway activates deployments based on health check responses. Configure per-service: - -- **api:** HTTP health check on `/healthz` port 8080 -- **indexer:** HTTP health check on `/health` port 9090 -- **embed-worker:** HTTP health check on `/health` port 9091 - -#### Autodeploy - -Connect the GitHub repository for automatic deployments on push. Railway builds from the Dockerfile and uses the start command configured per service. - -#### Internal Networking - -Railway services within the same project can communicate over private networking using `service.railway.internal` hostnames. The indexer connects to Tap via this internal network when both are in the same project. - -### Dockerfile - -```dockerfile -FROM golang:1.24-alpine AS builder - -WORKDIR /app - -COPY go.mod go.sum ./ -RUN go mod download - -COPY . . - -RUN CGO_ENABLED=0 GOOS=linux go build \ - -ldflags="-s -w" \ - -o /app/twister \ - ./main.go - -FROM alpine:3.21 - -RUN apk add --no-cache ca-certificates tzdata - -COPY --from=builder /app/twister /usr/local/bin/twister - -EXPOSE 8080 9090 9091 - -CMD ["twister", "api"] -``` - -Notes: - -- `CGO_ENABLED=0` for static binary (required if using `libsql-client-go`; not compatible with `go-libsql` which needs CGo) -- Railway overrides `CMD` with the start command configured per service -- Multiple ports exposed: 8080 (API), 9090 (indexer health), 9091 (embed-worker health) - -### Graceful Shutdown - -All processes handle `SIGTERM` and `SIGINT`: - -1. Stop accepting new requests/events -2. Drain in-flight work (with timeout) -3. Persist current cursor (indexer) -4. Close DB connections -5. Exit 0 - -Railway sends `SIGTERM` during deployments and restarts. diff --git a/docs/api/specs/07-graph-backfill.md b/docs/api/specs/07-graph-backfill.md deleted file mode 100644 index 0ad2285..0000000 --- a/docs/api/specs/07-graph-backfill.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -title: "Spec 07 — Graph Backfill" -updated: 2026-03-22 ---- - -## 1. Purpose - -Bootstrap the search index with existing Tangled content by discovering users from a seed set and triggering Tap backfill for their repositories. Without this, the index only captures new events after deployment. - -## 2. Seed Set - -A manually curated list of known Tangled users (DIDs or handles), stored in a plain text file: - -```text -# Known active Tangled users -did:plc:abc123 -did:plc:def456 -alice.tangled.sh -bob.tangled.sh -# Add more as discovered -``` - -Format: - -- One entry per line -- Lines starting with `#` are comments -- Blank lines are ignored -- Entries can be DIDs (`did:plc:...`) or handles (`alice.tangled.sh`) -- Handles are resolved to DIDs before processing - -## 3. Fan-Out Strategy - -From each seed user, discover connected users to expand the crawl set: - -### Discovery Sources - -1. **Follows**: Fetch `sh.tangled.graph.follow` records for the user → extract `subject` DIDs -2. **Collaborators**: For repos owned by the user, identify other users who have created issues, PRs, or comments → extract their DIDs - -### Depth Limit - -Fan-out is configurable with a max hops parameter (default: 2): - -- **Hop 0**: Seed users themselves -- **Hop 1**: Direct follows and collaborators of seed users -- **Hop 2**: Follows and collaborators of hop-1 users - -Higher hop counts discover more users but increase time and may pull in loosely related accounts. Start with 2 hops and adjust based on the size of the Tangled network. - -### Crawl Queue - -Discovered DIDs are added to a queue, deduplicated by DID. Each entry tracks: - -- DID -- Discovery hop (distance from seed) -- Source (which seed/user led to discovery) - -## 4. Backfill Mechanism - -For each discovered user: - -1. **Check Tap status**: Query Tap's `/info/:did` endpoint and classify by status: - - tracked + backfilled: skip - - tracked + backfilling/in-progress: skip and let current backfill finish - - untracked or tracked-without-backfill-state: submit to `/repos/add` -2. **Register with Tap**: POST to `/repos/add` with the DID — Tap handles the actual repo export and event delivery -3. **Tap backfill flow**: Tap fetches full repo history from PDS via `com.atproto.sync.getRepo`, then delivers historical events (`live: false`) through the normal WebSocket channel -4. **Indexer processes normally**: The indexer's existing ingestion loop handles backfill events the same as live events — no special backfill code path needed - -### Rate Limiting - -- Batch `/repos/add` calls (e.g., 10 DIDs per request) -- Add configurable delay between batches to avoid overwhelming Tap -- Respect Tap's processing capacity — monitor `/stats/repo-count` to track progress - -## 5. Deduplication - -- **User-level**: Maintain a visited set of DIDs during fan-out; skip already-seen DIDs -- **Tap-level**: Tap's `/repos/add` is idempotent — adding an already-tracked DID is a no-op -- **Record-level**: The indexer's upsert logic (keyed on `did|collection|rkey`) handles duplicate events naturally - -## 6. CLI Interface - -```bash -# Basic backfill from seed file -twister backfill --seeds seeds.txt - -# Limit fan-out depth -twister backfill --seeds seeds.txt --max-hops 1 - -# Preview discovered users without triggering backfill -twister backfill --seeds seeds.txt --dry-run - -# Control parallelism -twister backfill --seeds seeds.txt --concurrency 5 -``` - -### Flags - -| Flag | Default | Description | -| --------------- | -------- | ----------------------------------------------- | -| `--seeds` | required | Seed source: file path or comma-separated list | -| `--max-hops` | `2` | Max fan-out depth from seed users | -| `--dry-run` | `false` | List discovered users without submitting to Tap | -| `--concurrency` | `5` | Parallel discovery workers | -| `--batch-size` | `10` | DIDs per `/repos/add` call | -| `--batch-delay` | `1s` | Delay between batches | - -### Output - -Progress is logged to stdout: - -```text -[hop 0] Processing 5 seed users... -[hop 0] did:plc:abc123 → 12 follows, 3 collaborators -[hop 0] did:plc:def456 → 8 follows, 1 collaborator -[hop 1] Processing 24 discovered users (18 new)... -... -[done] Discovered 142 unique users across 2 hops -[done] Submitted 98 new DIDs to Tap (44 already tracked) -``` - -## 7. Idempotency - -The entire backfill process is safe to re-run: - -- Seed file parsing is stateless -- Fan-out discovery is deterministic for a given network state -- Tap's `/repos/add` is idempotent -- The indexer's upsert logic handles re-delivered events -- No local state is persisted between runs (the crawl queue is in-memory) - -## 8. Configuration - -| Variable | Default | Description | -| -------------------- | ---------- | ----------------------------- | -| `TAP_URL` | (existing) | Tap base URL for API calls | -| `TAP_AUTH_PASSWORD` | (existing) | Tap admin auth | -| `TURSO_DATABASE_URL` | (existing) | For checking existing records | -| `TURSO_AUTH_TOKEN` | (existing) | DB auth | - -No new environment variables are needed — backfill reuses existing Tap and DB configuration. diff --git a/docs/api/specs/08-app-integration.md b/docs/api/specs/08-app-integration.md deleted file mode 100644 index 60049b4..0000000 --- a/docs/api/specs/08-app-integration.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: "Spec 08 — App Integration" -updated: 2026-03-23 ---- - -## 1. Purpose - -Define the mobile-facing Twister API surface. - -The Twisted app should keep using Tangled's public knot and PDS APIs for canonical repo/profile detail. Twister is responsible for: - -- cross-network discovery via search -- index-backed summaries for data gaps such as followers - -## 2. Client Boundary - -The mobile client uses Twister only for: - -- Explore search -- index-backed profile summaries -- future feed and notification features - -The mobile client does not use Twister for: - -- repo tree/blob/detail reads -- direct profile record reads -- issue/PR detail reads - -Those remain on Tangled's public APIs. - -## 3. Search Contract - -`GET /search` - -Required query parameters: - -- `q` - -Optional query parameters: - -- `mode=keyword|semantic|hybrid` -- `type=repo|profile` -- `limit` -- `offset` - -For mobile clients, repo and profile results should include: - -- `did` -- `at_uri` -- `record_type` -- `title` -- `summary` -- `repo_name` -- `author_handle` -- `updated_at` -- `primary_language` for repos when known -- `stars` for repos when known -- `follower_count` and `following_count` for profiles when known - -## 4. Profile Summary Contract - -`GET /profiles/{did}/summary` - -Response: - -```json -{ - "did": "did:plc:abc123", - "handle": "desertthunder.dev", - "follower_count": 128, - "following_count": 84, - "indexed_at": "2026-03-23T10:15:00Z" -} -``` - -This endpoint exists because follower counts and follower lists are derived from indexed graph state, not from a single direct public Tangled API call. - -## 5. Failure Handling - -If Twister is unavailable: - -- the app should keep direct known-handle browsing working -- Explore should show a clear "index unavailable" state -- profile pages should omit index-backed follower counts rather than fail entirely - -## 6. Ownership - -- Twister owns search ranking, document normalization, and graph summary derivation -- The app owns result presentation, route transitions, and fallback behavior diff --git a/docs/api/specs/09-search-site.md b/docs/api/specs/09-search-site.md deleted file mode 100644 index 4a9b71d..0000000 --- a/docs/api/specs/09-search-site.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -title: "Spec 09 — Search Site" -updated: 2026-03-23 ---- - -A minimal static site that serves as both the public Twister API documentation and a live search showcase. Dark mode only, no framework or build step. - -## 1. Purpose - -- Give developers a browsable reference for the Twister search API -- Give anyone a way to try search against live indexed Tangled content -- Provide a shareable public URL before the mobile app ships - -## 2. Scope - -In scope: - -- Static HTML/CSS/JS (Alpine.js, no bundler) -- API reference pages generated from the spec docs -- Live search input wired to `GET /search` -- Result rendering with type-aware cards (repo, issue, PR, profile, string) -- Filter controls for collection, type, author, language, state -- Pagination -- Responsive layout (mobile-friendly, single breakpoint) - -Out of scope: - -- Auth, OAuth, or any write operations -- Semantic or hybrid mode toggle (keyword only for MVP) -- Server-side rendering or static-site generator -- Analytics or telemetry - -## 3. Pages - -| Route | Content | -| ----------------- | --------------------------------------------------------------------------- | -| `/` | Search input + results (the homepage is the search page) | -| `/docs` | API overview: base URL, auth (none for public), rate limits, response shape | -| `/docs/search` | `GET /search` — parameters, filters, response contract, examples | -| `/docs/documents` | `GET /documents/{id}` — request/response, examples | -| `/docs/health` | `GET /healthz`, `GET /readyz` — purpose and expected responses | - -## 4. Search Page Behavior - -1. Text input with a submit button. No debounce search-as-you-type for MVP. -2. On submit, fetch `GET {API_BASE}/search?q={query}&limit=20` (plus any active filters). -3. Render results as a vertical list of cards. -4. Each card shows: `record_type` badge, `title`, `body_snippet` (with `` highlights preserved), `author_handle`, `repo_name` (when present), `updated_at` relative time. -5. Clicking a result opens the canonical Tangled URL (`https://tangled.org/{handle}/{repo}` for repos, etc.) in a new tab. -6. "Load more" button appends the next page (`offset += limit`). -7. Empty state: "No results" message. -8. Error state: inline message if the API is unreachable. -9. Filter bar above results: dropdowns/inputs for `type`, `language`, `author`. Filters are query params so URLs are shareable. - -## 5. API Docs Pages - -Hand-written HTML mirroring the contracts in spec 05 (search) and spec 08 (app integration). Each page includes: - -- Endpoint signature (method, path) -- Parameter table (name, type, default, description) -- Example request (curl) -- Example response (JSON block with syntax highlighting via `
`)
-
-No generated docs tooling. The pages are static and updated manually when the API changes.
-
-## 6. Styling
-
-Minimal CSS, no utility framework.
-
-### Tokens
-
-```css
-:root {
-  --bg: #0e0e0e;
-  --surface: #1a1a1a;
-  --border: #2a2a2a;
-  --text: #e0e0e0;
-  --text-dim: #888;
-  --accent: #7aa2f7;
-  --mark-bg: #7aa2f733;
-  --mono: "Google Sans Mono", monospace;
-  --sans: "Google Sans", sans-serif;
-  --radius: 6px;
-}
-```
-
-### Rules
-
-- Dark theming.
-- `Google Sans` for body text. `Google Sans Mono` for code, JSON, and badges.
-- Fonts loaded via Google Fonts ``. System fallbacks: `sans-serif`, `monospace`.
-- Max content width: `720px`, centered.
-- Cards: `var(--surface)` background, `var(--border)` border, `var(--radius)` corners.
-- `` tags in snippets styled with `var(--mark-bg)` background and `var(--accent)` text.
-- Code blocks: `var(--surface)` background, horizontal scroll, no wrapping.
-- Links: `var(--accent)`, no underline, underline on hover.
-- Inputs and buttons: `var(--surface)` background, `var(--border)` border, `var(--text)` text.
-- One breakpoint at `640px` for mobile: full-width cards, stacked filter bar.
-
-## 7. Package Design
-
-The site lives in `internal/view/` as a self-contained Go package. It owns the templates, static assets, and HTTP handlers. The `api` package mounts `view.Handler()` into its router — nothing else leaks out.
-
-### Exports
-
-The package exposes a single constructor:
-
-```go
-// Handler returns an http.Handler that serves the site pages and static assets.
-func Handler() http.Handler
-```
-
-The `api` package calls `view.Handler()` and mounts it as a fallback after API routes.
-
-### Package Structure
-
-```text
-internal/view/
-  view.go               # Handler(), route setup, embed directives
-  templates/
-    layout.html         # Shared shell (head, nav, footer)
-    index.html          # Search page
-    docs/
-      index.html        # API overview
-      search.html       # GET /search docs
-      documents.html    # GET /documents/{id} docs
-      health.html       # Health endpoints docs
-  static/
-    style.css           # All styles, single file
-    search.js           # Search fetch, render, pagination, filters
-```
-
-### Embedding
-
-`view.go` uses `//go:embed` to bundle `templates/` and `static/`. Templates are parsed once at init. Static assets are served under `/static/` via `http.FileServer`.
-
-### Routing
-
-`view.Handler()` returns a mux that handles:
-
-| Pattern | Handler |
-| --- | --- |
-| `GET /` | Render `index.html` |
-| `GET /docs` | Render `docs/index.html` |
-| `GET /docs/search` | Render `docs/search.html` |
-| `GET /docs/documents` | Render `docs/documents.html` |
-| `GET /docs/health` | Render `docs/health.html` |
-| `GET /static/*` | Serve embedded CSS/JS files |
-
-## 9. Configuration
-
-Since the site is served by the same origin as the API, search requests use relative paths (`/search?q=...`). No `API_BASE` config needed — the browser's origin is the API.
-
-## 10. Local Development
-
-Run `twister api` locally. The site is served at `http://localhost:8080/` alongside the API endpoints. No separate dev server or file server required.
-
-The API docs pages render without any indexed data. The search page needs a running indexer and populated database to return results.
-
-## 11. Constraints
-
-- No dependencies besides Alpine via CDN.
-- Total site weight target: under 50 KB excluding fonts.
-- Works in modern browsers (last 2 versions of Chrome, Firefox, Safari).
-- All fetch calls include error handling for network failures and non-200 responses.
-- No CORS concerns — the site and API share an origin.
diff --git a/docs/api/specs/README.md b/docs/api/specs/README.md
deleted file mode 100644
index ea8546b..0000000
--- a/docs/api/specs/README.md
+++ /dev/null
@@ -1,23 +0,0 @@
----
-title: "Twister — Technical Specification Index"
-updated: 2026-03-22
----
-
-# Twister Technical Specifications
-
-Twister is a Go-based index and search service for [Tangled](https://tangled.org) content on AT Protocol.
-It ingests records through [Tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap), denormalizes them into search documents and graph summaries, indexes them in [Turso/libSQL](https://docs.turso.tech), and exposes public APIs for search and index-backed data gaps.
-
-## Specifications
-
-| #   | Document                                   | Description                                                     |
-| --- | ------------------------------------------ | --------------------------------------------------------------- |
-| 1   | [Architecture](01-architecture.md)         | Purpose, goals, design principles, system context, tech choices |
-| 2   | [Tangled Lexicons](02-tangled-lexicons.md) | `sh.tangled.*` record schemas and fields                        |
-| 3   | [Data Model](03-data-model.md)             | Database schema, search documents, sync state                   |
-| 4   | [Data Pipeline](04-data-pipeline.md)       | Tap integration, normalization, failure handling                |
-| 5   | [Search](05-search.md)                     | Search modes, API contract, scoring, filtering                  |
-| 6   | [Operations](06-operations.md)             | Configuration, observability, security, deployment              |
-| 7   | [Graph Backfill](07-graph-backfill.md)     | Seed-based user discovery and content backfill                  |
-| 8   | [App Integration](08-app-integration.md)   | Mobile-facing contracts for search and graph summaries          |
-| 9   | [Search Site](09-search-site.md)           | Static site for API docs and live search                        |
diff --git a/docs/api/tasks/README.md b/docs/api/tasks/README.md
deleted file mode 100644
index c0de2e5..0000000
--- a/docs/api/tasks/README.md
+++ /dev/null
@@ -1,41 +0,0 @@
----
-title: "Twister — Task Index"
-updated: 2026-03-22
----
-
-# Twister Tasks
-
-Assumes Go, Tap (deployed on Railway), Turso/libSQL, and Railway for deployment.
-
-## Delivery Strategy
-
-Build in four phases:
-
-1. **MVP** — ingestion, graph backfill, keyword search, deployment, operational tooling
-2. **Semantic Search** — embeddings, vector retrieval
-3. **Hybrid Search** — weighted merge of keyword + semantic
-4. **Quality Polish** — ranking refinement, advanced filters, analytics
-
-Ship keyword search before embeddings. That gives a testable, inspectable baseline before introducing model behavior.
-Within MVP, run graph backfill before calling the environment search-ready for users.
-
-## Phases
-
-| Phase | Title           | Document                                   | Status                                                                |
-| ----- | --------------- | ------------------------------------------ | --------------------------------------------------------------------- |
-| 1     | MVP             | [phase-1-mvp.md](phase-1-mvp.md)           | In progress (M0–M2 complete; backfill scheduled before public launch) |
-| 2     | Semantic Search | [phase-2-semantic.md](phase-2-semantic.md) | Not started                                                           |
-| 3     | Hybrid Search   | [phase-3-hybrid.md](phase-3-hybrid.md)     | Not started                                                           |
-| 4     | Quality Polish  | [phase-4-quality.md](phase-4-quality.md)   | Not started                                                           |
-
-## MVP Complete When
-
-- Tap ingests tracked `sh.tangled.*` records
-- Documents normalize into a stable store
-- Keyword search works publicly
-- Index-backed profile summaries can fill public API gaps such as followers
-- API and indexer are deployed on Railway
-- Restart does not lose sync position
-- Reindex exists for repair
-- Graph backfill populates initial content from seed users
-- A static search site with API docs is publicly accessible
diff --git a/docs/api/tasks/phase-1-mvp.md b/docs/api/tasks/phase-1-mvp.md
deleted file mode 100644
index 4c8a650..0000000
--- a/docs/api/tasks/phase-1-mvp.md
+++ /dev/null
@@ -1,388 +0,0 @@
----
-title: "Phase 1 — MVP"
-updated: 2026-03-22
----
-
-# Phase 1 — MVP
-
-Get a searchable product online: ingestion, keyword search, deployment, and operational tooling.
-
-## MVP Complete When
-
-- Tap ingests tracked `sh.tangled.*` records
-- Documents normalize into a stable store
-- Keyword search works publicly
-- API and indexer are deployed on Railway
-- Restart does not lose sync position
-- Reindex exists for repair
-- Graph backfill populates initial content from seed users
-- A static search site with API docs is publicly accessible
-
-## M0 — Repository Bootstrap ✅
-
-Executable layout, local tooling, and development conventions (completed 2026-03-22).
-
-## M1 — Database Schema and Store Layer ✅
-
-refs: [specs/03-data-model.md](../specs/03-data-model.md)
-
-Implemented the Turso/libSQL schema and Go store package for document persistence.
-
-## M2 — Normalization Layer ✅
-
-refs: [specs/02-tangled-lexicons.md](../specs/02-tangled-lexicons.md), [specs/04-data-pipeline.md](../specs/04-data-pipeline.md)
-
-Translate `sh.tangled.*` records into internal search documents.
-
-## M3 — Tap Client and Ingestion Loop
-
-refs: [specs/04-data-pipeline.md](../specs/04-data-pipeline.md), [specs/01-architecture.md](../specs/01-architecture.md)
-
-### Goal
-
-Connect the indexer to Tap (on Railway) and process live events into the store.
-
-### Deliverables
-
-- Tap WebSocket client package (`internal/tapclient/`)
-- Event decode layer (record events + identity events)
-- Ingestion loop with retry/backoff
-- Cursor persistence coupled to successful DB commits
-- Identity event handler (DID → handle cache)
-
-### Tasks
-
-- [x] Define Tap event DTOs matching the documented event shape:
-
-  ```go
-  type TapEvent struct {
-      ID       int64           `json:"id"`
-      Type     string          `json:"type"`     // "record" or "identity"
-      Record   *TapRecord      `json:"record"`
-      Identity *TapIdentity    `json:"identity"`
-  }
-  type TapRecord struct {
-      Live       bool            `json:"live"`
-      Rev        string          `json:"rev"`
-      DID        string          `json:"did"`
-      Collection string          `json:"collection"`
-      RKey       string          `json:"rkey"`
-      Action     string          `json:"action"`  // "create", "update", "delete"
-      CID        string          `json:"cid"`
-      Record     json.RawMessage `json:"record"`
-  }
-  type TapIdentity struct {
-      DID      string `json:"did"`
-      Handle   string `json:"handle"`
-      IsActive bool   `json:"isActive"`
-      Status   string `json:"status"`
-  }
-  ```
-
-- [x] Implement WebSocket client:
-  - Connect to `TAP_URL` (e.g., `wss://tap.railway.internal/channel`)
-  - HTTP Basic auth with `admin:TAP_AUTH_PASSWORD`
-  - Auto-reconnect with exponential backoff
-  - Ack protocol: send event `id` back after successful processing
-- [x] Implement ingestion loop:
-  1. Receive event from WebSocket
-  2. If `type == "identity"` → update handle cache, ack, continue
-  3. If `type == "record"` → check collection allowlist
-  4. Map `action` to operation (create/update → upsert, delete → tombstone)
-  5. Decode `record.record` via adapter registry
-  6. Normalize to `Document`
-  7. Upsert to store
-  8. Schedule embedding job if eligible ([Phase 2](phase-2-semantic.md))
-  9. Persist cursor (event ID) after successful DB commit
-  10. Ack the event
-- [x] Implement collection allowlist from `INDEXED_COLLECTIONS` config
-- [x] Handle state events (`sh.tangled.repo.issue.state`, `sh.tangled.repo.pull.status`) → update `record_state`
-- [x] Handle normalization failures: log, skip, advance cursor
-- [x] Handle DB failures: retry with backoff, do not advance cursor
-
-### Exit Criteria
-
-The system continuously ingests and persists `sh.tangled.*` records from Tap.
-
-## M4 — Graph Backfill from Seed Users
-
-refs: [specs/07-graph-backfill.md](../specs/07-graph-backfill.md)
-
-### Goal
-
-Bootstrap the index with historical Tangled content by discovering and backfilling users from a curated seed set.
-
-### Deliverables
-
-- `twister backfill` CLI command
-- Seed file parser and documented seed-file format
-- Graph fan-out discovery (follows and collaborators)
-- Tap `/repos/add` integration for discovered users
-- Deduplication against already-tracked repos
-- Dry-run mode and progress logging
-- Basic operator runbook for first bootstrap and repeat runs
-
-### Tasks
-
-- [x] Implement `backfill` subcommand with flags:
-  - `--seeds ` — required seed file path
-  - `--max-hops ` — depth limit for fan-out (default: 2)
-  - `--dry-run` — print the discovery plan without mutating Tap
-  - `--concurrency ` — parallel discovery workers (default: 5)
-  - `--batch-size ` — DIDs per `/repos/add` request
-  - `--batch-delay ` — delay between Tap registration batches
-- [x] Implement seed file parsing:
-  - One DID or handle per line
-  - `#` comments allowed
-  - Blank lines ignored
-  - Handles resolved to DIDs before graph expansion
-- [x] Decide and document the initial seed file location for operators:
-  - Repository-managed example file for format/reference
-  - Deployment-specific runtime file or mounted secret for real runs
-  - Implemented: `docs/api/seeds.txt` and `packages/api/internal/backfill/doc.go`
-- [x] Implement graph discovery:
-  1. Start from hop-0 seed users
-  2. Fetch `sh.tangled.graph.follow` records and collect subject DIDs
-  3. Fetch repo collaborators by inspecting repos, issues, PRs, and comments
-  4. Enqueue newly discovered DIDs with hop metadata
-  5. Stop expanding beyond `max-hops`
-- [x] Track discovery metadata for logs:
-  - source DID
-  - hop depth
-  - discovery reason (`seed`, `follow`, `collaborator`)
-- [x] Integrate with Tap admin endpoints:
-  - `GET /info/:did` to skip already-tracked repos when practical
-  - `POST /repos/add` to register new DIDs for backfill
-- [x] Make the command safe to re-run:
-  - in-memory visited DID set during crawl
-  - tolerate duplicate `/repos/add`
-  - rely on index upsert idempotency for re-delivered records
-- [x] Add operator-friendly logging:
-  - seed count
-  - users discovered per hop
-  - already-tracked vs newly-submitted DIDs
-  - batch progress
-  - final totals
-- [x] Add a short runbook covering:
-  - first bootstrap against an empty database
-  - repeat run after expanding the seed list
-  - dry-run before production mutation
-  - Implemented: `packages/api/internal/backfill/doc.go`
-
-### Exit Criteria
-
-Operators can bootstrap an empty environment to a usable historical baseline before public rollout.
-
-## M5 — Keyword Search API
-
-refs: [specs/05-search.md](../specs/05-search.md)
-
-### Goal
-
-Expose a usable public search API backed by Turso's Tantivy-backed FTS.
-
-### Deliverables
-
-- HTTP server (net/http)
-- `GET /healthz` — liveness
-- `GET /readyz` — readiness (DB connectivity)
-- `GET /search` — keyword search with configurable mode
-- `GET /search/keyword` — keyword-only search
-- `GET /documents/{id}` — document lookup
-- Search repository layer (FTS queries isolated from handlers)
-- Pagination, filtering, snippets
-
-### Tasks
-
-- [x] Set up HTTP server with net/http router
-- [x] Implement `/healthz` (always 200) and `/readyz` (SELECT 1 against DB)
-- [x] Implement search repository with FTS queries:
-
-  ```sql
-  SELECT id, title, summary, repo_name, author_handle, collection, record_type,
-         created_at, updated_at,
-         fts_score(title, body, summary, repo_name, author_handle, tags_json, ?) AS score,
-         fts_highlight(body, '', '', ?) AS body_snippet
-  FROM documents
-  WHERE fts_match(title, body, summary, repo_name, author_handle, tags_json, ?)
-    AND deleted_at IS NULL
-  ORDER BY score DESC
-  LIMIT ? OFFSET ?;
-  ```
-
-- [x] Implement request validation:
-  - `q` required, non-empty
-  - `limit` 1–100, default 20
-  - `offset` >= 0, default 0
-  - Reject unknown parameters with 400
-- [x] Implement filters (as WHERE clauses):
-  - `collection` → `d.collection = ?`
-  - `type` → `d.record_type = ?`
-  - `author` → `d.author_handle = ?` or `d.did = ?`
-  - `repo` → `d.repo_name = ?`
-- [x] Implement `/documents/{id}` — full document response
-- [x] Implement stable JSON response contract (see spec 05-search.md)
-- [x] Exclude tombstoned documents (`deleted_at IS NOT NULL`) by default
-- [x] Add request logging middleware (method, path, status, duration)
-- [x] Add CORS headers if needed
-
-### Exit Criteria
-
-A user can search Tangled content reliably with keyword search.
-
-## M5a — Search Site ✅
-
-refs: [specs/09-search-site.md](../specs/09-search-site.md)
-
-### Goal
-
-Ship a static site that doubles as public API documentation and a live search demo. Alpine.js via CDN for reactivity, no build step.
-
-### Deliverables
-
-- `internal/view/` package exporting `Handler() http.Handler`
-- Embedded templates (`templates/`) and static assets (`static/`) via `//go:embed`
-- Search page (`/`) wired to `GET /search` with result cards, filters, and pagination
-- API docs pages (`/docs/*`) covering search, documents, and health endpoints
-- Dark-mode-only styling with Google Sans fonts and minimal CSS tokens
-
-### Tasks
-
-- [x] Create `internal/view/` package with `view.go`, `templates/`, and `static/` directories
-- [x] Implement `Handler()` that returns an `http.Handler` with routes for all pages and `/static/*`
-- [x] Embed templates and static assets via `//go:embed`; parse templates once at init
-- [x] Use a shared `layout.html` template for the shell (head, nav, footer)
-- [x] Mount `view.Handler()` in the `api` package router as a fallback after API routes
-- [x] Build search page:
-  - Text input + submit
-  - Fetch `GET /search` with relative path (same origin)
-  - Render result cards with type badge, title, snippet (preserve ``), author, repo, relative time
-  - "Load more" pagination via offset
-  - Filter bar: type, language, author (reflected in URL query params)
-  - Empty and error states
-- [x] Build API docs pages:
-  - `/docs` — overview (base URL, response shape, no auth)
-  - `/docs/search` — `GET /search` params, filters, example curl, example response
-  - `/docs/documents` — `GET /documents/{id}` request/response
-  - `/docs/health` — `GET /healthz`, `GET /readyz`
-- [x] Implement `style.css` with design tokens (`--bg`, `--surface`, `--border`, `--accent`, etc.)
-- [x] Load Google Sans and Google Sans Mono via Google Fonts ``
-- [x] Result card links open canonical Tangled URLs in new tab
-- [x] Verify total site weight under 50 KB (excluding fonts and Alpine CDN) — 21 KB total
-
-### Exit Criteria
-
-A user can search Tangled content and read API docs from a public URL without installing anything.
-
-## M6 — Railway Deployment ✅
-
-refs: [specs/06-operations.md](../specs/06-operations.md), [deploy.md](../deploy.md)
-
-### Goal
-
-Deploy the API and indexer as Railway services alongside Tap.
-
-### Deliverables
-
-- Finalized Dockerfile
-- Railway project with services: `api`, `indexer`
-- Health checks configured per service
-- Secrets/env vars set
-- Production startup commands documented
-
-### Tasks
-
-- [x] Finalize Dockerfile (multi-stage, CGO_ENABLED=0, Alpine runtime)
-- [x] Create Railway services:
-  - `api` — start command: `twister api`
-  - `indexer` — start command: `twister indexer`
-- [x] Configure environment variables per service:
-  - Shared: `TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN`, `LOG_LEVEL`, `LOG_FORMAT`
-  - API: `HTTP_BIND_ADDR`, `SEARCH_DEFAULT_LIMIT`, `SEARCH_MAX_LIMIT`
-  - Indexer: `TAP_URL` (reference Tap service domain), `TAP_AUTH_PASSWORD`, `INDEXED_COLLECTIONS`
-- [x] Configure health checks:
-  - API: HTTP check on `/healthz` port 8080
-  - Indexer: HTTP check on `/health` port 9090
-- [x] Use Railway internal networking for indexer → Tap connection
-- [x] Connect GitHub repo for autodeploy
-- [x] Test graceful shutdown on redeploy (SIGTERM handling)
-- [x] Document deploy steps
-
-### Exit Criteria
-
-The system runs as a deployed service with health-checked processes on Railway.
-
-## M7 — Reindex and Repair ✅
-
-refs: [specs/05-search.md](../specs/05-search.md)
-
-### Goal
-
-Make the system recoverable and operable with repair tools.
-
-### Deliverables
-
-- `twister reindex` command with scoping options
-- Dry-run mode
-- Admin reindex endpoint
-- Progress logging and error summary
-
-### Tasks
-
-- [x] Implement `reindex` subcommand with flags:
-  - `--collection` — reindex one collection
-  - `--did` — reindex one DID's documents
-  - `--document` — reindex one document by ID
-  - `--dry-run` — show intended work without writes
-  - No flags → reindex all
-- [x] Implement reindex logic:
-  1. Select documents matching scope
-  2. For each document, re-run normalization from stored fields (or re-fetch if source available)
-  3. Update FTS-relevant fields
-  4. Upsert back to store
-  5. Run `OPTIMIZE INDEX idx_documents_fts` after bulk reindex to merge Tantivy segments
-  6. Log progress (N/total, errors)
-- [x] Implement `POST /admin/reindex` endpoint (behind `ENABLE_ADMIN_ENDPOINTS` + `ADMIN_AUTH_TOKEN`)
-- [x] Add error summary output on completion
-- [x] Exit non-zero on unrecoverable failures
-
-### Exit Criteria
-
-Operators can repair bad indexes without rebuilding everything manually.
-
-## M8 — Observability
-
-refs: [specs/06-operations.md](../specs/06-operations.md)
-
-### Goal
-
-Make the system diagnosable in production.
-
-### Deliverables
-
-- Structured slog fields across all services
-- Error classification
-- Ingestion lag visibility
-- Periodic state logs
-- Operator documentation
-
-### Tasks
-
-- [ ] Standardize slog fields across all packages:
-  - `service`, `event_name`, `event_id`, `did`, `collection`, `rkey`, `document_id`, `cursor`, `error_class`, `duration_ms`
-- [ ] Add error classification (normalize_error, db_error, tap_error, embed_error)
-- [ ] Add periodic state logs in indexer:
-  - Current cursor position
-  - Events processed since last log
-  - Documents in store (count)
-- [ ] Add request logging in API (method, path, status, duration, query)
-- [ ] Add search latency logging per query mode
-- [ ] Write operator documentation:
-  - Restart procedure
-  - Reindex procedure
-  - Backfill notes
-  - Failure triage guide
-
-### Exit Criteria
-
-The system is maintainable without guesswork.
diff --git a/docs/api/tasks/phase-2-semantic.md b/docs/api/tasks/phase-2-semantic.md
deleted file mode 100644
index 0906c0d..0000000
--- a/docs/api/tasks/phase-2-semantic.md
+++ /dev/null
@@ -1,167 +0,0 @@
----
-title: "Phase 2 — Semantic Search"
-updated: 2026-03-23
----
-
-# Phase 2 — Semantic Search
-
-Add embedding generation and vector-based retrieval on top of the keyword baseline, using self-hosted Ollama for embeddings instead of external API services.
-
-## M8 — Ollama Sidecar and Embedding Pipeline
-
-refs: [specs/01-architecture.md](../specs/01-architecture.md), [specs/03-data-model.md](../specs/03-data-model.md), [specs/05-search.md](../specs/05-search.md)
-
-### Goal
-
-Deploy Ollama as a Railway sidecar and add asynchronous embedding generation without blocking ingestion.
-
-### Deliverables
-
-- Ollama Railway service running nomic-embed-text-v1.5 (or EmbeddingGemma)
-- `embedding_jobs` table operational (schema from M1)
-- `embed-worker` subcommand
-- Ollama-backed embedding provider (with interface for future alternatives)
-- Retry and dead-letter behavior
-- `twister reembed` command
-
-### Tasks
-
-- [ ] Deploy Ollama on Railway:
-  - Use the nomic-embed Railway template as a starting point
-  - Configure as internal service (no public URL)
-  - Pre-pull `nomic-embed-text` model on startup
-  - Health check: `GET /api/tags` on port 11434
-  - Resource budget: 1–2 GB RAM, 1–2 vCPU
-- [ ] Define embedding provider interface:
-
-  ```go
-  type EmbeddingProvider interface {
-      Embed(ctx context.Context, texts []string) ([][]float32, error)
-      Model() string
-      Dimension() int
-  }
-  ```
-
-- [ ] Implement Ollama provider using the official Go client:
-
-  ```go
-  import "github.com/ollama/ollama/api"
-
-  // OllamaProvider calls Ollama's /api/embed endpoint
-  // over Railway internal networking (ollama.railway.internal:11434)
-  type OllamaProvider struct {
-      client *api.Client
-      model  string  // "nomic-embed-text"
-      dim    int     // 768
-  }
-  ```
-
-  - Configure via `OLLAMA_URL` env var (default: `http://ollama.railway.internal:11434`)
-  - Support batch embedding (Ollama accepts multiple inputs per request)
-  - Timeout per request (default: 30s)
-  - Connection health check on startup
-- [ ] Implement embedding input text composition (see spec 04-data-pipeline.md, section 5):
-  `title\nrepo_name\nauthor_handle\ntags\nsummary\nbody`
-- [ ] Add job enqueueing: on document upsert, insert `embedding_jobs` row with `status=pending`
-- [ ] Implement `embed-worker` loop:
-  1. Poll for `pending` jobs (batch by `EMBEDDING_BATCH_SIZE`, default: 32)
-  2. Compose input text per document
-  3. Call Ollama provider
-  4. Store vectors in `document_embeddings` with `vector32(?)`
-  5. Mark job `completed`
-  6. On failure: increment `attempts`, set `last_error`, backoff
-  7. After max attempts: mark `dead`
-- [ ] Create DiskANN vector index (see spec 03 for tuning params):
-  ```sql
-  CREATE INDEX idx_embeddings_vec ON document_embeddings(
-      libsql_vector_idx(embedding, 'metric=cosine')
-  );
-  ```
-- [ ] Implement `reembed` command (re-generate all embeddings, useful for model migration)
-- [ ] Skip deleted documents in embedding pipeline
-- [ ] Add health check endpoint for embed-worker (port 9091)
-- [ ] Add Ollama connectivity check to embed-worker readiness probe
-
-### Model Selection Notes
-
-**nomic-embed-text-v1.5** is the default recommendation:
-- 137M parameters, 768-dimension vectors
-- Matryoshka support (can truncate to 64/128/256/512 dims for storage tradeoff)
-- 8192 token context window
-- ~262 MB at F16 quantization, ~500 MB RAM at runtime
-- Battle-tested with llama.cpp/Ollama, Railway template exists
-
-**EmbeddingGemma** is the quality alternative:
-- 308M parameters, 768-dimension vectors
-- Best MTEB scores for models under 500M parameters
-- <200 MB quantized, similar RAM footprint
-- Released Sept 2025, less deployment track record
-
-**all-minilm** is the budget fallback:
-- 23M parameters, 384-dimension vectors (requires schema change)
-- ~46 MB model, minimal resources
-- Suitable for testing or cost-constrained environments
-
-### Verification
-
-- [ ] Ollama service starts on Railway and responds to health checks
-- [ ] Creating a new searchable document enqueues an embedding job
-- [ ] Worker processes the job and stores a vector in `document_embeddings`
-- [ ] Failed embedding calls retry with bounded attempts
-- [ ] Keyword search still works when embed-worker or Ollama is down
-- [ ] `reembed` regenerates embeddings for all eligible documents
-- [ ] Ollama connectivity failure is surfaced in embed-worker health check
-
-### Exit Criteria
-
-Embeddings are produced asynchronously via self-hosted Ollama and stored durably in Turso.
-
-## M9 — Semantic Search
-
-refs: [specs/05-search.md](../specs/05-search.md)
-
-### Goal
-
-Expose vector-based semantic retrieval.
-
-### Deliverables
-
-- `GET /search/semantic` endpoint
-- Query-time embedding (convert query text → vector via Ollama)
-- Vector similarity search via `vector_top_k`
-- Response parity with keyword search
-
-### Tasks
-
-- [ ] Implement query embedding: call Ollama provider with user's query text
-- [ ] Cache query embeddings for identical queries within a short TTL (optional, reduces Ollama load)
-- [ ] Implement semantic search repository:
-
-  ```sql
-  SELECT d.id, d.title, d.summary, d.repo_name, d.author_handle,
-         d.collection, d.record_type, d.created_at, d.updated_at
-  FROM vector_top_k('idx_embeddings_vec', vector32(?), ?) AS v
-  JOIN document_embeddings e ON e.rowid = v.id
-  JOIN documents d ON d.id = e.document_id
-  WHERE d.deleted_at IS NULL;
-  ```
-
-- [ ] Normalize distance to relevance score: `score = 1.0 - (distance / 2.0)`
-- [ ] Apply same filters as keyword search (collection, author, repo, type)
-- [ ] Add timeout and cost controls (limit vector search to reasonable K)
-- [ ] Wire `/search/semantic` handler
-- [ ] Return `matched_by: ["semantic"]` in results
-- [ ] Graceful degradation: if Ollama is unreachable, return 503 for semantic search while keyword search remains available
-
-### Verification
-
-- [ ] Semantically similar queries retrieve expected documents even with little lexical overlap
-- [ ] Documents without embeddings are omitted from semantic results
-- [ ] Semantic search returns the same JSON schema as keyword search
-- [ ] Latency is acceptable under small test load
-- [ ] Filters work correctly with semantic results
-- [ ] Semantic search degrades gracefully when Ollama is down
-
-### Exit Criteria
-
-The API supports true semantic search over Tangled documents, powered entirely by self-hosted infrastructure.
diff --git a/docs/api/tasks/phase-3-hybrid.md b/docs/api/tasks/phase-3-hybrid.md
deleted file mode 100644
index eb21220..0000000
--- a/docs/api/tasks/phase-3-hybrid.md
+++ /dev/null
@@ -1,51 +0,0 @@
----
-title: "Phase 3 — Hybrid Search"
-updated: 2026-03-22
----
-
-# Phase 3 — Hybrid Search
-
-Merge lexical and semantic search into the default high-quality retrieval mode.
-
-## M10 — Hybrid Search
-
-refs: [specs/05-search.md](../specs/05-search.md)
-
-### Deliverables
-
-- `GET /search/hybrid` endpoint
-- Weighted score blending (keyword 0.65 + semantic 0.35)
-- Score normalization
-- Result deduplication
-- `matched_by` metadata showing which modes contributed
-
-### Tasks
-
-- [ ] Implement hybrid search orchestrator:
-  1. Fetch top N keyword results (N=50 or configurable)
-  2. Fetch top N semantic results
-  3. Normalize keyword scores (min-max within result set)
-  4. Semantic scores already normalized (0–1)
-  5. Merge on `document_id`
-  6. For documents in both sets: `hybrid_score = 0.65 * keyword + 0.35 * semantic`
-  7. For documents in one set: use available score (other = 0)
-  8. Sort by hybrid_score descending
-  9. Deduplicate
-  10. Apply limit/offset
-- [ ] Populate `matched_by` field: `["keyword"]`, `["semantic"]`, or `["keyword", "semantic"]`
-- [ ] Make weights configurable via `HYBRID_KEYWORD_WEIGHT` / `HYBRID_SEMANTIC_WEIGHT`
-- [ ] Wire `/search/hybrid` handler
-- [ ] Make `/search?mode=hybrid` work
-
-### Verification
-
-- [ ] Hybrid returns documents found by either source
-- [ ] Duplicates are merged correctly (no duplicate IDs in results)
-- [ ] Exact-match queries still favor lexical relevance
-- [ ] Exploratory natural-language queries improve over keyword-only results
-- [ ] Score ordering is stable across repeated runs on the same corpus
-- [ ] `matched_by` accurately reflects which modes produced each result
-
-### Exit Criteria
-
-Hybrid search becomes the preferred default search mode.
diff --git a/docs/api/tasks/phase-4-quality.md b/docs/api/tasks/phase-4-quality.md
deleted file mode 100644
index 1a97ca1..0000000
--- a/docs/api/tasks/phase-4-quality.md
+++ /dev/null
@@ -1,47 +0,0 @@
----
-title: "Phase 4 — Ranking and Quality Polish"
-updated: 2026-03-22
----
-
-# Phase 4 — Ranking and Quality Polish
-
-Improve search quality without changing the core architecture.
-
-## M11 — Ranking and Quality Polish
-
-refs: [specs/05-search.md](../specs/05-search.md)
-
-### Deliverables
-
-- Boosted field weighting refinement
-- Recency boost
-- Collection-aware ranking
-- Better snippets/highlights
-- Issue/PR state filtering
-- Star count as ranking signal
-- Optional query analytics
-
-### Tasks
-
-- [ ] Tune FTS index weights based on real query results
-- [ ] Add small recency boost to ranking (e.g., decay function on `created_at`)
-- [ ] Add collection-aware ranking adjustments (repos ranked differently from comments)
-- [ ] Index `sh.tangled.repo.issue.comment` and `sh.tangled.repo.pull.comment` (P2 collections)
-- [ ] Aggregate `sh.tangled.feed.star` counts per repo and use as ranking signal
-- [ ] Implement `state` filter (open/closed/merged) using `record_state` table
-- [ ] Improve snippets: better truncation, multi-field highlights
-- [ ] Add curated relevance test fixtures (expected queries → expected top results)
-- [ ] Run `OPTIMIZE INDEX idx_documents_fts` as maintenance task
-- [ ] Optional: log queries for analytics (anonymized)
-
-### Verification
-
-- [ ] Exact repo lookups reliably rank the repo first
-- [ ] Recent active content gets a reasonable small boost without overwhelming exact relevance
-- [ ] Snippets show useful matched context
-- [ ] Ranking regression tests catch obvious degradations
-- [ ] State filter correctly excludes closed/merged items when requested
-
-### Exit Criteria
-
-Search quality is noticeably improved and more predictable.
diff --git a/docs/app/specs/README.md b/docs/app/specs/README.md
deleted file mode 100644
index 17b9fde..0000000
--- a/docs/app/specs/README.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# Twisted — Tangled Mobile Companion
-
-A mobile-first Tangled client for iOS, Android, and web. Built with Ionic Vue, Capacitor, and the `@atcute` AT Protocol client stack.
-
-## What is Tangled
-
-[Tangled](https://tangled.org) is a Git hosting and collaboration platform built on the [AT Protocol](https://atproto.com). Identity, social graph (follows, stars, reactions), repos, issues, and PRs are all AT Protocol records stored on users' Personal Data Servers. Git hosting runs on **knots** — headless servers exposing XRPC APIs. The **appview** at `tangled.org` aggregates and renders the network view.
-
-- Docs: 
-- Lexicon namespace: `sh.tangled.*`
-- Source: 
-
-## What Twisted Does
-
-**Reader and social companion** for Tangled. Focused on direct browsing, indexed discovery, and lightweight interactions.
-
-- Browse repos, files, READMEs, issues, PRs
-- Jump to profiles and repos from a known AT Protocol handle
-- Search indexed repos and profiles through the Twister API
-- Use index-backed graph summaries where the public API is incomplete
-- Sign in via AT Protocol OAuth
-- Star repos, follow users, react to content
-- Offline-capable with cached data
-
-Out of scope: repo creation, git push/pull, CI/CD, full code review authoring.
-
-## Technology
-
-| Layer       | Choice                                                                                        |
-| ----------- | --------------------------------------------------------------------------------------------- |
-| Framework   | Vue 3 + TypeScript                                                                            |
-| UI          | Ionic Vue                                                                                     |
-| Native      | Capacitor (iOS, Android, Web)                                                                 |
-| State       | Pinia                                                                                         |
-| Async data  | TanStack Query (Vue)                                                                          |
-| AT Protocol | `@atcute/client` (XRPC), `@atcute/oauth-browser-client` (OAuth), `@atcute/tangled` (lexicons) |
-
-## Architecture
-
-Three layers, strict dependency direction (presentation → domain → data):
-
-**Presentation** — Ionic pages, Vue components, composables, Pinia stores.
-**Domain** — Normalized models (`UserSummary`, `RepoDetail`, `ActivityItem`, etc.), action policies, pagination.
-**Data** — `@atcute/client` XRPC calls, `@atcute/tangled` type definitions, local cache, and the Twister API for search/index-backed summaries.
-
-Protocol isolation: no Vue component imports `@atcute/*` directly. All API access flows through `src/services/`.
-
-## Tangled API Surface
-
-Three distinct data hosts:
-
-| Host                               | Protocol                           | Data                                                              |
-| ---------------------------------- | ---------------------------------- | ----------------------------------------------------------------- |
-| Knots (`us-west.tangled.sh`, etc.) | XRPC at `/xrpc/sh.tangled.*`       | Git data: trees, blobs, commits, branches, diffs, tags            |
-| User's PDS                         | XRPC at `/xrpc/com.atproto.repo.*` | AT Protocol records: repos, issues, PRs, stars, follows, profiles |
-| Twister API                        | HTTP JSON                          | Global search and index-backed graph/profile summaries            |
-
-The appview (`tangled.org`) serves HTML — it's the web UI, not a JSON API. The mobile client talks to knots and PDS servers directly for canonical detail and uses the Twister API for cross-network discovery.
-
-Repo param format: `did:plc:xxx/repoName`.
-
-## Phases
-
-| Phase | Focus                                                                    | Spec                                 | Tasks                                |
-| ----- | ------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------ |
-| 1     | Project shell, tabs, mock data, design system                            | [phase-1.md](phase-1.md)             | [../tasks/phase-1.md](../tasks/phase-1.md) |
-| 2     | Public browsing — repos, files, profiles, issues, PRs                    | [phase-2.md](phase-2.md)             | [../tasks/phase-2.md](../tasks/phase-2.md) |
-| 3     | Index-backed search and handle-first public browsing                     | [phase-3.md](phase-3.md)             | [../tasks/phase-3.md](../tasks/phase-3.md) |
-| 4     | OAuth sign-in, star, follow, react, personalized feed                    | [phase-4.md](phase-4.md)             | [../tasks/phase-4.md](../tasks/phase-4.md) |
-| 5     | Offline persistence, performance, bundle optimization                    | [phase-5.md](phase-5.md)             | [../tasks/phase-5.md](../tasks/phase-5.md) |
-| 6     | Write features, project service integration, push notifications          | [phase-6.md](phase-6.md)             | [../tasks/phase-6.md](../tasks/phase-6.md) |
-| 7     | Real-time Jetstream feed, custom feeds, forking, labels, interdiff       | [phase-7.md](phase-7.md)             | [../tasks/phase-7.md](../tasks/phase-7.md) |
-
-## Key Design Decisions
-
-1. **`@atcute` end-to-end** for all AT Protocol interaction — no mixing client stacks.
-2. **Tangled lexicon handling in one module boundary** (`src/services/tangled/`) — don't scatter `sh.tangled.*` awareness across pages.
-3. **Read-first** — the primary product is a fast reader. Social mutations are a controlled second layer.
-4. **Use the project API sparingly and intentionally.** Search and index-backed graph gaps belong there; canonical repo detail stays on Tangled's public APIs.
-5. **Mobile-first, not desktop-forge-first** — prioritize readability, direct browsing, and small focused actions before broader discovery surfaces.
diff --git a/docs/app/specs/phase-1.md b/docs/app/specs/phase-1.md
deleted file mode 100644
index c83506a..0000000
--- a/docs/app/specs/phase-1.md
+++ /dev/null
@@ -1,180 +0,0 @@
-# Phase 1 — Project Shell & Design System
-
-## Goal
-
-Scaffold the Ionic Vue project with tab navigation, placeholder pages, mock data, and reusable UI primitives. Nothing touches the network. The result is a clickable prototype that validates navigation, layout, and component design before any API integration.
-
-## Technology Stack
-
-| Layer          | Choice                  |
-| -------------- | ----------------------- |
-| Framework      | Vue 3 + TypeScript      |
-| UI kit         | Ionic Vue               |
-| Native runtime | Capacitor               |
-| State          | Pinia                   |
-| Async data     | TanStack Query (Vue)    |
-| Routing        | Vue Router (Ionic tabs) |
-
-## Navigation Structure
-
-Five-tab layout:
-
-1. **Home** — trending repos, recent activity, personalized content (auth)
-2. **Explore** — search repos/users, filters
-3. **Repo** — deep-link target for repository detail (not a persistent tab icon — navigated to from Home/Explore/Activity)
-4. **Activity** — global feed (anon), social graph feed (auth)
-5. **Profile** — auth state, user card, follows, starred repos, settings
-
-> Repo is a routed detail destination, not a standing tab. The tab bar shows Home, Explore, Activity, Profile. Repo pages are pushed onto the Home/Explore/Activity stacks.
-
-## Directory Layout
-
-```sh
-src/
-  app/
-    router/           # route definitions, tab guards
-    boot/             # app-level setup (query client, plugins)
-    providers/        # provide/inject wrappers
-  core/
-    config/           # env, feature flags
-    errors/           # error types and normalization
-    storage/          # storage abstraction (IndexedDB / Capacitor Secure Storage)
-    query/            # TanStack Query client config, persister setup
-    auth/             # auth state machine, session store
-  services/
-    atproto/          # @atcute/client wrapper, identity helpers
-    tangled/          # Tangled API: endpoints, adapters, normalizers, queries, mutations
-  domain/
-    models/           # UserSummary, RepoSummary, RepoDetail, etc.
-    feed/             # feed-specific types and helpers
-    repo/             # repo-specific types and helpers
-    profile/          # profile-specific types and helpers
-  features/
-    home/
-    explore/
-    repo/
-    activity/
-    profile/
-  components/
-    common/           # cards, buttons, loaders, empty states, error boundaries
-    repo/             # repo card, file tree item, README viewer
-    feed/             # activity card, feed list
-    profile/          # user card, follow button
-```
-
-## Domain Models
-
-```ts
-export type UserSummary = {
-    did: string;
-    handle: string;
-    displayName?: string;
-    avatar?: string;
-    bio?: string;
-    followerCount?: number;
-    followingCount?: number;
-};
-
-export type RepoSummary = {
-    atUri: string;
-    ownerDid: string;
-    ownerHandle: string;
-    name: string;
-    description?: string;
-    primaryLanguage?: string;
-    stars?: number;
-    forks?: number;
-    updatedAt?: string;
-    knot: string;
-};
-
-export type RepoDetail = RepoSummary & {
-    readme?: string;
-    defaultBranch?: string;
-    languages?: Record;
-    collaborators?: UserSummary[];
-    topics?: string[];
-};
-
-export type RepoFile = {
-    path: string;
-    name: string;
-    type: "file" | "dir" | "submodule";
-    size?: number;
-    lastCommitMessage?: string;
-};
-
-export type PullRequestSummary = {
-    atUri: string;
-    title: string;
-    authorDid: string;
-    authorHandle: string;
-    status: "open" | "merged" | "closed";
-    createdAt: string;
-    updatedAt?: string;
-    sourceBranch: string;
-    targetBranch: string;
-    roundCount?: number;
-};
-
-export type IssueSummary = {
-    atUri: string;
-    title: string;
-    authorDid: string;
-    authorHandle: string;
-    state: "open" | "closed";
-    createdAt: string;
-    commentCount?: number;
-};
-
-export type ActivityItem = {
-    id: string;
-    kind:
-        | "repo_created"
-        | "repo_starred"
-        | "user_followed"
-        | "pr_opened"
-        | "pr_merged"
-        | "issue_opened"
-        | "issue_closed";
-    actorDid: string;
-    actorHandle: string;
-    targetUri?: string;
-    targetName?: string;
-    createdAt: string;
-};
-```
-
-## Repo Detail Page Structure
-
-Segmented tab layout within the repo detail view:
-
-| Segment  | Content                                                                              |
-| -------- | ------------------------------------------------------------------------------------ |
-| Overview | owner/repo header, description, topics, social action buttons, README preview, stats |
-| Files    | directory tree, file viewer (syntax-highlighted)                                     |
-| Issues   | issue list with state filters                                                        |
-| PRs      | pull request list with status filters                                                |
-
-## Design System Primitives
-
-Build these reusable components during this phase:
-
-- **RepoCard** — compact repo summary for lists
-- **UserCard** — avatar + handle + bio snippet
-- **ActivityCard** — icon + actor + verb + target + timestamp
-- **FileTreeItem** — icon (file/dir) + name + last commit message
-- **EmptyState** — icon + message + optional action button
-- **ErrorBoundary** — catch + retry UI
-- **SkeletonLoader** — content placeholder shimmer for each card type
-- **MarkdownRenderer** — render README content (Phase 2 will wire to real data)
-
-## Mock Data
-
-Create `src/mocks/` with factory functions returning typed domain models. All Phase 1 screens render from these factories. Mock data must be realistic — use real-looking handles (`alice.tngl.sh`), repo names, and timestamps.
-
-## Performance Targets
-
-- Shell first-paint under 2s on mid-range device
-- Tab switches feel instant (no layout shift)
-- Skeleton loaders shown within 100ms of navigation
diff --git a/docs/app/specs/phase-2.md b/docs/app/specs/phase-2.md
deleted file mode 100644
index bade0e4..0000000
--- a/docs/app/specs/phase-2.md
+++ /dev/null
@@ -1,120 +0,0 @@
-# Phase 2 — Public Tangled Browsing
-
-## Goal
-
-Replace mock data on the shippable public-browsing surface with live Tangled API calls. Users can browse repos, profiles, file trees, README content, issues, and pull requests without signing in. Public entry points are intentionally scoped down for now: Home is a known-handle jump surface, while Explore and Activity remain clearly labeled placeholders until their dedicated work lands.
-
-## Protocol Stack
-
-| Package           | Version | Role                                           |
-| ----------------- | ------- | ---------------------------------------------- |
-| `@atcute/client`  | ^4.2.1  | XRPC HTTP client — `query()` and `procedure()` |
-| `@atcute/tangled` | ^1.0.17 | `sh.tangled.*` lexicon type definitions        |
-
-All protocol access goes through `src/services/tangled/`. No Vue component may import `@atcute/*` directly.
-
-## Architecture: Protocol Isolation
-
-```sh
-Vue component
-  → composable (useRepoDetail, useFileTree, ...)
-    → TanStack Query hook
-      → service function (services/tangled/queries.ts)
-        → @atcute/client XRPC call
-          → normalizer (services/tangled/normalizers.ts)
-            → domain model
-```
-
-### Service Layer Responsibilities
-
-**`services/atproto/client.ts`** — singleton `XRPC` client instance, base URL config, error interceptor.
-
-**`services/tangled/endpoints.ts`** — typed wrappers around XRPC queries:
-
-| Endpoint                           | Params                                      | Returns             |
-| ---------------------------------- | ------------------------------------------- | ------------------- |
-| `sh.tangled.repo.tree`             | `repo: did:plc:xxx/name`, `ref`, `path?`    | directory listing   |
-| `sh.tangled.repo.blob`             | `repo`, `ref`, `path`                       | file content        |
-| `sh.tangled.repo.log`              | `repo`, `ref`, `path?`, `limit?`, `cursor?` | commit history      |
-| `sh.tangled.repo.branches`         | `repo`, `limit?`, `cursor?`                 | branch list         |
-| `sh.tangled.repo.tags`             | `repo`                                      | tag list            |
-| `sh.tangled.repo.getDefaultBranch` | `repo`                                      | default branch name |
-| `sh.tangled.repo.diff`             | `repo`, `ref`                               | diff output         |
-| `sh.tangled.repo.compare`          | `repo`, `rev1`, `rev2`                      | comparison          |
-| `sh.tangled.repo.languages`        | `repo`                                      | language breakdown  |
-
-The `repo` param format is `did:plc:xxx/repoName`. The XRPC calls go to the repo's **knot** hostname (e.g., `us-west.tangled.sh`), not to `tangled.org`.
-
-**`services/tangled/normalizers.ts`** — transform raw lexicon responses into domain models (`RepoSummary`, `RepoDetail`, `RepoFile`, etc.).
-
-**`services/tangled/queries.ts`** — TanStack Query wrapper functions with cache keys, stale times, and error handling.
-
-## Appview vs Knot Routing
-
-Tangled has two API surfaces:
-
-| Surface | Host                       | Protocol                    | Used for                                          |
-| ------- | -------------------------- | --------------------------- | ------------------------------------------------- |
-| Appview | `tangled.org`              | HTTP (HTML, HTMX)           | Profile pages, repo listings, timeline, search    |
-| Knots   | `us-west.tangled.sh`, etc. | XRPC (`/xrpc/sh.tangled.*`) | Git data — trees, blobs, commits, branches, diffs |
-
-For Phase 2, git data comes from knots via XRPC. Profile and repo metadata come from PDS records queried through `com.atproto.repo.getRecord` and `com.atproto.repo.listRecords`, not from the HTML appview. The service layer must route requests to the correct host based on the operation.
-
-## Features
-
-### Repository Browsing
-
-- List repos for a user (from their PDS records or appview)
-- Repo overview: metadata, description, topics, default branch, language stats
-- README rendering: fetch blob for `README.md` from default branch, render markdown
-- File tree: navigate directories, open files
-- File viewer: syntax-highlighted source display
-- Commit log: paginated history for a ref/path
-- Branch list with default branch indicator
-
-### Profile Browsing
-
-- View user profile: avatar, bio, links, pronouns, location, pinned repos
-- Profile data comes from `sh.tangled.actor.profile` record (key: `self`) on the user's PDS
-- List user's repos
-
-### Public Discovery (scoped down)
-
-- Home acts as the temporary public entry point: enter a known AT Protocol handle, then jump to profile or browse that handle's repos
-- Explore remains visible as a placeholder for future search work, but should not pretend global search already exists
-- Activity remains visible as a placeholder for future feed work, but should not pretend a public timeline already exists
-- Unsupported global search/trending behavior should be omitted or clearly labeled as future work, never filled with silent mock data
-
-### Pull Requests (read-only)
-
-- List PRs for a repo with status filter (open/closed/merged)
-- PR detail: title, body, author, source/target branches, round count
-- PR comments list
-
-### Issues (read-only)
-
-- List issues for a repo with state filter (open/closed)
-- Issue detail: title, body, author
-- Issue comments (threaded — `replyTo` field)
-
-## Caching Strategy
-
-| Data          | Stale time | Cache time |
-| ------------- | ---------- | ---------- |
-| Repo metadata | 5 min      | 30 min     |
-| File tree     | 2 min      | 10 min     |
-| File content  | 5 min      | 30 min     |
-| Commit log    | 2 min      | 10 min     |
-| Profile       | 10 min     | 60 min     |
-| README        | 5 min      | 30 min     |
-
-Use TanStack Query's `staleTime` and `gcTime`. Add a query persister (IndexedDB-backed) for offline reads.
-
-## Error Handling
-
-Normalize these failure modes at the service layer:
-
-- Network unreachable → offline banner, serve from cache
-- 404 from knot → "Repository not found" or "File not found"
-- XRPC error responses → map to typed app errors
-- Malformed response → log + generic error state
diff --git a/docs/app/specs/phase-3.md b/docs/app/specs/phase-3.md
deleted file mode 100644
index cc5183b..0000000
--- a/docs/app/specs/phase-3.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# Phase 3 — Indexed Search and Honest Discovery
-
-## Goal
-
-Introduce global discovery through the Twister project index while preserving honest product boundaries. Home continues to support direct known-handle browsing, Explore becomes index-backed search, and Activity remains a clearly labeled in-progress surface.
-
-## Current Product Shape
-
-### Home
-
-Home is the temporary public entry point for unauthenticated browsing:
-
-- Enter a known AT Protocol handle
-- Open that user's profile directly
-- Resolve the handle to DID + PDS via AT Protocol identity
-- List that user's public Tangled repos inline and open one directly
-
-This keeps public browsing fully real while still giving the app a lightweight direct-entry path.
-
-### Explore
-
-Explore becomes the network-level discovery surface:
-
-- Global repo search via the Twister index
-- Global profile search via the Twister index
-- Empty state should clearly distinguish "index unavailable" from "no results"
-- Search results route into the existing profile and repo detail screens
-
-### Activity
-
-Activity also remains a tab-level placeholder:
-
-- No public timeline yet
-- No curated public feed fallback
-- Empty state should explicitly say activity is in progress
-
-## Identity and Routing
-
-The app now uses two read paths:
-
-1. **Direct handle browsing**
-   Resolve `handle -> DID` via `com.atproto.identity.resolveHandle`
-   Fetch the DID document and extract the PDS endpoint
-   Query the user's PDS for `sh.tangled.repo` records via `com.atproto.repo.listRecords`
-2. **Indexed discovery**
-   Query the Twister API for global search results
-   Open the selected profile or repo in the existing screens
-   Continue detail fetching from Tangled's public APIs
-
-The Twister API is additive, not authoritative for repo detail. It fills discovery and graph gaps; knots and PDSes remain the source of truth for detail screens.
-
-## UI Expectations
-
-- Home shows one handle input plus explicit actions for profile jump and repo browsing
-- Home shows loading, invalid-handle, no-repos, and resolved-repo-list states
-- Explore shows a working search form, loading state, index-unavailable state, and no-results state
-- Activity shows a static in-progress empty state
-- Profile may show index-backed follower/following summaries when available
-
-## Deferred Work
-
-The following work is intentionally deferred out of this phase:
-
-- Trending or suggested discovery sections
-- Public activity feed ingestion, pagination, and caching
-- Jetstream or appview timeline investigation
-
-These capabilities will be revisited after the baseline search and graph-summary integration is stable.
diff --git a/docs/app/specs/phase-4.md b/docs/app/specs/phase-4.md
deleted file mode 100644
index 3ef97f9..0000000
--- a/docs/app/specs/phase-4.md
+++ /dev/null
@@ -1,167 +0,0 @@
-# Phase 4 — OAuth & Social Features
-
-## Goal
-
-Add AT Protocol OAuth sign-in and authenticated social actions: follow, star, react. Signed-in users get a personalized feed.
-
-## Authentication
-
-### Package
-
-`@atcute/oauth-browser-client` ^3.0.0 — minimal browser OAuth client for AT Protocol.
-
-### OAuth Flow
-
-1. User enters handle or DID
-2. Resolve handle → DID → PDS → authorization server metadata
-3. Initiate OAuth with PKCE + DPoP (P-256)
-4. Redirect to authorization server
-5. Callback with auth code
-6. Exchange code for access + refresh tokens
-7. Store session, bind to XRPC client
-
-### Key Functions
-
-| Function                   | Purpose                                                      |
-| -------------------------- | ------------------------------------------------------------ |
-| `configureOAuth(opts)`     | One-time setup: client metadata URL, redirect URI            |
-| `getSession(did)`          | Resume existing session (returns `Session` with `dpopFetch`) |
-| `listStoredSessions()`     | List all stored accounts                                     |
-| `deleteStoredSession(did)` | Remove stored session                                        |
-
-### Session Object
-
-A `Session` provides:
-
-- `did` — authenticated user's DID
-- `dpopFetch` — a `fetch` wrapper that auto-attaches DPoP + access token headers
-- Token refresh is handled internally
-
-### Client Metadata
-
-The mobile app needs its own OAuth client metadata hosted at a public URL:
-
-```json
-{
-    "client_id": "https://your-app-domain/oauth/client-metadata.json",
-    "client_name": "Twisted",
-    "client_uri": "https://your-app-domain",
-    "redirect_uris": ["https://your-app-domain/oauth/callback"],
-    "grant_types": ["authorization_code", "refresh_token"],
-    "response_types": ["code"],
-    "token_endpoint_auth_method": "none",
-    "application_type": "web",
-    "dpop_bound_access_tokens": true,
-    "scope": "atproto repo:sh.tangled.graph.follow repo:sh.tangled.feed.star repo:sh.tangled.feed.reaction repo:sh.tangled.actor.profile"
-}
-```
-
-Request only the scopes needed for Phase 4 social features. Expand scopes in later phases as write features are added.
-
-### Capacitor Considerations
-
-- Web: standard redirect flow works
-- iOS/Android via Capacitor: use `App.addListener('appUrlOpen')` to capture the OAuth callback via deep link or custom URL scheme
-- Session storage: abstract behind `core/storage/` — use `localStorage` on web, Capacitor Secure Storage plugin on native
-
-### Auth State Machine
-
-```sh
-idle → authenticating → authenticated
-                     → error
-authenticated → refreshing → authenticated
-                           → expired → idle
-authenticated → logging_out → idle
-```
-
-Store in Pinia (`core/auth/`). Expose via `useAuth()` composable.
-
-## Social Actions
-
-All social actions create or delete AT Protocol records on the user's PDS via the XRPC `com.atproto.repo.createRecord` / `com.atproto.repo.deleteRecord` procedures. The `dpopFetch` from the session handles auth.
-
-### Star a Repo
-
-Create record:
-
-```json
-{
-    "repo": "did:plc:user",
-    "collection": "sh.tangled.feed.star",
-    "record": {
-        "$type": "sh.tangled.feed.star",
-        "subject": "at://did:plc:owner/sh.tangled.repo/tid",
-        "createdAt": "2026-03-22T00:00:00Z"
-    }
-}
-```
-
-Unstar: delete the record by its `rkey`.
-
-### Follow a User
-
-Create record:
-
-```json
-{
-    "repo": "did:plc:user",
-    "collection": "sh.tangled.graph.follow",
-    "record": {
-        "$type": "sh.tangled.graph.follow",
-        "subject": "did:plc:target",
-        "createdAt": "2026-03-22T00:00:00Z"
-    }
-}
-```
-
-Unfollow: delete the record by its `rkey`.
-
-### React to Content
-
-Create record:
-
-```json
-{
-    "repo": "did:plc:user",
-    "collection": "sh.tangled.feed.reaction",
-    "record": {
-        "$type": "sh.tangled.feed.reaction",
-        "subject": "at://did:plc:owner/sh.tangled.repo.pull/tid",
-        "reaction": "thumbsup",
-        "createdAt": "2026-03-22T00:00:00Z"
-    }
-}
-```
-
-Available reactions: `thumbsup`, `thumbsdown`, `laugh`, `tada`, `confused`, `heart`, `rocket`, `eyes`.
-
-### Optimistic Updates
-
-All mutations use TanStack Query's `useMutation` with optimistic updates:
-
-1. Immediately update the cache (star count +1, follow state toggled)
-2. Fire the mutation
-3. On error, roll back the cache and show a toast
-
-## Personalized Feed
-
-When signed in, the Activity tab shows a filtered feed based on:
-
-- Users the signed-in user follows
-- Repos the signed-in user has starred
-
-Implementation depends on what the appview provides. If no personalized endpoint exists, filter the global feed client-side based on the user's follow/star records.
-
-## Profile Tab (Authenticated)
-
-When signed in, the Profile tab shows:
-
-- User's avatar, handle, bio, location, pronouns, links
-- Pinned repos
-- Stats (selected from: merged PRs, open PRs, open issues, repo count, star count)
-- Starred repos list
-- Following/followers lists
-- Edit profile (avatar, bio, links, pinned repos)
-- Settings
-- Logout
-- Account switcher (multiple account support via `listStoredSessions`)
diff --git a/docs/app/specs/phase-5.md b/docs/app/specs/phase-5.md
deleted file mode 100644
index 3c2ed3f..0000000
--- a/docs/app/specs/phase-5.md
+++ /dev/null
@@ -1,73 +0,0 @@
-# Phase 5 — Offline & Performance Polish
-
-## Goal
-
-Make the app feel native. Cached data loads instantly, offline mode is graceful, and navigation is smooth on mid-range devices.
-
-## Offline Strategy
-
-### Query Persistence
-
-Use TanStack Query's `persistQueryClient` with an IndexedDB adapter:
-
-- Persist all query cache to IndexedDB on each update (debounced)
-- On app launch, hydrate TanStack Query cache from IndexedDB before rendering
-- Stale-while-revalidate: show persisted data immediately, refresh in background
-
-### What to Persist
-
-| Data                           | Max cached items | TTL    |
-| ------------------------------ | ---------------- | ------ |
-| Repo metadata                  | 200              | 7 days |
-| File trees                     | 50               | 3 days |
-| File content (recently viewed) | 100              | 3 days |
-| README content                 | 100              | 7 days |
-| User profiles                  | 100              | 7 days |
-| Activity feed pages            | 10 pages         | 1 day  |
-| Search results                 | 20 queries       | 1 day  |
-
-### Offline Detection
-
-- Listen to `navigator.onLine` + `online`/`offline` events
-- Show a persistent banner when offline: "You're offline — showing cached data"
-- Disable mutation buttons (star, follow) when offline
-- Queue mutations for retry when back online (optional, simple queue)
-
-### Sensitive Data
-
-- Auth tokens: Capacitor Secure Storage on native, encrypted `localStorage` wrapper on web
-- Never persist tokens in IndexedDB alongside query cache
-- Clear auth storage on logout
-
-## Performance Optimizations
-
-### Navigation
-
-- Prefetch repo detail data on repo card hover/long-press
-- Keep previous tab's scroll position and data in memory (Ionic's `ion-router-outlet` + `keep-alive`)
-- Use `` or a virtualized list for long lists (repos, activity feed)
-
-### Images
-
-- Lazy-load avatars with `loading="lazy"` or Intersection Observer
-- Use `avatar.tangled.sh` CDN URLs with size params if available
-- Placeholder avatar component with initials fallback
-
-### Bundle
-
-- Route-level code splitting per feature folder
-- Tree-shake unused Ionic components
-- Measure and optimize with Lighthouse
-
-### Rendering
-
-- Skeleton screens for every data-driven view (already built in Phase 1)
-- Debounce search input (already in Phase 3)
-- Throttle scroll-based pagination triggers
-
-## Testing Focus
-
-- Offline → online transition: verify data refreshes without duplicates
-- Large repo file trees: ensure virtual scroll handles 1000+ items
-- Low-bandwidth simulation: verify skeleton → content transitions
-- Memory pressure: verify cache eviction works and app doesn't grow unbounded
diff --git a/docs/app/specs/phase-6.md b/docs/app/specs/phase-6.md
deleted file mode 100644
index 3cfc6ca..0000000
--- a/docs/app/specs/phase-6.md
+++ /dev/null
@@ -1,75 +0,0 @@
-# Phase 6 — Write Features & Project Services
-
-## Goal
-
-Add authenticated write operations (create issues, comment on PRs/issues, edit profile) and extend the Twister project services only where the client should not or cannot do the work directly.
-
-## Why Project Services
-
-Some operations are awkward or unsafe from a browser client:
-
-- **Token hardening**: DPoP keys in browser storage are less secure than server-held credentials
-- **Unstable procedures**: Tangled's API may change — a backend adapter isolates the mobile client from churn
-- **Push notifications**: require server-side registration and delivery
-- **Personalized feeds**: server-side aggregation is more efficient than client-side filtering
-- **Graph gaps**: follower lists/counts and other cross-network summaries may require index-backed derivation
-- **Rate limiting**: backend can batch and deduplicate requests
-
-### Service Scope
-
-Thin service layer — not a replacement for Tangled's public APIs. Use it for cross-network aggregation, search, notifications, and operations the SPA should not own.
-
-| Endpoint                             | Purpose                                             |
-| ------------------------------------ | --------------------------------------------------- |
-| `POST /auth/session`                 | OAuth token exchange and session management         |
-| `GET /feed/personalized`             | Pre-filtered activity feed for the user             |
-| `GET /search`, `GET /profiles/:did/summary` | Search and index-backed graph/profile summaries |
-| `POST /notifications/register`       | Push notification device registration               |
-| Passthrough for stable XRPC calls    | Avoid duplicating what the client already does well |
-
-## Write Features
-
-### Create Issue
-
-- Screen: issue creation form within repo detail
-- Fields: title (required), body (markdown), mentions
-- Creates `sh.tangled.repo.issue` record on user's PDS
-- Optimistic: add to local issue list, remove on failure
-
-### Comment on Issue / PR
-
-- Screen: comment input at bottom of issue/PR detail
-- Creates `sh.tangled.repo.issue.comment` or `sh.tangled.repo.pull.comment` record
-- Supports `replyTo` for threaded issue comments
-- Supports `mentions` (DID array) and `references` (AT-URI array)
-
-### Edit Profile
-
-- Screen: profile edit form
-- Updates `sh.tangled.actor.profile` record (key: `self`)
-- Fields: avatar (image upload, max 1MB, png/jpeg), bio (max 256 graphemes), links (max 5 URIs), location (max 40 graphemes), pronouns (max 40 chars), pinned repos (max 6 AT-URIs), display stats (max 2 from: merged-pr-count, closed-pr-count, open-pr-count, open-issue-count, closed-issue-count, repo-count, star-count), bluesky cross-posting toggle
-
-### Issue State Management
-
-- Close/reopen issues by creating `sh.tangled.repo.issue.state` records
-- State values: `sh.tangled.repo.issue.state.open`, `sh.tangled.repo.issue.state.closed`
-
-## Push Notifications
-
-- Register device token with project services
-- Project services subscribe to Jetstream or indexed events relevant to the user
-- Deliver via APNs (iOS) / FCM (Android)
-- Notification types: PR activity on your repos, issue comments, new followers, stars
-
-## Expanded OAuth Scopes
-
-Phase 6 requires additional scopes beyond Phase 4:
-
-```sh
-repo:sh.tangled.repo.issue
-repo:sh.tangled.repo.issue.comment
-repo:sh.tangled.repo.issue.state
-repo:sh.tangled.repo.pull.comment
-```
-
-Handle scope upgrades gracefully — re-authorize if the user's existing session lacks required scopes.
diff --git a/docs/app/specs/phase-7.md b/docs/app/specs/phase-7.md
deleted file mode 100644
index df62314..0000000
--- a/docs/app/specs/phase-7.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Phase 7 — Real-Time Feed & Advanced Features
-
-## Goal
-
-Add real-time event streaming, custom feed logic, and advanced social coding features. This phase makes the app feel alive.
-
-## Jetstream Integration
-
-### Package
-
-`@atcute/jetstream` — subscribe to the AT Protocol event stream.
-
-### Architecture
-
-Connect to a Jetstream relay and filter for `sh.tangled.*` collections:
-
-```sh
-Jetstream WebSocket
-  → filter: sh.tangled.* events
-    → normalize into ActivityItem
-      → merge into TanStack Query feed cache
-        → reactive UI update
-```
-
-### Connection Management
-
-- Connect on app foreground, disconnect on background
-- Reconnect with exponential backoff
-- Track cursor position for gap-fill on reconnect
-- Battery-aware: reduce polling frequency on low battery (Capacitor Battery API)
-
-### Live Indicators
-
-- Repo detail: show "new commits" banner when ref updates arrive
-- Activity feed: show "X new items" pill, tap to scroll to top and reveal
-- PR detail: live status updates (open → merged)
-
-## Custom Feeds
-
-Allow users to create saved feed configurations:
-
-- "My repos" — activity on repos I own
-- "Watching" — activity on repos I starred
-- "Team" — activity from users I follow
-- Custom filters: by repo, by user, by event type
-
-Feeds are stored locally in IndexedDB. If project services exist, they can optionally sync server-side for push notification filtering.
-
-## Advanced Features
-
-### Repo Forking
-
-- Fork button on repo detail (requires `rpc:sh.tangled.repo.create` scope)
-- Fork status indicator via `sh.tangled.repo.forkStatus` (up-to-date, fast-forwardable, conflict, missing branch)
-- Sync fork via `sh.tangled.repo.forkSync`
-
-### Label Support
-
-- Display labels on issues and PRs
-- Apply/remove labels (requires label scopes)
-- Color-coded label chips
-
-### Reaction Picker
-
-- Expand reaction support beyond star/follow
-- Emoji picker for: thumbsup, thumbsdown, laugh, tada, confused, heart, rocket, eyes
-- Show reaction counts on PRs, issues, comments
-
-### PR Interdiff
-
-- View diff between PR rounds (round N vs round N+1)
-- Useful for code review on mobile — see what changed since last review
-
-### Knot Information
-
-- Show which knot hosts a repo
-- Knot version and status
-- Useful for debugging and transparency
-
-## Testing
-
-- WebSocket reliability under network transitions (WiFi → cellular)
-- Feed deduplication when Jetstream replays events
-- Memory usage with long-running WebSocket connections
-- Battery impact measurement on mobile
diff --git a/docs/app/tasks/phase-1.md b/docs/app/tasks/phase-1.md
deleted file mode 100644
index 6aef587..0000000
--- a/docs/app/tasks/phase-1.md
+++ /dev/null
@@ -1,64 +0,0 @@
-# Phase 1 Tasks — Project Shell & Design System
-
-## Scaffold
-
-- [x] Create Ionic Vue project with TypeScript (`ionic start twisted tabs --type vue`)
-- [x] Configure Capacitor for iOS and Android
-- [x] Set up path aliases (`@/` → `src/`)
-- [x] Install and configure Pinia
-- [x] Install and configure TanStack Query for Vue
-- [x] Create the directory structure per spec (`app/`, `core/`, `services/`, `domain/`, `features/`, `components/`)
-
-## Routing & Navigation
-
-- [x] Define five-tab layout: Home, Explore, Activity, Profile (visible tabs) + Repo (pushed route)
-- [x] Configure Vue Router with Ionic tab routing
-- [x] Add route definitions for all Phase 1 placeholder pages
-- [ ] Verify tab-to-tab navigation preserves scroll position and component state
-
-## Domain Models
-
-- [x] Create `domain/models/user.ts` — `UserSummary` type
-- [x] Create `domain/models/repo.ts` — `RepoSummary`, `RepoDetail`, `RepoFile` types
-- [x] Create `domain/models/pull-request.ts` — `PullRequestSummary` type
-- [x] Create `domain/models/issue.ts` — `IssueSummary` type
-- [x] Create `domain/models/activity.ts` — `ActivityItem` type
-
-## Mock Data
-
-- [x] Use realistic data: fetch `desertthunder.dev` to create mock data for repo names, timestamps within last 30 days
-- [x] Create `src/mocks/users.ts` — factory for `UserSummary` instances
-- [x] Create `src/mocks/repos.ts` — factory for `RepoSummary` and `RepoDetail` instances
-- [x] Create `src/mocks/pull-requests.ts` — factory for `PullRequestSummary` instances
-- [x] Create `src/mocks/issues.ts` — factory for `IssueSummary` instances
-- [x] Create `src/mocks/activity.ts` — factory for `ActivityItem` instances
-
-## Design System Components
-
-- [x] `components/common/RepoCard.vue` — compact repo summary (name, owner, description, language, stars)
-- [x] `components/common/UserCard.vue` — avatar + handle + bio snippet
-- [x] `components/common/ActivityCard.vue` — icon + actor + verb + target + relative timestamp
-- [x] `components/common/EmptyState.vue` — icon + message + optional action button
-- [x] `components/common/ErrorBoundary.vue` — catch errors, show retry UI
-- [x] `components/common/SkeletonLoader.vue` — shimmer placeholders (variants: card, list-item, profile)
-- [x] `components/repo/FileTreeItem.vue` — file/dir icon + name
-- [x] `components/repo/MarkdownRenderer.vue` — render markdown to HTML (stub with basic styling)
-
-## Feature Pages (placeholder with mock data)
-
-- [x] `features/home/HomePage.vue` — trending repos list, recent activity list
-- [x] `features/explore/ExplorePage.vue` — search bar (non-functional), repo/user tabs, repo list
-- [x] `features/repo/RepoDetailPage.vue` — segmented layout: Overview, Files, Issues, PRs
-- [x] `features/repo/RepoOverview.vue` — header, description, README placeholder, stats
-- [x] `features/repo/RepoFiles.vue` — file tree list from mock data
-- [x] `features/repo/RepoIssues.vue` — issue list from mock data
-- [x] `features/repo/RepoPRs.vue` — PR list from mock data
-- [x] `features/activity/ActivityPage.vue` — filter chips + activity card list
-- [x] `features/profile/ProfilePage.vue` — sign-in prompt (unauthenticated state)
-
-## Quality
-
-- [ ] Verify all pages render with skeleton loaders before mock data appears
-- [ ] Verify tab switches don't cause layout shift
-- [ ] Run Lighthouse on the web build — target first-paint under 2s
-- [ ] Verify iOS and Android builds compile and launch via Capacitor
diff --git a/docs/app/tasks/phase-2.md b/docs/app/tasks/phase-2.md
deleted file mode 100644
index 91eb6e2..0000000
--- a/docs/app/tasks/phase-2.md
+++ /dev/null
@@ -1,67 +0,0 @@
-# Phase 2 Tasks — Public Tangled Browsing
-
-## Protocol Setup
-
-- [x] Install `@atcute/client` and `@atcute/tangled`
-- [x] Create `services/atproto/client.ts` — singleton XRPC client with configurable base URL
-- [x] Add error interceptor that normalizes XRPC errors into typed app errors
-- [x] Create `core/errors/tangled.ts` — error types: NotFound, NetworkError, MalformedResponse, RateLimited
-
-## API Validation
-
-- [x] Probe `tangled.org` for JSON API endpoints — returns HTML only (no JSON API); all metadata via PDS
-- [x] Confirm knot XRPC endpoints work from browser (CORS check against knot) — `Access-Control-Allow-Origin: *` confirmed on `knot1.tangled.sh`; knot hostname comes from `sh.tangled.repo` PDS record, not a fixed host
-- [x] Document which data comes from knots vs appview vs PDS (see endpoints.ts header comment)
-- [x] Test `com.atproto.repo.getRecord` for fetching user profiles and repo records from PDS — confirmed working on `bsky.social`
-
-## Service Layer
-
-- [x] Create `services/tangled/endpoints.ts` — typed wrappers for each XRPC query
-- [x] Create `services/tangled/normalizers.ts` — transform raw responses → domain models
-- [x] Create `services/tangled/queries.ts` — TanStack Query hooks with cache keys and stale times
-- [x] Implement knot routing: determine correct knot hostname for a given repo
-
-## Repository Browsing
-
-- [x] Wire `RepoDetailPage` to live repo data (metadata from PDS record + git data from knot)
-- [x] Implement repo overview: description, topics, default branch, language breakdown
-- [x] Implement README fetch: `sh.tangled.repo.blob` for `README.md` on default branch
-- [x] Wire `MarkdownRenderer` to render real README content
-- [x] Implement file tree: `sh.tangled.repo.tree` → navigate directories
-- [x] Implement file viewer: `sh.tangled.repo.blob` → syntax-highlighted display
-- [x] Implement commit log: `sh.tangled.repo.log` with cursor pagination
-- [x] Implement branch list: `sh.tangled.repo.branches`
-
-## Profile Browsing
-
-- [x] Fetch user profile from PDS: `com.atproto.repo.getRecord` for `sh.tangled.actor.profile`
-- [x] Display profile: avatar (via `avatar.tangled.sh`), bio, links, location, pronouns, pinned repos
-- [x] List user's repos: fetch `sh.tangled.repo` records from user's PDS
-- [x] Wire `UserCard` component to real data
-
-## Issues (read-only)
-
-- [x] Fetch issues for a repo from PDS records (`listIssueRecords` + `listIssueStateRecords` from owner's PDS)
-- [x] Display issue list with state filter (open/closed)
-- [x] Issue detail view: title, body, author, state
-- [x] Issue comments: fetch `sh.tangled.repo.issue.comment` records, render threaded
-
-## Pull Requests (read-only)
-
-- [x] Fetch PRs for a repo from PDS records (`listPullRecords` + `listPullStatusRecords` from owner's PDS)
-- [x] Display PR list with status filter (open/closed/merged)
-- [x] PR detail view: title, body, author, source/target branches
-- [x] PR comments: fetch `sh.tangled.repo.pull.comment` records
-
-## Caching
-
-- [x] Configure TanStack Query stale/gc times per data type (see spec)
-- [x] Set up IndexedDB query persister for offline reads
-
-## Quality
-
-- [x] Replace default mock-backed Home/Explore/Activity surfaces with scoped-down curated live discovery/activity
-- [ ] Verify stale-while-revalidate behavior: cached data shows immediately, refreshes in background
-- [ ] Test with real Tangled repos (e.g., `tangled.org/core`)
-- [ ] Verify error states render correctly: 404, network failure, empty repos
-- [ ] Test on slow network (throttled devtools) — verify skeleton → content transition
diff --git a/docs/app/tasks/phase-3.md b/docs/app/tasks/phase-3.md
deleted file mode 100644
index d6058eb..0000000
--- a/docs/app/tasks/phase-3.md
+++ /dev/null
@@ -1,60 +0,0 @@
-# Phase 3 Tasks — Search & Activity Feed
-
-## Search API Discovery
-
-- [ ] Probe `tangled.org` for search endpoints (try `/search?q=`, `/api/search`, check network tab on live site)
-- [ ] If JSON search exists, document the request/response format
-- [ ] If no JSON search, decide: curated discovery now + backend search later, or HTML scraping
-
-## Search Implementation
-
-- [ ] Create `services/tangled/search.ts` — search service (real endpoint or fallback)
-- [ ] Implement debounced search input (300ms) in Explore tab
-- [ ] Implement segmented search results: Repos tab, Users tab
-- [ ] Implement search result rendering with `RepoCard` and `UserCard`
-- [ ] Implement empty search state with suggestions
-- [ ] Persist recent searches in local storage (max 20)
-- [ ] Clear search history action
-
-## Discovery Sections (if search API unavailable)
-
-- [ ] Implement "Trending repos" section (source TBD — may require appview scraping or curated list)
-- [ ] Implement "Recently created repos" section
-- [ ] Implement "Suggested users" section
-- [ ] Wire discovery sections into Home and Explore tabs
-
-## Activity Feed — Data Source
-
-- [ ] Investigate `tangled.org/timeline` for JSON variant (check with Accept headers)
-- [ ] If no JSON timeline, evaluate `@atcute/jetstream` for real-time feed
-- [ ] If neither works, implement polling-based feed from known users' PDS records
-- [ ] Document chosen approach and any limitations
-
-## Activity Feed — Implementation
-
-- [ ] Create `services/tangled/feed.ts` — feed data source
-- [ ] Create normalizer: raw AT Protocol events → `ActivityItem` domain model
-- [ ] Implement `ActivityPage` with real feed data
-- [ ] Implement filter chips: All, Repos, PRs, Issues, Social
-- [ ] Implement infinite scroll with cursor-based pagination
-- [ ] Implement pull-to-refresh
-- [ ] Implement tap-to-navigate: activity card → repo/profile/PR/issue detail
-
-## Feed Caching
-
-- [ ] Cache last 100 feed items in IndexedDB via query persister
-- [ ] Show cached feed immediately on tab switch
-- [ ] Stale time: 1 minute
-- [ ] Verify feed persists across app restarts
-
-## Home Tab
-
-- [ ] Wire Home tab to real data: trending repos + recent activity
-- [ ] Add "personalized" section placeholder (shows sign-in prompt when unauthenticated)
-
-## Quality
-
-- [ ] Test search with various queries — verify results are relevant
-- [ ] Test activity feed with pull-to-refresh and pagination
-- [ ] Test offline: cached feed shows, search degrades gracefully
-- [ ] Verify no duplicate items in feed after refresh
diff --git a/docs/app/tasks/phase-4.md b/docs/app/tasks/phase-4.md
deleted file mode 100644
index d8fa6df..0000000
--- a/docs/app/tasks/phase-4.md
+++ /dev/null
@@ -1,84 +0,0 @@
-# Phase 4 Tasks — OAuth & Social Features
-
-## OAuth Setup
-
-- [ ] Install `@atcute/oauth-browser-client`
-- [ ] Host OAuth client metadata JSON at a public URL & configure for local dev
-- [ ] Create `core/auth/oauth.ts` — call `configureOAuth()` with client metadata URL and redirect URI
-- [ ] Create `core/auth/session.ts` — session management: get, list, delete stored sessions
-- [ ] Create `core/auth/store.ts` — Pinia auth store with state machine (idle → authenticating → authenticated → error)
-
-## Login Flow
-
-- [ ] Create `features/profile/LoginPage.vue` — handle input field + "Sign in" button
-- [ ] Implement handle → DID resolution
-- [ ] Implement OAuth redirect initiation
-- [ ] Create `/oauth/callback` route to handle redirect back
-- [ ] Implement token exchange on callback
-- [ ] Store session and update auth store
-- [ ] Redirect to Profile tab after successful login
-- [ ] Handle auth errors: invalid handle, OAuth denied, network failure
-
-## Session Management
-
-- [ ] Implement session restoration on app launch (call `getSession()` for stored DID)
-- [ ] Implement automatic token refresh (handled by `@atcute/oauth-browser-client` internally)
-- [ ] Implement logout: clear session, reset auth store, redirect to Home
-- [ ] Implement account switcher: `listStoredSessions()`, switch between accounts
-
-## Capacitor Deep Links
-
-- [ ] Configure custom URL scheme for OAuth callback on iOS/Android
-- [ ] Add `App.addListener('appUrlOpen')` handler to capture callback
-- [ ] Test OAuth flow on iOS simulator and Android emulator
-
-## Auth-Aware XRPC Client
-
-- [ ] Create authenticated XRPC client that uses `session.dpopFetch` for requests
-- [ ] Service layer: use authenticated client for mutations, public client for queries
-- [ ] Handle 401/expired session: trigger re-auth flow
-
-## Social Actions — Star
-
-- [ ] Create `services/tangled/mutations.ts` — mutation functions
-- [ ] Implement `starRepo(repoAtUri)` — creates `sh.tangled.feed.star` record on user's PDS
-- [ ] Implement `unstarRepo(rkey)` — deletes star record
-- [ ] Add star/unstar button to `RepoDetailPage` overview
-- [ ] Optimistic update: toggle star state and count immediately, rollback on error
-- [ ] Track user's existing stars to show correct initial state
-
-## Social Actions — Follow
-
-- [ ] Implement `followUser(targetDid)` — creates `sh.tangled.graph.follow` record
-- [ ] Implement `unfollowUser(rkey)` — deletes follow record
-- [ ] Add follow/unfollow button to profile pages and user cards
-- [ ] Optimistic update: toggle follow state immediately
-- [ ] Track user's existing follows to show correct initial state
-
-## Social Actions — React
-
-- [ ] Implement `addReaction(subjectUri, reaction)` — creates `sh.tangled.feed.reaction` record
-- [ ] Implement `removeReaction(rkey)` — deletes reaction record
-- [ ] Add reaction button/picker to PR and issue detail views
-- [ ] Show reaction counts grouped by type
-
-## Profile Tab (Authenticated)
-
-- [ ] Wire Profile tab to show current user's profile data
-- [ ] Show pinned repos, stats, starred repos, following list
-- [ ] Add logout button
-- [ ] Add account switcher UI
-
-## Personalized Feed
-
-- [ ] When signed in, filter activity feed to show activity from followed users and starred repos
-- [ ] Add "For You" / "Global" toggle on Activity tab
-- [ ] If appview provides a personalized endpoint, use it; otherwise filter client-side
-
-## Quality
-
-- [ ] Test full OAuth flow: login → browse → star → follow → logout
-- [ ] Test session restoration after app restart
-- [ ] Test on web, iOS simulator, Android emulator
-- [ ] Test error cases: denied OAuth, expired session, failed mutation
-- [ ] Verify optimistic updates roll back correctly on mutation failure
diff --git a/docs/app/tasks/phase-5.md b/docs/app/tasks/phase-5.md
deleted file mode 100644
index 9785225..0000000
--- a/docs/app/tasks/phase-5.md
+++ /dev/null
@@ -1,64 +0,0 @@
-# Phase 5 Tasks — Offline & Performance Polish
-
-## Query Persistence
-
-- [ ] Set up `persistQueryClient` with IndexedDB adapter
-- [ ] Configure persistence: debounced writes, max cache size, TTL per data type
-- [ ] Hydrate query cache from IndexedDB before first render
-- [ ] Verify: kill app → relaunch → cached data appears immediately without network
-
-## Offline Detection
-
-- [ ] Create `core/network/status.ts` — reactive online/offline state (composable)
-- [ ] Show persistent offline banner when `navigator.onLine` is false
-- [ ] Disable mutation buttons (star, follow, react) when offline
-- [ ] Show toast when network returns: "Back online — refreshing"
-
-## Secure Storage
-
-- [ ] Abstract auth token storage behind `core/storage/secure.ts`
-- [ ] Web: encrypted localStorage wrapper
-- [ ] Native: Capacitor Secure Storage plugin
-- [ ] Verify tokens are never stored in IndexedDB query cache
-- [ ] Clear secure storage on logout
-
-## Cache Eviction
-
-- [ ] Implement max-item limits per data type (repos: 200, files: 100, profiles: 100)
-- [ ] Implement TTL eviction (remove entries older than their configured TTL)
-- [ ] Run eviction on app launch and periodically (every 30 min)
-- [ ] Measure IndexedDB size and log warnings if approaching limits
-
-## Navigation Performance
-
-- [ ] Verify Ionic `keep-alive` preserves tab state and scroll position
-- [ ] Implement data prefetch on repo card visibility (Intersection Observer)
-- [ ] Test tab switch speed — should feel instant with cached data
-- [ ] Profile and fix any layout shifts during navigation
-
-## List Virtualization
-
-- [ ] Replace flat lists with virtualized scroll for: repo lists, activity feed, file trees
-- [ ] Test with 1000+ item lists — verify smooth scrolling
-- [ ] Verify scroll position restoration when navigating back
-
-## Image Optimization
-
-- [ ] Lazy-load all avatars
-- [ ] Add initials fallback for missing avatars
-- [ ] Use appropriate image sizes from `avatar.tangled.sh`
-
-## Bundle Optimization
-
-- [ ] Add route-level code splitting (lazy imports per feature)
-- [ ] Tree-shake unused Ionic components (configure Ionic's component imports)
-- [ ] Measure bundle size — target under 500KB initial JS
-- [ ] Run Lighthouse audit — target 90+ performance score on mobile
-
-## Quality
-
-- [ ] Test offline → online transition: data refreshes without duplicates
-- [ ] Test low-bandwidth (3G throttle): skeleton → content transitions are smooth
-- [ ] Test memory usage over extended use: navigate many repos, check heap doesn't grow unbounded
-- [ ] Test on real iOS and Android devices (not just simulators)
-- [ ] Measure and document cold start time, tab switch time, scroll performance
diff --git a/docs/app/tasks/phase-6.md b/docs/app/tasks/phase-6.md
deleted file mode 100644
index 10a34b6..0000000
--- a/docs/app/tasks/phase-6.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Phase 6 Tasks — Write Features & Project Services
-
-## Project Services Setup
-
-- [ ] Decide which write and notification operations belong in `packages/api` versus a separate service
-- [ ] Implement health and readiness endpoints for all public client-facing services
-- [ ] Configure CORS for the mobile app's origins
-- [ ] Document the mobile-facing service contract in `docs/api`
-
-## Project Services — Auth Proxy
-
-- [ ] Implement OAuth token exchange endpoint (if moving auth server-side)
-- [ ] Implement session endpoint that returns user info
-- [ ] Decide: keep client-side OAuth or migrate to service-mediated auth
-
-## Project Services — Search and Graph
-
-- [ ] Implement `GET /search` endpoint for repo/profile discovery
-- [ ] Return enough repo/profile metadata for the mobile client to render result cards directly
-- [ ] Implement `GET /profiles/:did/summary` for follower/following counts and other graph-derived gaps
-- [ ] Wire mobile client's search and profile summary services to these endpoints
-
-## Project Services — Personalized Feed
-
-- [ ] Implement `GET /feed/personalized` — aggregate activity for the user's follows and stars
-- [ ] Index relevant events from Jetstream
-- [ ] Wire mobile client's feed to the project service endpoint when authenticated
-
-## Create Issue
-
-- [ ] Create `features/repo/CreateIssuePage.vue` — title + body (markdown) form
-- [ ] Implement `createIssue()` mutation in `services/tangled/mutations.ts`
-- [ ] Create `sh.tangled.repo.issue` record on user's PDS
-- [ ] Optimistic update: add issue to local list
-- [ ] Navigate to new issue detail on success
-
-## Comment on Issue
-
-- [ ] Add comment input to issue detail view
-- [ ] Implement `createIssueComment()` mutation
-- [ ] Create `sh.tangled.repo.issue.comment` record with `replyTo` support for threading
-- [ ] Optimistic update: append comment to list
-
-## Comment on PR
-
-- [ ] Add comment input to PR detail view
-- [ ] Implement `createPRComment()` mutation
-- [ ] Create `sh.tangled.repo.pull.comment` record
-
-## Issue State Management
-
-- [ ] Add close/reopen button to issue detail (author and repo owner only)
-- [ ] Implement `closeIssue()` / `reopenIssue()` — create `sh.tangled.repo.issue.state` record
-- [ ] Optimistic update: toggle state badge
-
-## Edit Profile
-
-- [ ] Create `features/profile/EditProfilePage.vue`
-- [ ] Implement avatar upload (max 1MB, png/jpeg) via blob upload + record update
-- [ ] Implement bio edit (max 256 graphemes)
-- [ ] Implement links edit (max 5 URIs)
-- [ ] Implement location, pronouns, pinned repos, stats selection, bluesky toggle
-- [ ] Update `sh.tangled.actor.profile` record (key: `self`)
-
-## Scope Upgrade
-
-- [ ] Detect when user's session lacks scopes needed for write operations
-- [ ] Prompt user to re-authorize with expanded scopes
-- [ ] Handle scope upgrade flow gracefully (no data loss)
-
-## Push Notifications (if services exist)
-
-- [ ] Implement `POST /notifications/register` — register device token
-- [ ] Configure Capacitor Push Notifications plugin
-- [ ] Register device token on login
-- [ ] Services: subscribe to events relevant to user, deliver via APNs/FCM
-- [ ] Handle notification tap → deep link to relevant content
-
-## Quality
-
-- [ ] Test issue creation end-to-end: create → verify on tangled.org
-- [ ] Test commenting on issues and PRs
-- [ ] Test profile editing: avatar upload, bio change
-- [ ] Test scope upgrade flow
-- [ ] Verify mutations work offline-queued (if implemented) or show appropriate offline errors
diff --git a/docs/app/tasks/phase-7.md b/docs/app/tasks/phase-7.md
deleted file mode 100644
index f862e71..0000000
--- a/docs/app/tasks/phase-7.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# Phase 7 Tasks — Real-Time Feed & Advanced Features
-
-## Jetstream Integration
-
-- [ ] Install `@atcute/jetstream`
-- [ ] Create `services/atproto/jetstream.ts` — WebSocket connection manager
-- [ ] Filter events for `sh.tangled.*` collections
-- [ ] Normalize events into `ActivityItem` domain model
-- [ ] Merge live events into TanStack Query feed cache
-- [ ] Implement connection lifecycle: connect on foreground, disconnect on background
-- [ ] Implement reconnection with exponential backoff and cursor tracking
-- [ ] Add battery-aware throttling (Capacitor Battery API)
-
-## Live UI Indicators
-
-- [ ] Activity feed: "X new items" pill at top, tap to reveal
-- [ ] Repo detail: "New commits available" banner on ref update events
-- [ ] PR detail: live status badge updates (open → merged)
-- [ ] Issue detail: live comment count updates
-
-## Custom Feeds
-
-- [ ] Create `domain/feed/custom-feed.ts` — feed configuration model
-- [ ] Implement feed builder UI: name + filter rules (by repo, user, event type)
-- [ ] Store custom feeds in IndexedDB
-- [ ] Render custom feed as a selectable tab/option on Activity page
-- [ ] "My repos" preset, "Watching" preset, "Team" preset
-
-## Repo Forking
-
-- [ ] Add fork button to repo detail (authenticated only)
-- [ ] Implement fork creation via `sh.tangled.repo.create` with `source` field
-- [ ] Show fork status badge: up-to-date, fast-forwardable, conflict, missing branch
-- [ ] Implement "Sync fork" action via `sh.tangled.repo.forkSync`
-
-## Labels
-
-- [ ] Fetch label definitions for a repo
-- [ ] Display color-coded label chips on issues and PRs
-- [ ] Implement label filtering on issue/PR lists
-- [ ] Add/remove labels on issues and PRs (authenticated, with label scopes)
-
-## Expanded Reactions
-
-- [ ] Add reaction picker component: thumbsup, thumbsdown, laugh, tada, confused, heart, rocket, eyes
-- [ ] Show grouped reaction counts on PRs, issues, and comments
-- [ ] Add/remove reactions with optimistic updates
-
-## PR Interdiff
-
-- [ ] Detect PR round count
-- [ ] Add round selector to PR detail
-- [ ] Fetch and display diff between selected rounds
-- [ ] Use `sh.tangled.repo.compare` for cross-round comparison
-
-## Knot Info
-
-- [ ] Show knot hostname on repo detail
-- [ ] Fetch knot version via `sh.tangled.knot.version`
-- [ ] Display knot status/health indicator
-
-## Quality
-
-- [ ] Test Jetstream under network transitions (WiFi → cellular → offline → online)
-- [ ] Verify no duplicate events after reconnection with cursor
-- [ ] Measure battery impact of WebSocket connection on iOS and Android
-- [ ] Test memory usage with long-running Jetstream connection
-- [ ] Load test: simulate high-frequency events, verify UI stays responsive
diff --git a/docs/qa.md b/docs/qa.md
deleted file mode 100644
index ea48e4d..0000000
--- a/docs/qa.md
+++ /dev/null
@@ -1,77 +0,0 @@
----
-title: "QA Checklist"
-updated: 2026-03-23
----
-
-# QA Checklist
-
-## Ingestion (end-to-end)
-
-Walk a record through the full pipeline: Tap event → indexer → store → searchable.
-
-- [ ] Indexer connects to Tap via WebSocket and begins processing events
-- [ ] Creating a tracked record on Tangled produces a row in `documents`
-- [ ] Updating that record changes the existing row (new CID)
-- [ ] Deleting that record tombstones the row (`deleted_at` set)
-- [ ] Tombstoned documents do not appear in search results
-- [ ] Identity events update the handle cache; new documents show resolved handles
-- [ ] Unsupported collections are silently skipped (no errors logged)
-- [ ] Connection drop triggers automatic reconnect and resumes from last cursor
-
-## Cursor durability
-
-- [ ] Kill the indexer mid-stream, restart — processing resumes without duplicating documents
-- [ ] Redeploy the indexer — cursor is persisted before shutdown, no gap or replay
-
-## Backfill
-
-Run `twister backfill` against a small seed file and verify the discovery graph.
-
-- [ ] Seed file with known Tangled users produces a non-empty discovery graph
-- [ ] `--max-hops 1` limits discovery to direct follows/collaborators only
-- [ ] `--dry-run` logs the plan but does not call Tap mutation endpoints
-- [ ] Already-tracked DIDs are reported and not re-submitted
-- [ ] Re-running the same seeds is idempotent
-- [ ] After backfill + Tap sync, search returns historical content that wasn't there before
-
-## Search API
-
-- [ ] `GET /search?q=` returns the expected repo as top result
-- [ ] Searching by title keyword returns expected documents
-- [ ] Searching by author handle returns their content
-- [ ] `collection`, `type`, `author`, `repo` filters restrict results correctly
-- [ ] Pagination: `offset=0&limit=5` then `offset=5&limit=5` return disjoint result sets
-- [ ] Missing `q` param returns 400 with error JSON
-- [ ] Unknown query param returns 400
-- [ ] `GET /documents/{id}` returns the full document; 404 for missing or tombstoned
-- [ ] `GET /healthz` returns 200
-- [ ] `GET /readyz` returns 503 when DB is unreachable
-
-## Deployment (Railway)
-
-- [ ] API service healthy and routable at public URL
-- [ ] Indexer service healthy on `:9090/health`
-- [ ] A new Tangled record ingested post-deploy becomes searchable within seconds
-- [ ] Redeploying the API preserves availability (health-check-gated rollout)
-- [ ] Restarting the indexer does not lose sync position
-- [ ] Environment variables match the documented set in `docs/api/deploy.md`
-
-## Mobile — Navigation & Shell
-
-- [ ] All five tabs render and switch without layout shift
-- [ ] Tab-to-tab navigation preserves scroll position and component state
-- [ ] Pages show skeleton loaders before data appears
-- [ ] iOS and Android builds compile and launch via Capacitor
-
-## Mobile — Live Tangled Browsing
-
-- [ ] Repo detail page loads metadata from PDS + git data from knot
-- [ ] README renders via markdown renderer
-- [ ] File tree navigates directories; file viewer shows syntax-highlighted content
-- [ ] Commit log paginates with cursor
-- [ ] Profile page shows avatar, bio, and repos from PDS
-- [ ] Issue list filters by state (open/closed); detail shows body + threaded comments
-- [ ] PR list filters by status; detail shows source/target branches + comments
-- [ ] Stale-while-revalidate: cached data shows immediately, refreshes in background
-- [ ] Error states render correctly: 404, network failure, empty repo
-- [ ] Slow network: skeleton → content transition is smooth (test with throttled devtools)
diff --git a/docs/reference/api.md b/docs/reference/api.md
new file mode 100644
index 0000000..101b7f1
--- /dev/null
+++ b/docs/reference/api.md
@@ -0,0 +1,161 @@
+---
+title: API Service Reference
+updated: 2026-03-24
+---
+
+Twister is a Go service that indexes Tangled content and serves a search API. It connects to the AT Protocol ecosystem via Tap (firehose consumer), XRPC (direct record lookups), and Constellation (backlink queries), storing indexed data in Turso/libSQL with FTS5 full-text search.
+
+## Architecture
+
+The service is a single Go binary with multiple subcommands, each running a different runtime mode. All modes share the same database and configuration layer.
+
+**Runtime modes:**
+
+| Command       | Purpose                                                             |
+| ------------- | ------------------------------------------------------------------- |
+| `api` (serve) | HTTP search API server                                              |
+| `indexer`     | Consumes Tap firehose events, normalizes and indexes records        |
+| `backfill`    | Discovers users from seed files, registers them with Tap            |
+| `enrich`      | Backfills missing metadata (repo names, handles, web URLs) via XRPC |
+| `reindex`     | Re-syncs all documents into the FTS index                           |
+| `healthcheck` | One-shot liveness probe for container orchestration                 |
+
+The `embed-worker` and `reembed` commands exist as stubs for the upcoming semantic search pipeline (Nomic Embed Text v1.5 deployed via Railway template).
+
+All commands accept a `--local` flag that switches to a local SQLite file and text-format logging for development.
+
+## HTTP API
+
+The API server binds to `:8080` by default (configurable via `HTTP_BIND_ADDR`). CORS is open (`*` origin, GET/OPTIONS).
+
+### Search
+
+**`GET /search`** — Main search endpoint. Routes to keyword, semantic, or hybrid based on `mode` parameter.
+
+**`GET /search/keyword`** — Full-text search via FTS5 with BM25 scoring.
+
+Parameters:
+
+- `q` (required) — Query string
+- `limit` (1–100, default 20) — Results per page
+- `offset` (default 0) — Pagination offset
+- `collection` — Filter by AT Protocol collection NSID
+- `type` — Filter by record type (repo, issue, pull, profile, string)
+- `author` — Filter by handle or DID
+- `repo` — Filter by repo name or DID
+- `language` — Filter by primary language
+- `from`, `to` — Date range (ISO 8601)
+- `state` — Filter issues/PRs by state (open, closed, merged)
+- `mode` — Search mode (keyword, semantic, hybrid)
+
+Response includes query metadata, total count, and an array of results each containing: ID, collection, record type, title, summary, body snippet (with `` highlights), score, repo name, author handle, DID, AT-URI, web URL, and timestamps.
+
+**`GET /documents/{id}`** — Fetch a single document by stable ID.
+
+### Health
+
+- **`GET /healthz`** — Liveness probe, always 200
+- **`GET /readyz`** — Readiness probe, pings database
+
+### Admin
+
+When `ENABLE_ADMIN_ENDPOINTS=true` with a configured `ADMIN_AUTH_TOKEN`:
+
+- **`POST /admin/reindex`** — Trigger FTS re-sync
+
+### Static Content
+
+The API also serves a search site with live search and API documentation at `/` and `/docs*`, built with Alpine.js (no build step, embedded in `internal/view/`).
+
+## Database
+
+Turso (libSQL) with the following tables:
+
+**documents** — Core search index. Each record gets a stable ID of `did|collection|rkey`. Stores title, body, summary, metadata (repo name, author handle, web URL, language, tags), and timestamps. Soft-deleted via `deleted_at`.
+
+**documents_fts** — FTS5 virtual table for full-text search over title, body, summary, repo name, author handle, and tags. Uses `unicode61` tokenizer with tuned BM25 weights (title weighted highest at 2.5, then author handle at 2.0, summary at 1.5).
+
+**sync_state** — Cursor tracking for the Tap consumer. Stores consumer name, current cursor, high water mark, and last update time. Enables crash-safe resume.
+
+**identity_handles** — DID-to-handle cache. Updated from Tap identity events and XRPC lookups.
+
+**record_state** — Issue and PR state cache (open/closed/merged). Keyed by subject AT-URI.
+
+**document_embeddings** — Vector storage (768-dim F32_BLOB with DiskANN cosine index). Schema ready but not yet populated.
+
+**embedding_jobs** — Async embedding job queue. Schema ready but worker not yet active.
+
+## Indexing Pipeline
+
+The indexer connects to Tap via WebSocket, consuming AT Protocol record events in real-time. For each event:
+
+1. Filter against the configured collection allowlist (supports wildcards like `sh.tangled.*`)
+2. Route to the appropriate normalizer based on collection
+3. Normalize into a document (extract title, body, summary, metadata)
+4. Optionally enrich via XRPC (resolve author handle, repo name, web URL)
+5. Upsert into the database (auto-syncs FTS)
+6. Advance cursor and acknowledge to Tap
+
+The indexer resumes from its last cursor on restart (no duplicate processing). It logs status every 30 seconds and uses exponential backoff (1s–5s) for transient failures.
+
+## Record Normalizers
+
+Each AT Protocol collection has a dedicated normalizer that extracts searchable content:
+
+| Collection                      | Record Type   | Searchable               | Content                     |
+| ------------------------------- | ------------- | ------------------------ | --------------------------- |
+| `sh.tangled.repo`               | repo          | Yes (if named)           | Name, description, topics   |
+| `sh.tangled.repo.issue`         | issue         | Yes                      | Title, body, repo reference |
+| `sh.tangled.repo.pull`          | pull          | Yes                      | Title, body, target branch  |
+| `sh.tangled.repo.issue.comment` | issue_comment | Yes (if has body)        | Comment body                |
+| `sh.tangled.repo.pull.comment`  | pull_comment  | Yes (if has body)        | Comment body                |
+| `sh.tangled.string`             | string        | Yes (if has content)     | Filename, contents          |
+| `sh.tangled.actor.profile`      | profile       | Yes (if has description) | Profile description         |
+| `sh.tangled.graph.follow`       | follow        | No                       | Graph edge only             |
+
+State records (`sh.tangled.repo.issue.state`, `sh.tangled.repo.pull.status`) update the `record_state` table rather than creating documents.
+
+## XRPC Client
+
+The built-in XRPC client provides typed access to AT Protocol endpoints with caching (1-hour TTL for DID docs and repo names):
+
+- DID resolution via PLC Directory (`did:plc:`) or `.well-known/did.json` (`did:web:`)
+- Identity resolution (PDS endpoint + handle from DID document)
+- Record fetching (`com.atproto.repo.getRecord`, `com.atproto.repo.listRecords`)
+- Repo name resolution from `sh.tangled.repo` records
+- Web URL construction for Tangled entities
+
+## Backfill
+
+The backfill command discovers users from a seed file and registers them with Tap for indexing. Discovery fans out via follow graphs and repo collaborators up to a configurable hop depth (default 2). Supports dry-run mode, configurable concurrency and batch sizes, and is idempotent.
+
+## Configuration
+
+All configuration is via environment variables (with `.env` file support):
+
+| Variable                   | Default                 | Purpose                                        |
+| -------------------------- | ----------------------- | ---------------------------------------------- |
+| `TURSO_DATABASE_URL`       | —                       | Database connection (required)                 |
+| `TURSO_AUTH_TOKEN`         | —                       | Auth token (required for remote)               |
+| `TAP_URL`                  | —                       | Tap WebSocket URL                              |
+| `TAP_AUTH_PASSWORD`        | —                       | Tap admin password                             |
+| `INDEXED_COLLECTIONS`      | all                     | Collection allowlist (CSV, supports wildcards) |
+| `HTTP_BIND_ADDR`           | `:8080`                 | API server bind address                        |
+| `INDEXER_HEALTH_ADDR`      | `:9090`                 | Indexer health probe address                   |
+| `LOG_LEVEL`                | info                    | debug/info/warn/error                          |
+| `LOG_FORMAT`               | json                    | json or text                                   |
+| `ENABLE_ADMIN_ENDPOINTS`   | false                   | Enable admin routes                            |
+| `ADMIN_AUTH_TOKEN`         | —                       | Bearer token for admin                         |
+| `ENABLE_INGEST_ENRICHMENT` | true                    | XRPC enrichment at ingest time                 |
+| `PLC_DIRECTORY_URL`        | `https://plc.directory` | PLC Directory                                  |
+| `XRPC_TIMEOUT`             | 15s                     | XRPC HTTP timeout                              |
+
+## Deployment
+
+Deployed on Railway with three services:
+
+- **api** — HTTP server (port 8080, health at `/healthz`)
+- **indexer** — Tap consumer (health at `:9090/healthz`)
+- **tap** — Tap instance (external dependency)
+
+All services share the same Turso database. The API and indexer are separate deployments of the same binary with different subcommands.
diff --git a/docs/reference/app.md b/docs/reference/app.md
new file mode 100644
index 0000000..f534191
--- /dev/null
+++ b/docs/reference/app.md
@@ -0,0 +1,78 @@
+---
+title: Mobile App Reference
+updated: 2026-03-24
+---
+
+Twisted is an Ionic Vue mobile app for browsing Tangled, a git hosting platform built on the AT Protocol. It targets iOS and Android via Capacitor (no web target).
+
+## Tech Stack
+
+- **Vue 3** with TypeScript and Composition API
+- **Ionic Vue** for native-feeling UI components
+- **Capacitor** for iOS/Android builds
+- **Pinia** for state management
+- **TanStack Query** for async data with caching
+- **@atcute/client** and **@atcute/tangled** for AT Protocol XRPC
+
+TypeScript files use `.js` extensions in imports. Package management via pnpm.
+
+## Architecture
+
+Three-layer design:
+
+**Presentation** — Vue components and pages using Ionic's component library. Five-tab navigation: Home, Explore, Activity, Profile (visible tabs) plus Repo (pushed route). Repo detail uses segmented tabs: Overview, Files, Issues, PRs.
+
+**Domain** — TypeScript types modeling the app's data: UserSummary, RepoSummary, RepoDetail, RepoFile, PullRequestSummary, IssueSummary, ActivityItem. These are app-internal representations, decoupled from API response shapes.
+
+**Data** — Service layer that fetches from external sources and normalizes into domain types. The flow is: Vue component → composable → TanStack Query hook → service function → XRPC call → normalizer → domain model.
+
+## Directory Structure
+
+```sh
+src/
+  app/          — App shell, router, global config
+  core/         — Shared utilities, constants
+  services/     — API clients and data fetching
+    atproto/    — @atcute client setup, error handling
+    tangled/    — Endpoints, normalizers, TanStack Query hooks
+  domain/       — TypeScript type definitions
+  features/     — Feature modules (home, explore, repo, etc.)
+  components/   — Shared UI components
+```
+
+## Data Sources
+
+The app reads from multiple sources depending on what's needed:
+
+- **Knots** (Tangled XRPC servers) — Git data: file trees, blobs, commits, branches, diffs. Each repo is hosted on a specific knot.
+- **PDS** (Personal Data Servers) — AT Protocol records: profiles, issues, PRs, comments, stars, follows. Accessed via `com.atproto.repo.getRecord` and `com.atproto.repo.listRecords`.
+- **Twister API** — Search and index-backed summaries (when available).
+- **Constellation** — Social signal counts and backlinks (stars, followers, reactions).
+
+Knots serve XRPC endpoints for git operations. The appview at `tangled.org` returns HTML only (no JSON API), so the app goes directly to knots for git data and PDS for AT Protocol records.
+
+## Completed Features
+
+### Navigation & Shell (Phase 1)
+
+Five-tab layout with Vue Router, skeleton loaders, placeholder pages. Design system components: RepoCard, UserCard, ActivityCard, FileTreeItem, EmptyState, ErrorBoundary, SkeletonLoader, MarkdownRenderer.
+
+### Public Browsing (Phase 2)
+
+All read-only browsing works without authentication:
+
+**Repository browsing** — Metadata display, README rendering (markdown), file tree navigation, file viewer with syntax context, commit log with pagination, branch listing.
+
+**Profile browsing** — Avatar, bio, links fetched from PDS. User's repos listed.
+
+**Issues** — List view with open/closed state filter, detail view with threaded comments.
+
+**Pull Requests** — List view with status filter (open/closed/merged), detail view with comments.
+
+**Caching** — TanStack Query configured with per-data-type stale times. Persistence via Dexie (IndexedDB) — works in Capacitor's WebView on device and in the browser during local dev.
+
+## Routing
+
+The app resolves identities through AT Protocol: handle → DID (via PDS resolution) → records. For repo git data, the knot hostname is extracted from the repo's DID document.
+
+Home tab currently provides direct handle-based browsing: enter a known handle to view their profile and repos. This works without any index or search dependency.
diff --git a/docs/reference/lexicons.md b/docs/reference/lexicons.md
new file mode 100644
index 0000000..8ddcf4a
--- /dev/null
+++ b/docs/reference/lexicons.md
@@ -0,0 +1,138 @@
+---
+title: Tangled Lexicons
+updated: 2026-03-24
+---
+
+Tangled defines its AT Protocol record types under the `sh.tangled.*` namespace. These are the records stored on users' Personal Data Servers (PDS) and consumed by the indexing pipeline.
+
+## Searchable Records
+
+### sh.tangled.repo
+
+Repository metadata. Created when a user registers a repo with Tangled.
+
+- `name` (string, required) — Repository name
+- `description` (string) — Short description
+- `createdAt` (datetime) — Creation timestamp
+- `knot` (string) — Knot DID hosting the git data
+- `topics` (array of strings) — Tags/topics
+
+### sh.tangled.repo.issue
+
+Issue on a repository.
+
+- `repo` (at-uri, required) — Reference to the parent repo record
+- `title` (string, required) — Issue title
+- `body` (string) — Issue body (markdown)
+- `createdAt` (datetime)
+
+### sh.tangled.repo.pull
+
+Pull request on a repository.
+
+- `repo` (at-uri, required) — Reference to the parent repo record
+- `title` (string, required) — PR title
+- `body` (string) — PR body (markdown)
+- `head` (string) — Source branch
+- `base` (string) — Target branch
+- `createdAt` (datetime)
+
+### sh.tangled.string
+
+Code snippet or gist.
+
+- `filename` (string) — File name with extension
+- `contents` (string, required) — Code content
+- `language` (string) — Programming language
+- `createdAt` (datetime)
+
+### sh.tangled.actor.profile
+
+User profile information.
+
+- `displayName` (string) — Display name
+- `description` (string) — Bio/about text
+- `avatar` (blob) — Profile image
+- `pronouns` (string) — Pronouns
+- `location` (string) — Location
+- `links` (array of strings) — External links
+- `pinnedRepos` (array of at-uri) — Pinned repository references
+
+## Interaction Records
+
+### sh.tangled.feed.star
+
+Star/favorite on a repository.
+
+- `subject` (object, required) — `{ uri: at-uri, cid: cid }` referencing the starred repo
+
+### sh.tangled.graph.follow
+
+Follow relationship between users.
+
+- `subject` (did, required) — DID of the followed user
+- `createdAt` (datetime)
+
+### sh.tangled.feed.reaction
+
+Emoji reaction on content.
+
+- `subject` (object, required) — `{ uri: at-uri, cid: cid }` referencing the target
+- `emoji` (string, required) — Reaction emoji
+
+## State Records
+
+### sh.tangled.repo.issue.state
+
+Tracks whether an issue is open or closed.
+
+- `issue` (at-uri, required) — Reference to the issue
+- `state` (string, required) — `open` or `closed`
+
+### sh.tangled.repo.pull.status
+
+Tracks pull request lifecycle.
+
+- `pull` (at-uri, required) — Reference to the PR
+- `status` (string, required) — `open`, `closed`, or `merged`
+
+## Comment Records
+
+### sh.tangled.repo.issue.comment
+
+Comment on an issue.
+
+- `issue` (at-uri, required) — Reference to the parent issue
+- `body` (string, required) — Comment body (markdown)
+- `parent` (at-uri) — Parent comment for threading
+- `createdAt` (datetime)
+
+### sh.tangled.repo.pull.comment
+
+Comment on a pull request.
+
+- `pull` (at-uri, required) — Reference to the parent PR
+- `body` (string, required) — Comment body (markdown)
+- `parent` (at-uri) — Parent comment for threading
+- `createdAt` (datetime)
+
+## Infrastructure Records
+
+### sh.tangled.knot.member
+
+Knot membership record.
+
+- `knot` (did, required) — Knot DID
+- `permission` (string) — Permission level
+
+### sh.tangled.knot.version
+
+Knot software version metadata.
+
+## Stable ID Format
+
+Documents in the search index use the stable ID format: `did|collection|rkey` (e.g., `did:plc:abc123|sh.tangled.repo|repo-name`). This ensures idempotent upserts regardless of CID changes.
+
+## AT-URI Format
+
+Records are addressed as `at://did/collection/rkey` (e.g., `at://did:plc:abc123/sh.tangled.repo/repo-name`).
diff --git a/docs/roadmap.md b/docs/roadmap.md
new file mode 100644
index 0000000..4da90de
--- /dev/null
+++ b/docs/roadmap.md
@@ -0,0 +1,126 @@
+---
+title: Roadmap
+updated: 2026-03-24
+---
+
+## API: Constellation Integration
+
+Add a Constellation client to the Go API for enriching search results with social signals.
+
+- [ ] Constellation XRPC client (`internal/constellation/`) with `getBacklinksCount` and `getBacklinks`
+- [ ] User-agent header with project name and contact
+- [ ] Enrich search results with star counts from Constellation
+- [ ] Profile summary endpoint (`GET /profiles/{did}/summary`) with follower/following counts from Constellation
+- [ ] Cache Constellation responses with short TTL (star/follower counts change infrequently)
+
+## API: Semantic Search Pipeline
+
+Nomic Embed Text v1.5 via Railway template, async embedding pipeline.
+
+- [ ] Deploy nomic-embed Railway template (`POST /api/embeddings` with Bearer auth)
+- [ ] Embedding client in Go API (`internal/embedding/`) calling the Nomic service
+- [ ] Embed-worker: consume `embedding_jobs` queue, generate 768-dim vectors, store in `document_embeddings`
+- [ ] `GET /search/semantic` endpoint using DiskANN vector_top_k
+- [ ] Reembed command for bulk re-generation
+
+## API: Hybrid Search
+
+Combine keyword and semantic results.
+
+- [ ] Score normalization (keyword BM25 → [0,1], semantic cosine → [0,1])
+- [ ] Weighted merge (0.65 keyword + 0.35 semantic, configurable)
+- [ ] Deduplication by document ID
+- [ ] `matched_by` metadata in results
+
+## API: Search Quality
+
+- [ ] Field weight tuning based on real queries
+- [ ] Recency boost for recently updated content
+- [ ] Star count ranking signal (via Constellation)
+- [ ] State filtering defaults (exclude closed issues)
+- [ ] Better snippets with longer context
+- [ ] Relevance test fixtures
+
+## API: Observability
+
+- [ ] Structured metrics: ingestion rate, search latency, embedding throughput
+- [ ] Dashboard or log-based monitoring
+
+## App: Search & Discovery
+
+Wire the Explore tab to the search API and add activity feed.
+
+**Depends on:** API: Constellation Integration
+
+- [ ] Search service pointing at Twister API
+- [ ] Constellation service for star/follower counts
+- [ ] Debounced search on Explore tab with segmented results
+- [ ] Recent search history (local)
+- [ ] Graceful fallback when search API unavailable
+- [ ] Activity feed data source investigation (Jetstream vs polling)
+- [ ] Activity tab with filters, infinite scroll, pull-to-refresh
+- [ ] Home tab: surface recently viewed repos/profiles
+
+## App: Authentication & Social
+
+Bluesky OAuth and authenticated actions.
+
+**Depends on:** App: Search & Discovery (for Constellation service), API: Constellation Integration
+
+- [ ] OAuth setup with `@atcute/oauth-browser-client`
+- [ ] Login page, OAuth flow, callback handling
+- [ ] Capacitor deep link configuration
+- [ ] Session management (restore, refresh, logout, account switcher)
+- [ ] Auth-aware XRPC client using dpopFetch
+- [ ] Star repos (write to PDS, count from Constellation)
+- [ ] Follow users (write to PDS, count from Constellation)
+- [ ] React to content (write to PDS, count from Constellation)
+- [ ] Authenticated profile tab (pinned repos, stats, starred, following)
+- [ ] Personalized feed ("For You" / "Global" toggle)
+
+## App: Write Features
+
+**Depends on:** App: Authentication & Social
+
+- [ ] Create issue (title + markdown body)
+- [ ] Comment on issues and PRs (threaded)
+- [ ] Close/reopen issues
+- [ ] Edit profile (bio, links, avatar, pinned repos)
+- [ ] OAuth scope upgrade flow
+
+## App: Offline & Performance
+
+**Depends on:** App: Search & Discovery (for cache persistence of search/feed data)
+
+- [ ] Dexie setup with database schema (query cache + pinned content tables)
+- [ ] TanStack Query persister backed by Dexie
+- [ ] Pinned content store (save/unsave files for offline reading)
+- [ ] Pinned files UI (list, pin/unpin actions on file viewer, last-fetched timestamp)
+- [ ] Offline detection and banner
+- [ ] Secure token storage (Capacitor Secure Storage)
+- [ ] Cache eviction (per-type limits and TTL, pinned content exempt)
+- [ ] List virtualization for large datasets
+- [ ] Lazy-load avatars, prefetch on hover
+- [ ] Code splitting and bundle optimization (target <500KB JS)
+
+## App: Real-Time & Advanced
+
+**Depends on:** App: Authentication & Social, App: Offline & Performance
+
+- [ ] Jetstream integration for live `sh.tangled.*` events
+- [ ] Live UI indicators (new commits, new feed items, PR status)
+- [ ] Custom feed presets ("My repos", "Watching", "Team")
+- [ ] Repo forking
+- [ ] Labels (display, filter, manage)
+- [ ] Expanded reactions with emoji picker
+- [ ] PR interdiff (compare rounds)
+- [ ] Knot info display
+
+## App: Push Notifications
+
+**Depends on:** App: Authentication & Social
+
+- [ ] Register device token on login
+- [ ] Subscribe to relevant events
+- [ ] Deliver via APNs/FCM
+- [ ] Handle notification taps (deep link to relevant screen)
diff --git a/docs/specs/app-features.md b/docs/specs/app-features.md
new file mode 100644
index 0000000..a578e3b
--- /dev/null
+++ b/docs/specs/app-features.md
@@ -0,0 +1,176 @@
+---
+title: App Features
+updated: 2026-03-24
+---
+
+## Search & Discovery
+
+**Depends on:** Search API (Twister), Constellation API
+
+### Search
+
+- Create search service pointing at Twister API
+- Debounced search input on Explore tab
+- Segmented results: repos, users, issues/PRs
+- Recent search history (local storage, clearable)
+- Graceful fallback when search API is unavailable
+
+### Discovery Sections
+
+- Explore tab shows search prominently
+- Optional: trending repos or recently active repos (if data supports it)
+- Profile summaries enriched with Constellation data (star counts, follower counts)
+
+### Home Tab
+
+- Handle-based direct browsing (already works)
+- Surface recently viewed repos/profiles from local history
+- Optional: personalized suggestions for signed-in users (later)
+
+### Activity Feed
+
+- Investigate data sources: Jetstream, polling PDS, or Twister-aggregated feed
+- Activity tab shows recent events from followed users and starred repos
+- Filters by event type (commits, issues, PRs, stars)
+- Infinite scroll with pull-to-refresh
+
+## Authentication & Social
+
+**Depends on:** Bluesky OAuth, Constellation API
+
+### OAuth Sign-In
+
+- Install `@atcute/oauth-browser-client`
+- Host client metadata JSON with required scopes
+- Login page: handle input → resolution → OAuth redirect → callback
+- Capacitor deep link handling for native redirect
+- Session restoration on app launch, automatic token refresh
+- Logout, account switcher for multiple accounts
+- Auth state: idle → authenticating → authenticated → error
+
+### Social Actions
+
+All social actions are AT Protocol record writes to the user's PDS. Counts come from Constellation.
+
+- **Star:** Create/delete `sh.tangled.feed.star` record. Show star count via Constellation `getBacklinksCount`.
+- **Follow:** Create/delete `sh.tangled.graph.follow` record. Show follower count via Constellation.
+- **React:** Create `sh.tangled.feed.reaction` record. Show reaction counts via Constellation.
+- Optimistic UI updates via TanStack Query mutation + cache invalidation.
+
+### Authenticated Profile
+
+- Profile tab shows current user's data when signed in
+- Pinned repos, stats (repos, stars, followers via Constellation)
+- Starred repos list
+- Following/followers lists (via Constellation `getBacklinks`)
+- Settings and logout
+
+### Personalized Feed
+
+- Filter activity feed to followed users and starred repos
+- "For You" / "Global" toggle on activity tab
+
+## Write Features
+
+**Depends on:** Authentication
+
+### Issues
+
+- Create issue: title + markdown body, posted as `sh.tangled.repo.issue` record
+- Comment on issue: threaded comments as `sh.tangled.repo.issue.comment` records
+- Close/reopen: create `sh.tangled.repo.issue.state` record
+
+### Pull Requests
+
+- Comment on PR: `sh.tangled.repo.pull.comment` records
+
+### Profile Editing
+
+- Edit bio, links, location, pronouns, pinned repos
+- Avatar upload (max 1MB, png/jpeg)
+- Cross-posting toggle
+- Posted as updated `sh.tangled.actor.profile` record
+
+### OAuth Scope Upgrade
+
+- Detect when an action requires a scope not yet granted
+- Prompt user to re-authorize with expanded scopes
+
+## Offline & Performance
+
+### Local Storage
+
+All local persistence uses **Dexie** over IndexedDB. This works natively in Capacitor's WebView on both iOS and Android, and in the browser during local development — no platform branching or plugins needed.
+
+Three storage layers, each with a distinct purpose:
+
+- **TanStack Query persister (Dexie-backed)** — Automatic cache persistence. Previously-viewed data hydrates on launch and serves from cache when offline. Subject to normal cache eviction (stale times, GC).
+- **Pinned content store (Dexie)** — User-initiated "save for offline" storage for files, READMEs, and other reference content. Exempt from cache eviction — only the user removes pinned items. Stores file content, metadata, repo handle, pinned timestamp.
+- **Capacitor Preferences** — Small key-value settings (theme, recent search history, feed preferences).
+- **Capacitor Secure Storage** — Auth tokens only. Never in Dexie or the query cache.
+
+### Offline Behavior
+
+- TanStack Query serves cached data when offline (stale-while-revalidate)
+- Pinned files always available regardless of connectivity
+- Offline detection via `navigator.onLine`, persistent banner
+- Mutations disabled when offline
+- Background refresh when connectivity returns
+
+### Pinned Files
+
+Users can pin/save references to files for offline reading:
+
+- Pin action on file viewer saves content + metadata to the Dexie pinned store
+- Pinned files list accessible from profile or a dedicated section
+- Content persists until the user explicitly unpins
+- Pinned items show last-fetched timestamp; refresh when online
+
+### Cache Management
+
+- Per-type limits: repo metadata (200 items/7 days), file trees (50/3 days), profiles (100/7 days), search results (20/1 day)
+- Eviction on app launch and periodically
+- Pinned content exempt from eviction
+- Measure and cap IndexedDB usage
+
+### Performance
+
+- Prefetch on hover/visibility for likely navigation targets
+- Virtualized lists for large datasets (1000+ items)
+- Lazy-load avatars with initials fallback
+- Route-level code splitting
+- Tree-shake Ionic components
+- Target: under 500KB JS, shell first-paint under 2s
+
+## Real-Time & Advanced
+
+**Depends on:** Authentication, Activity Feed
+
+### Jetstream Integration
+
+- Connect to Jetstream for real-time `sh.tangled.*` events
+- Filter and normalize into ActivityItem, merge into TanStack Query cache
+- Connect on foreground, disconnect on background
+- Cursor tracking for gap-free resume
+- Battery-aware throttling
+
+### Live UI Indicators
+
+- "New commits" banner in repo detail
+- "X new items" pill on activity feed
+- Live status updates on PR detail
+- Issue comment count updates
+
+### Custom Feeds
+
+- Presets: "My repos", "Watching", "Team"
+- Feed builder UI for custom filters
+- Local storage in IndexedDB
+
+### Advanced Features
+
+- **Repo forking:** Create repo with source field, show fork status, sync action
+- **Labels:** Display color-coded chips, filter by label, add/remove with auth
+- **Expanded reactions:** Emoji picker, grouped counts, add/remove
+- **PR interdiff:** Compare rounds via `sh.tangled.repo.compare`
+- **Knot info:** Show hostname, version, health status on repo detail
diff --git a/docs/specs/data-sources.md b/docs/specs/data-sources.md
new file mode 100644
index 0000000..675623b
--- /dev/null
+++ b/docs/specs/data-sources.md
@@ -0,0 +1,154 @@
+---
+title: Data Sources & Integration
+updated: 2026-03-24
+---
+
+Twisted pulls data from four external sources and authenticates users via Bluesky OAuth. Each source has a distinct role — no single source is authoritative for everything.
+
+## Source Overview
+
+| Source                   | What it provides                                                               | Access pattern                                             |
+| ------------------------ | ------------------------------------------------------------------------------ | ---------------------------------------------------------- |
+| **Tangled XRPC (Knots)** | Git data — file trees, blobs, commits, branches, diffs, tags                   | Direct XRPC calls to the knot hosting each repo            |
+| **AT Protocol (PDS)**    | User records — profiles, repos, issues, PRs, comments, stars, follows          | `com.atproto.repo.getRecord` / `listRecords` on user's PDS |
+| **Constellation**        | Social signals — star counts, follower counts, reaction counts, backlink lists | Public JSON API at `constellation.microcosm.blue`          |
+| **Tap**                  | Real-time firehose of AT Protocol record events for indexing                   | WebSocket consumer, feeds our search index                 |
+
+## Constellation
+
+[Constellation](https://constellation.microcosm.blue) is a public, self-hosted index of AT Protocol backlinks. It answers "who linked to this?" across the entire network — making it the right source for aggregated social signals instead of maintaining our own counters.
+
+### Key Endpoints
+
+**`GET /xrpc/blue.microcosm.links.getBacklinks`** — Get records linking to a target.
+
+- `subject` (required) — The target (AT-URI, DID, or URL)
+- `source` (required) — Collection and path, e.g. `sh.tangled.feed.star:subject.uri`
+- `did` — Filter to specific users (repeatable)
+- `limit` — Default 16, max 100
+- `reverse` — Reverse ordering
+
+**`GET /xrpc/blue.microcosm.links.getBacklinksCount`** — Count of links to a target.
+
+- `subject`, `source` — Same as above
+
+**`GET /xrpc/blue.microcosm.links.getManyToManyCounts`** — Secondary link counts in many-to-many relationships.
+
+- `subject`, `source`, `pathToOther` (required)
+- `did`, `otherSubject`, `limit` (optional)
+
+### Usage in Twisted
+
+| Need                      | Constellation call                                                                       |
+| ------------------------- | ---------------------------------------------------------------------------------------- |
+| Star count for a repo     | `getBacklinksCount(subject=repo_at_uri, source=sh.tangled.feed.star:subject.uri)`        |
+| Who starred a repo        | `getBacklinks(subject=repo_at_uri, source=sh.tangled.feed.star:subject.uri)`             |
+| Follower count for a user | `getBacklinksCount(subject=user_did, source=sh.tangled.graph.follow:subject)`            |
+| Who follows a user        | `getBacklinks(subject=user_did, source=sh.tangled.graph.follow:subject)`                 |
+| Reaction count on content | `getBacklinksCount(subject=content_at_uri, source=sh.tangled.feed.reaction:subject.uri)` |
+
+This replaces the need to index and count interaction records ourselves. Our Tap pipeline still indexes interaction records for search and graph discovery, but Constellation is the source of truth for counts and lists.
+
+### Integration Notes
+
+- No authentication required. Constellation asks for a user-agent header with project name and contact.
+- Responses are paginated via cursor. Plan for multiple pages when listing (e.g., all followers).
+- The API is read-only — social actions (star, follow, react) are still AT Protocol record writes to the user's PDS.
+
+## Tangled XRPC (Knots)
+
+Knots are Tangled's git hosting servers. Each repo lives on a specific knot, identified by the knot DID in the repo's AT Protocol record.
+
+### Endpoints Used
+
+- `sh.tangled.repo.tree` — File tree for a ref
+- `sh.tangled.repo.blob` — File content
+- `sh.tangled.repo.log` — Commit history
+- `sh.tangled.repo.branches` / `sh.tangled.repo.tags` — Refs
+- `sh.tangled.repo.getDefaultBranch` — Default branch name
+- `sh.tangled.repo.diff` / `sh.tangled.repo.compare` — Diffs
+- `sh.tangled.repo.languages` — Language breakdown
+- `sh.tangled.knot.version` — Knot software version
+
+### Routing
+
+The app resolves which knot hosts a repo by reading the repo's AT Protocol record (which contains the knot DID), then resolving the knot DID to its service endpoint. XRPC calls go directly to that knot.
+
+The Tangled appview at `tangled.org` serves HTML only — there is no JSON API at the appview level.
+
+## AT Protocol (PDS)
+
+Standard AT Protocol record access for reading and writing user data.
+
+### Read Operations
+
+- `com.atproto.repo.getRecord` — Fetch a single record by collection + rkey
+- `com.atproto.repo.listRecords` — List records in a collection with pagination
+
+Used for: profiles, repo metadata, issues, PRs, comments, stars, follows, reactions.
+
+### Write Operations (Authenticated)
+
+- `com.atproto.repo.createRecord` — Create a new record (star, follow, react, issue, comment)
+- `com.atproto.repo.deleteRecord` — Delete a record (unstar, unfollow)
+
+All writes go to the authenticated user's PDS using their OAuth session.
+
+### Identity Resolution
+
+- Handle → DID via `com.atproto.identity.resolveHandle`
+- DID → DID document via PLC Directory (`plc.directory`) or `.well-known/did.json`
+- DID document → PDS endpoint (from `#atprotoPersonalDataServer` service)
+
+## Tap (Firehose)
+
+Tap provides a filtered firehose of AT Protocol events. Our indexer consumes Tap via WebSocket, indexing records into the search database.
+
+### What We Index via Tap
+
+- Repos, issues, PRs, comments, strings, profiles — for full-text search
+- Follows — for graph discovery during backfill
+- Issue state and PR status changes — for state filtering in search
+
+### What We Don't Need to Count via Tap
+
+Stars, followers, reactions — Constellation handles counts and lists. We still process these events for graph discovery but don't need to maintain our own counters.
+
+### Tap Protocol
+
+- WebSocket connection with cursor-based resume
+- Events contain: operation (create/update/delete), DID, collection, rkey, CID, record payload
+- Acks required after processing each event
+- Backfill via `/repos/add` endpoint to request historical data for specific users
+
+## Bluesky OAuth
+
+Authentication uses AT Protocol OAuth via `@atcute/oauth-browser-client`.
+
+### Flow
+
+1. User enters their handle
+2. App resolves handle → DID → PDS → authorization server metadata
+3. App initiates OAuth with requested scopes
+4. User authorizes in browser, redirected back to app
+5. App exchanges code for tokens
+6. Session provides `dpopFetch` for authenticated XRPC calls
+
+### Scopes
+
+The app requests scopes for:
+
+- `sh.tangled.feed.star` — Star/unstar repos
+- `sh.tangled.graph.follow` — Follow/unfollow users
+- `sh.tangled.feed.reaction` — Add reactions
+- `sh.tangled.actor.profile` — Edit profile
+- `sh.tangled.repo.issue` / `sh.tangled.repo.issue.comment` — Create issues and comments
+- `sh.tangled.repo.pull.comment` — Comment on PRs
+
+### Capacitor Integration
+
+On native platforms, OAuth callback uses a deep link URL scheme registered with Capacitor. The app listens via `App.addListener('appUrlOpen', ...)` to catch the redirect.
+
+### Session Management
+
+Tokens are stored in secure storage (encrypted localStorage on web, Capacitor Secure Storage on native). Sessions auto-refresh. The app supports multiple accounts with an account switcher.
diff --git a/docs/specs/search.md b/docs/specs/search.md
new file mode 100644
index 0000000..0b51497
--- /dev/null
+++ b/docs/specs/search.md
@@ -0,0 +1,116 @@
+---
+title: Search
+updated: 2026-03-24
+---
+
+Search lets users find repos, issues, PRs, profiles, and code snippets across the Tangled network. The API supports three modes with progressive capability.
+
+## Modes
+
+### Keyword Search (Implemented)
+
+Full-text search powered by SQLite FTS5 with BM25 scoring. Queries are tokenized, matched against title, body, summary, repo name, author handle, and tags. Results are ranked by relevance with field-specific weights (title highest, then author handle, summary, body).
+
+Snippets are generated from the body field with match terms wrapped in `` tags.
+
+### Semantic Search (Planned)
+
+Vector similarity search using **Nomic Embed Text v1.5**, deployed on Railway via the [nomic-embed template](https://railway.com/deploy/nomic-embed). The template runs Ollama behind an authenticated Caddy proxy.
+
+**Embedding service:**
+
+- Model: `nomic-embed-text:latest` (8192-token context, 768-dimensional vectors, Matryoshka support for variable dimensionality)
+- Endpoint: `POST /api/embeddings` with Bearer token auth
+- Request: `{ "model": "nomic-embed-text:latest", "prompt": "text to embed" }`
+- Deployed as a separate Railway service alongside the API and indexer
+
+**Pipeline:**
+
+- The embed-worker consumes the `embedding_jobs` queue, calls the Nomic Embed service, and stores 768-dim vectors in the `document_embeddings` table
+- Documents are embedded asynchronously after indexing — the embed-worker runs independently of the ingestion loop
+- Search queries are embedded at request time (single prompt, low latency)
+- Vectors are matched via DiskANN cosine similarity index in Turso
+
+### Hybrid Search (Planned)
+
+Weighted combination of keyword and semantic results. Default blend: 0.65 keyword + 0.35 semantic (configurable). Scores are normalized to [0, 1] before blending. Results are deduplicated by document ID with the higher score retained. Each result includes a `matched_by` field indicating which mode(s) contributed.
+
+## API Contract
+
+**`GET /search`** — Unified endpoint, routes by `mode` parameter.
+
+### Parameters
+
+| Param        | Required | Default | Description                           |
+| ------------ | -------- | ------- | ------------------------------------- |
+| `q`          | Yes      | —       | Query string                          |
+| `mode`       | No       | keyword | keyword, semantic, or hybrid          |
+| `limit`      | No       | 20      | Results per page (1–100)              |
+| `offset`     | No       | 0       | Pagination offset                     |
+| `collection` | No       | —       | Filter by collection NSID             |
+| `type`       | No       | —       | Filter by record type                 |
+| `author`     | No       | —       | Filter by handle or DID               |
+| `repo`       | No       | —       | Filter by repo name or DID            |
+| `language`   | No       | —       | Filter by primary language            |
+| `from`       | No       | —       | Created after (ISO 8601)              |
+| `to`         | No       | —       | Created before (ISO 8601)             |
+| `state`      | No       | —       | Issue/PR state (open, closed, merged) |
+
+### Response
+
+```json
+{
+  "query": "tangled vue",
+  "mode": "keyword",
+  "total": 42,
+  "limit": 20,
+  "offset": 0,
+  "results": [
+    {
+      "id": "did:plc:abc|sh.tangled.repo|my-repo",
+      "collection": "sh.tangled.repo",
+      "record_type": "repo",
+      "title": "my-repo",
+      "summary": "A Vue component library",
+      "body_snippet": "...building Vue components for Tangled...",
+      "score": 4.82,
+      "matched_by": ["keyword"],
+      "repo_name": "my-repo",
+      "author_handle": "alice.bsky.social",
+      "did": "did:plc:abc",
+      "at_uri": "at://did:plc:abc/sh.tangled.repo/my-repo",
+      "web_url": "https://tangled.sh/alice.bsky.social/my-repo",
+      "created_at": "2026-01-15T10:00:00Z",
+      "updated_at": "2026-03-20T14:30:00Z"
+    }
+  ]
+}
+```
+
+## Pragmatic Search Strategy
+
+Indexing via Tap is useful but has proven unreliable for maintaining complete, up-to-date coverage. The approach:
+
+1. **Keyword search is the foundation.** It works now and covers the primary use case — finding repos, issues, and people by name or content.
+
+2. **Constellation supplements search results.** Star counts and follower counts from Constellation can be used as ranking signals without needing to index interaction records ourselves.
+
+3. **Semantic search is additive.** It improves discovery for vague queries but isn't required for the app to be useful. It ships when the embedding pipeline is stable.
+
+4. **Graceful degradation.** The mobile app treats the search API as optional. If Twister is unavailable, handle-based direct browsing still works. Search results link into the same browsing screens.
+
+## Quality Improvements (Planned)
+
+- Field weight tuning based on real query patterns
+- Recency boost for recently updated content
+- Collection-aware ranking (repos weighted higher for short queries)
+- Star count as a ranking signal (via Constellation)
+- State filtering (exclude closed issues by default)
+- Better snippet generation with longer context windows
+- Relevance test fixtures for regression testing
+
+## Mobile Integration
+
+The app calls the search API from the Explore tab. Results are displayed in segmented views (repos, users, issues/PRs). Each result links to the corresponding browsing screen (repo detail, profile, issue detail).
+
+When the search API is unavailable, the Explore tab shows an appropriate state rather than breaking. The Home tab's handle-based browsing is fully independent of search.
diff --git a/docs/todo.md b/docs/todo.md
new file mode 100644
index 0000000..198180b
--- /dev/null
+++ b/docs/todo.md
@@ -0,0 +1,7 @@
+---
+title: Parking Lot
+updated: 2026-03-24
+---
+
+- Constellation requests would be a good opportunity to dispatch a job to index what
+  was requested.