/healthz | Liveness | Restart the process |
/readyz | Readiness | Remove from load balancer |
diff --git a/packages/api/internal/view/templates/docs/identity.html b/packages/api/internal/view/templates/docs/identity.html
new file mode 100644
index 0000000..79b6f60
--- /dev/null
+++ b/packages/api/internal/view/templates/docs/identity.html
@@ -0,0 +1,9 @@
+{{define "title"}}Identity — Twister API{{end}}
+{{define "content"}}
+Identity
+Identity endpoints resolve ATProto identifiers. Both routes proxy upstream
+services verbatim, making them safe to call from browser clients without
+additional CORS configuration.
+{{range .}}{{template "doc-entry" .}}{{end}}
+{{end}}
+{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/docs/index.html b/packages/api/internal/view/templates/docs/index.html
index 8913820..056162f 100644
--- a/packages/api/internal/view/templates/docs/index.html
+++ b/packages/api/internal/view/templates/docs/index.html
@@ -1,34 +1,50 @@
{{define "title"}}API Docs — Twister{{end}}
{{define "content"}}
API Documentation
-Twister exposes a public JSON API for searching indexed Tangled content. No authentication is required for read endpoints.
+Twister exposes a public JSON API for searching indexed
+Tangled content.
+No authentication is required for read endpoints.
Base URL
https://<your-twister-domain>
-All endpoints are relative to the base URL. When using the search site, the API is on the same origin.
+All endpoints are relative to the base URL. When using the search site, the
+API is on the same origin.
-Response Shape
-All responses are JSON. Successful responses return the resource directly. Errors return:
-{
- "error": "error_code",
- "message": "Human-readable description"
-}
+Response shape
+All responses are JSON. Successful responses return the resource directly.
+Errors return:
+{"error": "error_code", "message": "what went wrong"}
Endpoints
- | Method | Path | Description |
|---|
+ | Resource | Description |
|---|
- GET | /search | Search indexed documents |
- GET | /documents/{id} | Fetch a single document |
- GET | /healthz | Liveness probe |
- GET | /readyz | Readiness probe |
+ | Search |
+ FTS5 keyword search over indexed documents |
+ | Documents |
+ Fetch a single indexed document by stable ID |
+ | Actors & Repos |
+ Actor profiles, repo records, and repo sub-resources |
+ | Issues |
+ Issue detail and comments with pre-joined state |
+ | Pull Requests |
+ PR detail and comments with pre-joined status |
+ | Identity |
+ Handle resolution and DID document lookup |
+ | Activity |
+ Cached Jetstream events and real-time WebSocket stream |
+ | Profiles & Backlinks |
+ Social signals and Constellation backlinks counts |
+ | XRPC Proxy |
+ Pass-through proxy to knots, PDS hosts, and Bluesky |
+ | Health |
+ Liveness and readiness probes |
Pagination
-List endpoints use limit and offset query parameters. The default limit is 20, maximum is 100.
-
-Filtering
-Search supports filters via query parameters: collection, type, author, repo, language, state, from, to. See the search endpoint docs for details.
+List endpoints use limit and offset query
+parameters. Defaults and maximums vary per endpoint — see each resource page
+for details.
{{end}}
{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/docs/issues.html b/packages/api/internal/view/templates/docs/issues.html
new file mode 100644
index 0000000..1d8fae5
--- /dev/null
+++ b/packages/api/internal/view/templates/docs/issues.html
@@ -0,0 +1,9 @@
+{{define "title"}}Issues — Twister API{{end}}
+{{define "content"}}
+Issues
+Issue endpoints fetch records and comments from an actor’s PDS.
+Responses include a pre-joined state field derived from
+sh.tangled.repo.issue.state records.
+{{range .}}{{template "doc-entry" .}}{{end}}
+{{end}}
+{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/docs/profiles.html b/packages/api/internal/view/templates/docs/profiles.html
new file mode 100644
index 0000000..fc15cd5
--- /dev/null
+++ b/packages/api/internal/view/templates/docs/profiles.html
@@ -0,0 +1,9 @@
+{{define "title"}}Profiles & Backlinks — Twister API{{end}}
+{{define "content"}}
+Profiles & Backlinks
+These endpoints surface derived social data from Constellation, the external
+graph service. Both are best-effort: counts default to zero when Constellation
+is not configured or unavailable.
+{{range .}}{{template "doc-entry" .}}{{end}}
+{{end}}
+{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/docs/pulls.html b/packages/api/internal/view/templates/docs/pulls.html
new file mode 100644
index 0000000..0614dc5
--- /dev/null
+++ b/packages/api/internal/view/templates/docs/pulls.html
@@ -0,0 +1,10 @@
+{{define "title"}}Pull Requests — Twister API{{end}}
+{{define "content"}}
+Pull Requests
+Pull request endpoints fetch records and comments from an actor’s PDS.
+Responses include a pre-joined status field derived from
+sh.tangled.repo.pull.status records:
+open, closed, or merged.
+{{range .}}{{template "doc-entry" .}}{{end}}
+{{end}}
+{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/docs/search.html b/packages/api/internal/view/templates/docs/search.html
index 55f08fa..937266d 100644
--- a/packages/api/internal/view/templates/docs/search.html
+++ b/packages/api/internal/view/templates/docs/search.html
@@ -1,56 +1,12 @@
-{{define "title"}}GET /search — Twister API{{end}}
+{{define "title"}}Search — Twister API{{end}}
{{define "content"}}
-GET /search
-Search indexed Tangled documents with keyword matching. Returns a paginated list of results with highlighted snippets.
-
-Parameters
-
- | Name | Type | Default | Description |
|---|
-
- q | string | required | Search query. Supports boolean operators, phrases, and prefix matching. |
- mode | string | keyword | Search mode: keyword, semantic, or hybrid. |
- limit | int | 20 | Results per page (1–100). |
- offset | int | 0 | Pagination offset. |
- collection | string | — | Filter by ATProto collection NSID. |
- type | string | — | Filter by record type: repo, issue, pull, profile, string. |
- author | string | — | Filter by author handle or DID. |
- repo | string | — | Filter by repo name. |
- language | string | — | Filter by programming language. |
- state | string | — | Filter by state: open, closed, merged. |
- from | string | — | Created after (ISO 8601). |
- to | string | — | Created before (ISO 8601). |
-
-
-
-Example Request
-curl "https://your-domain/search?q=rust+tui&type=repo&limit=5"
-
-Example Response
-{
- "query": "rust tui",
- "mode": "keyword",
- "total": 12,
- "limit": 5,
- "offset": 0,
- "results": [
- {
- "id": "did:plc:abc|sh.tangled.repo|3kb3fge5lm32x",
- "collection": "sh.tangled.repo",
- "record_type": "repo",
- "title": "glow-rs",
- "body_snippet": "A <mark>TUI</mark> markdown viewer written in <mark>Rust</mark>...",
- "summary": "Rust TUI markdown viewer",
- "repo_name": "glow-rs",
- "author_handle": "desertthunder.dev",
- "score": 0.842,
- "created_at": "2026-03-20T10:00:00Z",
- "updated_at": "2026-03-22T15:03:11Z"
- }
- ]
-}
-
-Query Syntax
-The keyword search uses Tantivy query syntax:
+Search
+Search endpoints run FTS5 keyword queries over indexed Tangled documents.
+Results include highlighted body snippets and BM25 relevance scores. If
+Constellation is configured, repository results are enriched with star counts.
+{{range .}}{{template "doc-entry" .}}{{end}}
+Query syntax
+The keyword search uses SQLite FTS5 query syntax:
| Feature | Example |
|---|
@@ -61,12 +17,11 @@
| Field-scoped | title:parser |
-
Errors
| Status | Condition |
|---|
- | 400 | Missing q, invalid limit/offset, unknown parameters. |
+ | 400 | Missing q, invalid limit/offset, or unknown parameter. |
| 500 | Internal search failure. |
diff --git a/packages/api/internal/view/templates/docs/xrpc.html b/packages/api/internal/view/templates/docs/xrpc.html
new file mode 100644
index 0000000..3a1b815
--- /dev/null
+++ b/packages/api/internal/view/templates/docs/xrpc.html
@@ -0,0 +1,10 @@
+{{define "title"}}XRPC — Twister API{{end}}
+{{define "content"}}
+XRPC
+XRPC endpoints provide a single-origin pass-through to Tangled knots, PDS
+hosts, and the Bluesky public API. All query parameters are forwarded verbatim.
+Responses are streamed back unchanged, making these suitable for browser clients
+that cannot make direct cross-origin requests.
+{{range .}}{{template "doc-entry" .}}{{end}}
+{{end}}
+{{template "layout" .}}
diff --git a/packages/api/internal/view/templates/partials/doc-entry.html b/packages/api/internal/view/templates/partials/doc-entry.html
new file mode 100644
index 0000000..a2d609f
--- /dev/null
+++ b/packages/api/internal/view/templates/partials/doc-entry.html
@@ -0,0 +1,38 @@
+{{define "doc-entry"}}
+
+ {{.Method}} {{.Route}}
+ {{.Summary}}
+ {{if .Details}}{{.Details}}
{{end}}
+ {{if .PathParams}}
+ Path parameters
+
+ | Name | Type | Description |
|---|
+
+ {{range .PathParams}}
+
+ {{.Name}} |
+ {{.Type}} |
+ {{.Description}} |
+
+ {{end}}
+
+
+ {{end}}
+ {{if .QueryParams}}
+ Query parameters
+
+ | Name | Type | Required | Description |
|---|
+
+ {{range .QueryParams}}
+
+ {{.Name}} |
+ {{.Type}} |
+ {{if .Required}}yes{{else}}—{{end}} |
+ {{.Description}} |
+
+ {{end}}
+
+
+ {{end}}
+
+{{end}}
diff --git a/packages/api/internal/view/view.go b/packages/api/internal/view/view.go
index 1c27b7e..3651e8a 100644
--- a/packages/api/internal/view/view.go
+++ b/packages/api/internal/view/view.go
@@ -7,22 +7,39 @@ import (
"net/http"
)
-//go:embed templates static
+//go:embed templates static api-docs.json
var content embed.FS
var pageTemplates map[string]*template.Template
func init() {
pageTemplates = make(map[string]*template.Template)
+
for _, name := range []string{
"index.html",
+ } {
+ pageTemplates[name] = template.Must(template.New("layout").ParseFS(content,
+ "templates/layout.html",
+ "templates/"+name,
+ ))
+ }
+
+ for _, name := range []string{
"docs/index.html",
"docs/search.html",
"docs/documents.html",
"docs/health.html",
+ "docs/actors.html",
+ "docs/issues.html",
+ "docs/pulls.html",
+ "docs/identity.html",
+ "docs/activity.html",
+ "docs/profiles.html",
+ "docs/xrpc.html",
} {
pageTemplates[name] = template.Must(template.New("layout").ParseFS(content,
"templates/layout.html",
+ "templates/partials/doc-entry.html",
"templates/"+name,
))
}
@@ -35,9 +52,16 @@ func Handler() http.Handler {
staticFS, _ := fs.Sub(content, "static")
mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServerFS(staticFS)))
- mux.HandleFunc("GET /docs/search", renderPage("docs/search.html"))
- mux.HandleFunc("GET /docs/documents", renderPage("docs/documents.html"))
- mux.HandleFunc("GET /docs/health", renderPage("docs/health.html"))
+ mux.HandleFunc("GET /docs/search", renderDocPage("docs/search.html", "search"))
+ mux.HandleFunc("GET /docs/documents", renderDocPage("docs/documents.html", "documents"))
+ mux.HandleFunc("GET /docs/health", renderDocPage("docs/health.html", "health"))
+ mux.HandleFunc("GET /docs/actors", renderDocPage("docs/actors.html", "actors"))
+ mux.HandleFunc("GET /docs/issues", renderDocPage("docs/issues.html", "issues"))
+ mux.HandleFunc("GET /docs/pulls", renderDocPage("docs/pulls.html", "pulls"))
+ mux.HandleFunc("GET /docs/identity", renderDocPage("docs/identity.html", "identity"))
+ mux.HandleFunc("GET /docs/activity", renderDocPage("docs/activity.html", "activity"))
+ mux.HandleFunc("GET /docs/profiles", renderDocPage("docs/profiles.html", "profiles"))
+ mux.HandleFunc("GET /docs/xrpc", renderDocPage("docs/xrpc.html", "proxy"))
mux.HandleFunc("GET /docs", renderPage("docs/index.html"))
mux.HandleFunc("GET /{$}", renderPage("index.html"))
@@ -51,10 +75,23 @@ func renderPage(name string) http.HandlerFunc {
http.NotFound(w, r)
return
}
-
w.Header().Set("Content-Type", "text/html; charset=utf-8")
if err := tmpl.ExecuteTemplate(w, "layout", nil); err != nil {
http.Error(w, "template error", http.StatusInternalServerError)
}
}
}
+
+func renderDocPage(name, page string) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
+ tmpl, ok := pageTemplates[name]
+ if !ok {
+ http.NotFound(w, r)
+ return
+ }
+ w.Header().Set("Content-Type", "text/html; charset=utf-8")
+ if err := tmpl.ExecuteTemplate(w, "layout", PageDocs(page)); err != nil {
+ http.Error(w, "template error", http.StatusInternalServerError)
+ }
+ }
+}
diff --git a/packages/api/overview.md b/packages/api/overview.md
new file mode 100644
index 0000000..3e35ee2
--- /dev/null
+++ b/packages/api/overview.md
@@ -0,0 +1,144 @@
+# Twister Backend Overview
+
+Twister is a Go backend service that indexes Tangled network content and serves
+search queries for the Twisted mobile client. It complements Tangled's public
+APIs by providing global search and derived data that's hard to compute client-side.
+
+## Core Responsibilities
+
+1. **Real-time Indexing**: Consumes record changes from Tap (Tangled's event
+ stream) and indexes them into SQLite FTS5 for full-text search.
+
+2. **Search API**: Exposes HTTP endpoints for keyword search over repos,
+ profiles, issues, pull requests, and follows.
+
+3. **Graph Augmentation**: Caches follower relationships and provides profile
+ summaries not easily derivable from the public knot/PDS APIs.
+
+## Architecture
+
+```text
+┌─────────────┐ ┌─────────────┐ ┌─────────────┐
+│ Tap │────▶│ Indexer │────▶│ Turso │
+│ (event stream) │ (ingest) │ │ (SQLite) │
+└─────────────┘ └─────────────┘ └──────┬──────┘
+ │
+┌─────────────┐ ┌─────────────┐ │
+│ Twisted │────▶│ API │◀───────────┘
+│ (mobile) │ │ (search) │
+└─────────────┘ └─────────────┘
+```
+
+## Commands
+
+The `twister` binary provides six subcommands:
+
+- `api` / `serve`: HTTP server for search and proxy endpoints
+- `indexer`: Tap consumer that processes record events into the index
+- `backfill`: Discovers users from seeds and registers repos with Tap
+- `reindex`: Rebuilds the FTS index from existing documents
+- `enrich`: Backfills derived fields (handles, URLs) on existing documents
+- `healthcheck`: One-shot health probe
+
+## Data Flow
+
+### Indexing Pipeline (internal/ingest)
+
+1. **Tap Consumer** (`internal/tapclient`): Connects to Tap WebSocket, reads
+ record events with cursor-based resume.
+
+2. **Normalizers** (`internal/normalize`): Collection-specific adapters convert
+ raw records into normalized `Document` structs. Each collection (repos,
+ issues, PRs, follows, profiles) has a dedicated adapter.
+
+3. **Store** (`internal/store`): Upserts documents into SQLite with FTS5 index.
+ Tracks cursor position in `sync_state` table for resume-after-restart.
+
+### Search (internal/search)
+
+- **Keyword Search**: SQLite FTS5 full-text search with BM25 ranking
+- **Filters**: By collection, author, repo, language, date range, state
+- **Results**: Denormalized documents with snippet highlighting
+
+## Key Packages
+
+| Package | Purpose |
+| ------------------------ | ------------------------------------------ |
+| `internal/api` | HTTP router, handlers, middleware |
+| `internal/store` | Database access layer (Turso/local SQLite) |
+| `internal/ingest` | Tap event processing pipeline |
+| `internal/normalize` | Record adapters for each collection type |
+| `internal/search` | FTS5 query execution |
+| `internal/backfill` | Graph discovery and Tap registration |
+| `internal/xrpc` | XRPC client for Tangled PDS/knot APIs |
+| `internal/constellation` | External service for graph queries |
+
+## API Endpoints
+
+### Search
+
+- `GET /search` - Unified search across all collections
+- `GET /search/keyword` - FTS5 keyword search
+
+### Actors & Repos
+
+- `GET /actors/{handle}` - Profile data
+- `GET /actors/{handle}/repos` - User's repositories
+- `GET /actors/{handle}/repos/{repo}` - Repo details (tree, log, branches)
+
+### Issues & Pulls
+
+- `GET /issues/{handle}/{rkey}` - Issue detail
+- `GET /pulls/{handle}/{rkey}` - Pull request detail
+
+### Identity
+
+- `GET /identity/resolve` - Resolve handle to DID
+- `GET /identity/did/{did}` - DID document lookup
+
+### Proxy
+
+- `GET /xrpc/knot/{knot}/{nsid}` - Proxy to Tangled knot
+- `GET /xrpc/pds/{pds}/{nsid}` - Proxy to PDS
+- `GET /xrpc/bsky/{nsid}` - Proxy to Bluesky
+
+## Database Schema
+
+Core tables in Turso/libSQL:
+
+- `documents`: Denormalized search documents (title, body, metadata)
+- `documents_fts`: FTS5 virtual table for full-text search
+- `identity_handles`: DID → handle mapping with active status
+- `record_state`: Issue/PR state (open/closed/merged)
+- `sync_state`: Cursor tracking for Tap consumer resume
+- `indexing_jobs`: Queue for async read-through indexing
+
+## Configuration
+
+Environment variables (loaded by `internal/config`):
+
+| Variable | Purpose |
+| --------------------- | ------------------------------------------ |
+| `HTTP_BIND_ADDR` | API server address (default `:8080`) |
+| `TURSO_DATABASE_URL` | libsql:// URL for Turso or file: for local |
+| `TURSO_AUTH_TOKEN` | Turso auth token (empty for local) |
+| `TAP_URL` | Tap WebSocket URL (required for indexer) |
+| `TAP_AUTH_PASSWORD` | Tap authentication password |
+| `INDEXED_COLLECTIONS` | Comma-separated allowlist (empty = all) |
+
+## Local Development
+
+```bash
+# API only (uses local SQLite file)
+just api-dev
+
+# Indexer only
+just api-run-indexer
+
+# Both in parallel (three terminals)
+pnpm dev # Frontend
+pnpm api:run:api # API
+pnpm api:run:indexer # Indexer
+```
+
+For remote Turso, use `--remote` flag or set `TURSO_DATABASE_URL` in `.env`.