Effem AppView #

The AT Protocol AppView behind the Effem podcast app, written in Swift on Hummingbird 2.

It reads xyz.effem.* records from Jetstream into Postgres, serves the xyz.effem.* XRPC read routes, proxies the Podcast Index API with a server-side key, and sends APNs notifications. It replaces the Go service in effem/AppView; the database schema, the migration ledger, the JSON wire contract and the metric names are unchanged, so a database migrated by the Go service comes up with only 0011 and 0012 to apply.

It is deployed to Vercel as a native Swift function with swift-vercel-runtime: the same binary, cross-compiled with the Static Linux SDK, with Jetstream read in scheduled runs instead of by a resident process. See "Deploying to Vercel" below and effem/swift-appview.md section 14 for the reasoning.

Layout #

Path	What it holds
Sources/App	The effem-appview executable: reads configuration and runs the server.
Sources/EffemAppView/Config	AppConfig, the database URL parser and the Go-style validation rules.
Sources/EffemAppView/Database	The PostgresNIO wrapper, the SQL migrator and the statement splitter.
Sources/EffemAppView/Jetstream	The Jetstream reader, its bounded cron runs, the ingest lease, cursor store and host failover.
Sources/EffemAppView/Indexer	Record validation, per-collection indexing, stats, notifications, backfill and the watchdog.
Sources/EffemAppView/Catalog	The canonical episode and podcast records, their record keys and CIDs.
Sources/EffemAppView/Identity	DID document resolution and the handle cache.
Sources/EffemAppView/PodcastIndex	The Podcast Index gateway: cache, in-flight sharing, budgets and stale copies.
Sources/EffemAppView/Push	APNs clients and the push dispatcher.
Sources/EffemAppView/API	Request contexts, middleware, the XRPC routes, the cron routes and the comment WebSocket.
Sources/EffemAppView/Observability	Prometheus metrics and the health report, which is read from the database.
Sources/EffemAppView/Resources	Migrations, the lexicon JSON files and the bundled category list.
Tests/EffemAppViewTests	Swift Testing suites; the Database suite needs Postgres.
scripts/vercel-build.sh, vercel.json	The Vercel build (Static Linux SDK) and the cron schedules.

The models shared with the clients live in the EffemModels product of EffemKit. AT Protocol syntax, DAG-CBOR, CIDs, the Jetstream frame types and DID resolution come from CoreATProtocol. The Podcast Index client is PodcastIndexKit.

Running locally #

Requirements: Swift 6.4 and Docker (OrbStack works).

docker compose up -d                       # Postgres on localhost:5433
cp .env.example .env                       # then edit the tokens
swift run effem-appview
curl localhost:8080/_health

Locally the process reads Jetstream itself (EFFEM_INGEST_MODE=service). To exercise the cron path the way Vercel drives it, set EFFEM_INGEST_MODE=cron and CRON_SECRET=dev in .env and call the route:

curl -H 'Authorization: Bearer dev' localhost:8080/_cron/ingest

Configuration #

Settings are read from command line flags, then the environment, then a .env file in the working directory. Environment names are EFFEM_ plus the key in upper snake case. A few platform names are read as fallbacks so a Vercel project with a Neon database needs no renaming: PORT, DATABASE_URL_UNPOOLED, DATABASE_URL and CRON_SECRET.

Environment variable	Default	Meaning
EFFEM_BIND	0.0.0.0:$PORT, else 0.0.0.0:8080	Listen address as host:port, :port or a bare port.
EFFEM_DATABASE_URL	DATABASE_URL_UNPOOLED, then DATABASE_URL, then the local effem database	postgres://user:password@host:port/db?sslmode=.... sslmode=disable is only accepted for local hosts and .internal names. Prefer a direct connection over a transaction pooler: the migrator's advisory lock needs a session.
EFFEM_DATABASE_LISTEN_URL	DATABASE_URL_UNPOOLED, then the database URL	The one connection that LISTENs for comment events. Must not go through a transaction pooler.
EFFEM_DB_MAX_OPEN_CONNS	25	Connection pool size.
EFFEM_DB_MAX_IDLE_CONNS	5	Idle connections kept open.
EFFEM_DB_CONN_MAX_IDLE_TIME	5m	Idle connection lifetime, Go duration syntax.
EFFEM_JETSTREAM_HOSTS	the two Bluesky hosts	Comma-separated wss:// hosts, tried in order.
EFFEM_INGEST_MODE	cron when VERCEL=1, else service	service: a Jetstream reader runs inside the process. cron: GET /_cron/ingest performs one bounded run per call.
EFFEM_INGEST_RUN_SECONDS	50	How long one cron run stays connected (5 to 1700).
CRON_SECRET	unset	Bearer token the /_cron routes require; without it they answer 503. Vercel sends it with scheduled requests.
EFFEM_PLC_HOST	https://plc.directory	PLC directory for DID resolution.
EFFEM_CATALOG_DID	did:web:catalog.effem.app	Authority of the catalog records.
EFFEM_LABELER_DID	did:web:labeler.effem.app	Authority of moderation labels.
EFFEM_PI_KEY, EFFEM_PI_SECRET	unset	Podcast Index credentials. The Podcast Index routes answer 503 until both are set.
EFFEM_PI_USER_AGENT	effem-appview/1.0	Sent on every Podcast Index call.
EFFEM_AUTH_REQUIRED	true	When false, anonymous callers get read scope.
EFFEM_AUTH_READ_TOKENS	none	token=subject pairs; the subject is a DID or * for a service token.
EFFEM_AUTH_ADMIN_TOKENS	none	Same format; grants the admin scope.
EFFEM_ADMIN_DIDS	none	DIDs whose read tokens also get the admin scope.
EFFEM_CORS_ALLOWED_ORIGINS	http://localhost:3000	Exact origins; * is rejected.
EFFEM_RATE_LIMIT_ENABLED	true	Per-address and per-subject token buckets.
EFFEM_RATE_LIMIT_RPS, EFFEM_RATE_LIMIT_BURST	20, 200	Per client address.
EFFEM_RATE_LIMIT_SUB_RPS, EFFEM_RATE_LIMIT_SUB_BURST	40, 400	Per authenticated subject.
EFFEM_PI_CLIENT_RPS, EFFEM_PI_CLIENT_BURST	1, 120	Podcast Index calls one client may cause.
EFFEM_PI_SERVICE_RPS, EFFEM_PI_SERVICE_BURST	2, 120	Podcast Index calls the whole service may make.
EFFEM_TRUSTED_PROXY_CIDRS	none	Extra ranges whose X-Forwarded-For is trusted; private ranges always are. Not needed on Vercel, whose edge rewrites the header to the caller's address and whose bootstrap proxies over loopback.
EFFEM_APNS_KEY_PATH or EFFEM_APNS_KEY, plus EFFEM_APNS_KEY_ID, EFFEM_APNS_TEAM_ID, EFFEM_APNS_BUNDLE_ID	unset	The .p8 key as a file or as PEM text (\n accepted for newlines); all four enable push delivery.
EFFEM_LOG_LEVEL	info	trace to critical.

Secrets never appear in logs: the database URLs, Podcast Index credentials, token maps, the APNs key and the cron secret are marked secret for the configuration reader.

Routes #

Every route lives under /xrpc/ and is listed by GET /xrpc/xyz.effem.feed.describeLexicons, which serves the lexicon files in Resources/Lexicons. GET /_health reports the database and the age of the newest indexed event, read from the database so every instance answers the same; GET /metrics needs the admin scope and is Prometheus text for this process only. Comment updates stream over a WebSocket at /xrpc/xyz.effem.feed.subscribeComments; the events travel between processes through Postgres NOTIFY, so a stream sees comments whichever instance indexed them.

The scheduled work has routes under /_cron/, guarded by CRON_SECRET: ingest (one bounded Jetstream run, then the identity refreshes, backfills and push notifications it queued), watchdog (one retry sweep of failed records) and cache-cleanup (one pass over expired Podcast Index rows). They are safe to call while the in-process services also run: an ingest call finds the lease taken and reports skipped.

JSON responses use snake_case keys. The two catalog record routes return the record exactly as its CID was computed, with camelCase keys, and the lexicon route returns the files as written.

Jetstream #

The reader subscribes to the four Jetstream kinds for the twelve xyz.effem.* collections. The cursor and host are saved in jetstream_cursor; a resume on a different host rewinds five seconds by event time, and a cursor the host no longer has is dropped so the reader restarts live and queues a backfill of every known account.

Backfills queue in the backfill_queue table (a sync event for a known account, or a lost cursor, which queues them all), so the queue outlives the process that filled it. A worker claims one account at a time with an owner stamp; a claim older than fifteen minutes is handed to the next worker, a failed attempt waits five minutes, and the fifth failure drops the account with its error in the log. In service mode a loop drains the queue; in cron mode each ingest run drains it after its Jetstream session, within the run's time, and reports what is left as backfills_pending.

Only one reader runs at a time, whichever process it is in: a reader takes the row in ingest_lease before connecting, renews it every five seconds while it reads, and stops if another reader takes it over after it expires (thirty seconds without a renewal). In service mode the reader lives as long as the process; in cron mode each GET /_cron/ingest call reads for EFFEM_INGEST_RUN_SECONDS from the saved cursor, so a schedule of every minute reads the stream almost continuously and a sparser one replays what happened in between.

Tests #

swift test
EFFEM_TEST_DATABASE_URL='postgres://effem:effem@localhost:5433/effem?sslmode=disable' swift test

The Database suite is skipped without EFFEM_TEST_DATABASE_URL. It drops and recreates the public schema before every test, so point it at a throwaway database only.

scripts/linux-build.sh and scripts/linux-test.sh run the same build and tests, plus the package tests, inside the swift:6.4-noble image.

Deploying to Vercel #

The server runs on Vercel's native executable runtime through swift-vercel-runtime: the binary is cross-compiled here with the Static Linux SDK, and a small bootstrap in the same function starts it and proxies requests to it. Nothing in the server is Vercel-specific apart from VercelRequestMiddleware, which attributes logs to their request, and the cron routes.

Once, on the machine that builds:

swiftly install 6.4.0 && swiftly use 6.4.0      # the swift.org toolchain, not Xcode's
swift sdk install https://download.swift.org/swift-6.4.0-release/static-sdk/swift-6.4.0-RELEASE/swift-6.4.0-RELEASE_static-linux-0.1.0.artifactbundle.tar.gz \
  --checksum 47d2fd89eebfdf9eb4d536b6710414297f755c17926cdebc4742c08982b40a9e
npm i -g vercel && vercel login

Once, in the Vercel project:

• Add a Postgres database (Neon from the Vercel Marketplace works; it sets DATABASE_URL and DATABASE_URL_UNPOOLED, which the server reads). Set EFFEM_DB_MAX_OPEN_CONNS=5 and EFFEM_DB_MAX_IDLE_CONNS=1: every instance has its own pool against a database with a connection cap.
• Set CRON_SECRET (any long random string), the EFFEM_AUTH_* token maps, EFFEM_CORS_ALLOWED_ORIGINS, EFFEM_PI_KEY and EFFEM_PI_SECRET, and for push EFFEM_APNS_KEY (the PEM text), EFFEM_APNS_KEY_ID, EFFEM_APNS_TEAM_ID and EFFEM_APNS_BUNDLE_ID. Leave EFFEM_BIND unset so the bootstrap's PORT is used, and EFFEM_INGEST_MODE unset so it defaults to cron.
• Expose the system environment variables in the project settings (VERCEL=1 is what switches the ingest mode).

Every deployment:

scripts/vercel-build.sh                  # swift package --disable-sandbox vercel ... → .vercel/output
vercel link                              # once: create or pick the project
vercel deploy --prebuilt                 # preview; add --prod for production
vercel curl https://<deployment>/_health

vercel.json declares the schedules: /_cron/ingest every minute, /_cron/watchdog hourly, /_cron/cache-cleanup daily. Vercel's Hobby plan runs crons at most once a day, which makes the index a day stale; the Pro plan allows the per-minute schedule. Any external scheduler that sends Authorization: Bearer <CRON_SECRET> works too, with crons removed from vercel.json so the deployment is accepted on Hobby. The function's maximum duration is set to 120 seconds by the build script, which covers a fifty-second run plus the close handshake and the post-run work.

What changes on Vercel compared with one resident process:

• Nothing reads Jetstream between runs. A record indexed while a run's socket is open shows up within a second, as before; one that lands in the gap between runs waits for the next run, about fifteen seconds at worst with the per-minute schedule and the fifty-second budget.
• Work queued by one request may run in another instance, so the backfill queue is a table (backfill_queue) and the known-DID set reloads from the tables when a miss finds it older than five minutes. BackfillWorkerService runs only in service mode; a cron ingest run drains the queue itself.
• /metrics, the per-address rate limiter and the per-client Podcast Index budget are per instance. The service-wide Podcast Index ceiling is shared through the pi_service_budget row, so scaling out does not raise it.
• The caller's address comes from X-Forwarded-For, which Vercel's edge sets to the caller's public address whatever the caller sent; the bootstrap is the socket peer on loopback. EFFEM_TRUSTED_PROXY_CIDRS stays unset.
• Comment streams are WebSockets, which Vercel serves in public beta; a connection ends at the function's maximum duration, so the client reopens the stream when it ends: EffemKit's CommentSubscription reconnects with a backoff and yields .connected(resumed: true), on which the app reloads the newest page once, since the server does not replay.
• The local swift-vercel-runtime checkout carries a one-line fix: Swift 6.4's build system names resource bundles .bundle and the released plugin only copied .resources. scripts/vercel-build.sh copies any bundle the plugin misses and refuses to produce a function without the EffemAppView one, so a build with the released plugin is safe too.
• Each instance's first request runs the migrator (a few milliseconds when nothing is pending) and loads the known DIDs before the server listens.
• The function must reach Vercel's logs: VercelLogHandler is the swift-log backend, so request logs appear under their request in the dashboard.

Running the image instead #

The Dockerfile still builds the same server for any host that keeps a process running; there EFFEM_INGEST_MODE defaults to service and no cron is needed.

docker build -t effem-appview .
docker run --rm -p 8080:8080 --env-file .env effem-appview

The image is Ubuntu with the statically linked server, its resource bundles, libcurl4 for FoundationNetworking and the Swift backtracer. Point the platform's health check at /_health, run one instance per database or rely on the ingest lease, and keep the Podcast Index key and secret in the platform's secret store. A graceful stop takes about five seconds, which is the TLS close timeout on the Jetstream connection, so give the process at least ten seconds after SIGTERM.