operator guide #

This is the operator-facing guide for running ZDS: local startup, configuration, containers, invite administration, account migration, and release policy. The concrete pds.zat.dev Fly workflow is documented in the production deployment runbook.

For account suspension, takedown, and test-account cleanup, see the account takedown runbook. ZDS supports temporary account deactivation and operator takedown for hosted repo subjects; suspended and deleted remain protocol vocabulary without local operator workflows.

local checks #

Run these before committing:

just test
just smoke
git diff --check
zig zen

just smoke starts a local server, creates a session, writes records, checks blob upload, verifies repo/sync endpoints, and asserts that known app records use valid key shapes.

Run just smoke-permissioned as well when changing permissioned-data routes, storage, OAuth space scopes, or the resident space browser.

benchmarks #

Local store benchmarks live under bench/:

just bench all
just bench write
just bench read
just bench repo
just bench blob
just bench blob-gc 512 65536
just bench http
just bench websocket-churn
just bench metastore 10 1000
just bench get-cid 10 1000
just bench write-profile 10 500
just bench run --records 1000

These measure repo writes, indexed reads, full repo CAR materialization, blob disk IO and collection, and selected HTTP route overhead against temporary SQLite/blobstore state. Treat the numbers as local trend data; compare runs on the same machine and optimize against the shape of the curve, not a single absolute value.

blob lifecycle #

Uploaded blobs begin untethered. ZDS gives an untethered upload 24 hours to be referenced by either a public repo record or a permissioned-space record. Once a blob is old enough, removing its final record reference makes it collectable. A reference from either record system keeps the blob.

Collection is bounded and automatic. A background worker starts with the service and reclaims at most 256 eligible blobs per minute. Upload bodies are streamed through a fixed 64 KiB buffer into unique temporary files without holding a repo-write lane; only the atomic promotion and metadata update are serialized with collection for the same CID. Collection rechecks both record systems under the account write lane before deleting metadata and disk bytes. The service logs deleted counts, bytes, and failures. just bench blob-gc measures the deletion path against temporary state.

Keep experimental permissioned-data probes separate from this baseline. Their first job is to catch obviously bad access patterns in com.atproto.space.*, not to expand the normal benchmark gate.

See bench/README.md for the Tranquil comparison target and the permissioned-data benchmark notes.

stats #

/stats charts request history over 1h, 24h, 7d or 30d in three tabs: overview (traffic by kind, latency, slowest routes), repos & sync (writes, relay and backfill traffic, records and storage) and accounts & apps (sign-in routes and authenticated requests per app, for every resident or one). It is public and does not count its own requests.

Each request adds to its kind's minute and its route's hour in memory: a count, a server-error count and a latency histogram doubling from 128µs. A background worker adds those to the database every 20 seconds, so a restart loses at most that much. It also counts authenticated requests per account and app each hour, and once an hour snapshots the database size, blob bytes, records and accounts.

An OAuth app is named by its client_id's host. A password or app-password session carries no client_id, so it is named by the request's Origin host (a browser app such as bsky.app), else the product its User-Agent starts with (a native app such as Bluesky), else "password sessions" or "app passwords".

table	rows	kept
telemetry_minutes	one per minute and kind	2 days
telemetry_hours	one per hour, route and method	90 days
telemetry_apps	one per hour, account and app	90 days
telemetry_storage	one per hour	90 days

The last 32 slow (250 ms or more) or failed requests since start, with their full paths, are listed under the overview; they live only in memory.

local server #

zig build run -- --port 2583 --db dev/zds.sqlite3

Use Caddy for local HTTPS when a client requires TLS:

caddy run --config dev/Caddyfile

configuration #

Common deployment settings:

• ZDS_PUBLIC_URL: public PDS origin.
• ZDS_SERVER_DID: PDS service DID, usually did:web:<host>.
• ZDS_HOST: bind address. Default: 127.0.0.1, or 0.0.0.0 in the container image.
• ZDS_PORT: bind port. Default: 2583.
• ZDS_DB: SQLite database path. Default: dev/zds.sqlite3, or /data/zds.sqlite3 in the container image. ZDS_DB_PATH is accepted as a compatibility alias when ZDS_DB is unset.
• ZDS_PLC_ROTATION_KEY: PDS rotation authority private key. Accepts the official PDS 32-byte secp256k1 hex form and the private multikey form emitted by PDS Moover. New did:plc accounts use this as the PDS rotation authority.
• ZDS_RECOVERY_DID_KEY: optional recovery did:key returned before the PDS rotation key in recommended DID credentials.
• ZDS_JWT_SECRET: stable secret for access and refresh JWTs.
• ZDS_DPOP_SECRET: stable secret for stateless DPoP nonce generation. If unset, ZDS uses ZDS_JWT_SECRET. A separate value is recommended for production. Multi-node deployments must configure every node with the same effective DPoP secret so each node accepts the same nonce window.
• ZDS_HANDLE_DOMAINS: comma-separated leading-dot hosted-handle suffixes advertised by describeServer, for example .pds.example.com. Each suffix needs wildcard DNS and TLS (*.pds.example.com) routed to ZDS so hosted handles can answer /.well-known/atproto-did. ZDS treats configured suffixes as authoritative, exposes only active accounts through public handle resolution, and limits handle updates to 10 per five minutes and 50 per day. Do not include the bare PDS hostname.
• ZDS_MAIL_PROVIDER: email delivery provider. Default: comail. Supported: comail, resend, and outbox. outbox sends nothing: it keeps each message for a day in the database, where dev.zat.admin.listMail (admin token, optional to and limit) returns it with the code it carried. Use it only for test and eval hosts whose accounts have no real inbox.
• ZDS_EMAIL_FROM: sender address for account and PLC email tokens. The domain must be enrolled with the configured provider. Comail requires a bare email address such as noreply@example.com, not a display-name form such as ZDS <noreply@example.com>.
• ZDS_COMAIL_API_KEY and ZDS_COMAIL_DID: Comail HTTP Send API credentials. ZDS_COMAIL_DID is the atproto DID that enrolled the sending domain; ZDS sends it as X-Atmos-DID.
• ZDS_RESEND_API_KEY: Resend API key, used only when ZDS_MAIL_PROVIDER=resend. See comail for the default provider's identity model, request shape, and production troubleshooting notes.
• ZDS_BLOB_UPLOAD_LIMIT: upload body limit. Default: 100000000.
• ZDS_BLOBSTORE_PATH: disk blobstore root.
• ZDS_CRAWLERS: comma-separated relay crawl targets.
• ZDS_OPERATOR_HANDLE: atproto handle of the person operating this PDS, shown on the public signup page as the contact for invite codes.
• ZDS_PLC_DIRECTORY: PLC directory origin. Default: https://plc.directory.
• ZDS_PROXY_SERVICE_URL, ZDS_PROXY_SERVICE_DID, ZDS_PROXY_SERVICE_ID: appview that receives app.bsky.* requests sent without an atproto-proxy header, and the endpoint used when the header names this DID. Defaults: https://api.bsky.app, did:web:api.bsky.app, bsky_appview.
• ZDS_LOG_LEVEL: err, info, or debug. Default: info.
• ZDS_DEBUG: set to true as a shorthand for ZDS_LOG_LEVEL=debug.
• ZDS_MAX_CONCURRENT_REPO_EXPORTS: maximum concurrent full com.atproto.sync.getRepo exports. Default: 4. Incremental getRepo?since=... exports are not intended to consume this full-export backpressure budget.
• ZDS_SEQ_FLOOR (--seq-floor): sequence events above this number. Set it when ZDS replaces another PDS on the same hostname, to at least the old host's seq as relays report it (com.atproto.sync.getHostStatus), so relays reconnecting with their stored cursor pick up new events. Once events above the floor exist it has no further effect. Default: 0.
• ZDS_PLC_TOMBSTONE_ON_DELETE (--plc-tombstone-on-delete): set to true to tombstone an account's did:plc with this PDS's rotation key whenever the account is deleted, by itself or by an operator. The tombstone is submitted before anything is deleted; if the directory refuses it, or the DID no longer lists this PDS's rotation key, the deletion answers UpstreamFailure and the account stays. A tombstone is permanent, so leave this off unless the host is a test or eval host whose identities should not outlive their accounts.
• ZDS_ADMIN_TOKEN: token for privileged PDS administration, including invite-code minting, sent as Bearer <token> or as HTTP Basic admin:<token>.
• ZDS_INVITE_REQUIRED: set to true to require invite codes for account creation. describeServer reflects this value.
• ZDS_ALLOW_PRIVATE_FETCH: set to true only for local development and smoke tests. URLs that come from callers, DID documents, or client metadata (OAuth client metadata and JWKS, permission-set lexicons, did:web proxy targets and their service endpoints, managing apps, and Spaces notification recipients) are otherwise fetched only over HTTPS from public unicast addresses, pinned at resolution, with redirects refused. Operator settings such as the PLC directory, crawlers, and the configured appview are trusted.
• ZDS_PERMISSIONED_DATA: set to true to opt into experimental permissioned-data routes. Default: disabled. ZDS exposes protocol reads and writes and space lifecycle under com.atproto.space.*, and the baseline policies' methods under com.atproto.memberspace.* and com.atproto.managedspace.*, tracking the lexicons published for the spaces alpha. Enable it if residents want to use alpha apps such as Bulletin. The implementation remains experimental: reader/group semantics should stay in application policy, while the baseline space policies provide only the small required set. Use just smoke-permissioned for the experimental smoke lane.
• Passkeys do not require a separate deployment secret. WebAuthn RP ID is derived from the public URL host.

PLC key lifecycle:

• Repo signing keys are for repository commits only.
• PLC rotation keys are for DID PLC operations only.
• getRecommendedDidCredentials follows the official PDS shape: the account signing key is returned under verificationMethods.atproto; rotation keys are the optional recovery DID key followed by the configured PDS rotation key.
• ZDS rejects new PLC operations that put the account signing key in rotationKeys.
• Existing early accounts can be repaired with just plc-repair <did> after setting ZDS_DB or ZDS_DB_PATH, ZDS_PUBLIC_URL, ZDS_PLC_ROTATION_KEY, and optional ZDS_REPAIR_RECOVERY_DID_KEY, ZDS_REPAIR_RECOVERY_KEY, or ZDS_RECOVERY_DID_KEY. The repair tool is intentionally separate from normal XRPC handling; it only uses the account signing key if the current public PLC state already authorizes that key as a rotation key, and submits a replacement operation containing the intended recovery/PDS rotation keys. The ZDS_REPAIR_RECOVERY_KEY form accepts a PDS Moover private multikey and derives the public did:key locally.

When invites are required and the database has no accounts or invite codes, ZDS creates one bootstrap code and logs it during startup. Invite codes are stored as metadata rows and uses are recorded separately, matching the official PDS model.

container #

Images are published to ATCR:

docker pull atcr.io/zat.dev/zds:latest
docker run -p 2583:2583 \
  -v zds-data:/data \
  -e ZDS_PUBLIC_URL=https://pds.example.com \
  -e ZDS_SERVER_DID=did:web:pds.example.com \
  -e ZDS_JWT_SECRET='at-least-32-random-bytes-here' \
  -e ZDS_DB=/data/zds.sqlite3 \
  atcr.io/zat.dev/zds:latest

Set a random JWT secret and serve the configured public origin over HTTPS.

Tagged releases are published by CI (see releases). To build and publish from a local checkout instead:

just docker-publish
just docker-publish vX.Y.Z
just docker-publish-current
just docker-publish-release vX.Y.Z

releases #

ZDS uses SemVer tags (vMAJOR.MINOR.PATCH). Before v1.0.0, patch releases are still appropriate for user-visible compatibility fixes and operator-safe bug fixes. Minor releases are for meaningful protocol surface, storage shape, or operator workflow changes. Major releases are reserved for explicit incompatible production contracts once ZDS declares a stable surface.

The /release skill (.agents/skills/release/SKILL.md) walks this end to end, including the changelog promotion, the spindle deploy watch, and the live version check. Release checklist:

just test
just smoke-all
zig fmt --check build.zig build.zig.zon src bench tools
git diff --check
zig zen
git tag -a vX.Y.Z -m "vX.Y.Z"
git push origin main vX.Y.Z

The push to main deploys pds.zat.dev. The tag push runs .tangled/workflows/publish-image.yml, which publishes the image as vX.Y.Z, latest, and the 12-character commit.

For protocol-parity or storage-behavior releases, also complete the benchmark stage checklist: revisit the Zat boundary, run matching Tranquil and official PDS probes where comparable, and document any remaining mismatch as an explicit decision or follow-up.

migration #

replacing a host on the same hostname #

Relays keep a cursor for each host. A subscribeRepos cursor ahead of ZDS's latest seq gets a FutureCursor error frame and a close, as the spec and the reference PDS do, and a relay that hears it marks the host idle. Before pointing an existing hostname at ZDS, read the old host's seq from each relay's com.atproto.sync.getHostStatus?hostname=… and start ZDS with ZDS_SEQ_FLOOR at or above the highest one.

ZDS supports browser-based account migration flows such as PDS Moover:

• service-auth-gated createAccount for an existing DID
• repo CAR import/export
• missing blob reporting and blob upload
• recommended DID credentials
• PLC operation email token, signing, and submission
• account activation/deactivation and status checks

Migration requires working public HTTPS, stable JWT secret, email delivery, blob storage, and writable SQLite storage.

account security #

The resident account surface lives at /account. The security section, /account/manage, is for account-owner management of passkeys and app passwords and requires the account password for management actions. App-password revocation also revokes active sessions that were created with that app password. /security and /passkeys are compatibility redirects.

conformance #

Useful external checks:

• https://check.cirrus.earth/?target=<handle> for PDS and OAuth conformance.
• bsky.app and pdsls.dev for browser-visible OAuth, proxy, repo, and blob behavior.

Treat browser failures as request/response evidence: inspect the failing XRPC, status code, response body, and relevant server logs before changing behavior.

Web drawings #

Operators can assign individual Doodl drawings to UI slots through config/drawings.json and just drawings, then explicitly select the cached manifest with -Dweb-drawings=src/internal/web/icons/generated.zig at build time. Unconfigured builds use generic icons. The pds.zat.dev deployment builds with that option (tools/ci-build.sh and tools/fly-deploy.sh), and the Docker build takes it as the ZDS_WEB_DRAWINGS build argument; its selections live in config/pds.zat.dev.drawings.json. See Doodl drawings for the refresh and deployment workflow. Assets are bundled at build time; startup and account requests do not contact drawing authors’ PDSes.

Grouped account access #

GET /xrpc/dev.zat.account.listAccess requires an account password session, matching listSessions. kind=oauth|password selects app authorizations or password credentials; state=authorized|past|all defaults to authorized. search matches the app client ID or credential label, case-insensitively. Without group, the response contains grouped summaries (key, name, count, newest). With a returned group key, connections contains that group's existing connection details. OAuth groups use the complete client ID; password groups use authentication method and credential name.

Both modes default to 20 results and accept limit from 1 through 100. Pass the returned cursor with the same filters for the next page; null ends pagination. Cursors are non-secret offsets, not snapshot guarantees: changing sessions between requests can shift pages. authorized and total always count the entire signed-in account's selected kind, independently of filters and pagination. The overview uses these totals; search results show per-group counts. No JWT, refresh token, or OAuth family identifier is returned.

The app-access UI loads summaries first and connection pages only on expansion. "Authorized" describes unrevoked, unexpired credentials, not online activity. This endpoint does not revoke grants or change session lifetime policy.