# production deployment

This runbook describes the instance used to exercise ZDS with real resident
accounts and interoperability clients.

## production shape

| setting | value |
| --- | --- |
| public origin | `https://pds.zat.dev` |
| Fly app | `zds-pds` |
| primary region | `ord` |
| machine | one shared CPU, 1 GiB RAM |
| persistent data | Fly volume `zds_data` mounted at `/data` |
| database | `/data/zds.sqlite3` |
| blobstore | `/data/blobs` |
| hosted handles | `*.pds.zat.dev` |
| account policy | invite required |
| permissioned data | enabled, experimental |

The committed non-secret configuration is in [`fly.toml`](https://tangled.org/zat.dev/zds/blob/main/fly.toml).
Secrets are Fly secrets and must not be copied into source, logs, or issue
reports. The repository `.env` is ignored local operator state; commands that
need one of its values should source it without printing it.

SQLite and blob bytes live on the mounted volume. A new application image must
reuse that volume. Never create a replacement machine with an empty `/data` and
send production traffic to it as a deployment shortcut.

## normal deployment

Every push to `main` runs [`.tangled/workflows/ci.yml`](https://tangled.org/zat.dev/zds/blob/main/.tangled/workflows/ci.yml),
which gates deployment on formatting, unit tests, the ARMv6 ReleaseSafe build,
both HTTP smoke suites, and the `@atproto/space` SDK interop lane. After those
checks pass, it executes:

```sh
tools/fly-deploy.sh
```

The workflow owns `FLY_API_TOKEN`. A normal release is therefore:

1. Confirm the worktree is the canonical `main` checkout and is based on
   `origin/main`.
2. Run the required checks from the repository root.
3. Commit and push `main`.
4. Watch the Tangled workflow (`tg pipeline status zat.dev/zds`) and verify the
   live service after Fly replaces the machine. The deployed image is labeled
   `git-<12-character commit>`.

Required checks:

```sh
just test
just smoke-all
zig fmt --check build.zig build.zig.zon src bench tools
git diff --check
zig zen
```

Also run `just smoke-permissioned` for permissioned-data storage, OAuth space
scope, `com.atproto.{space,memberspace,managedspace}.*`, or `/account/spaces`
changes. Run the relevant `just bench ...` lane for performance-sensitive
storage, repo, blob, proxy, or permissioned-data work.

## manual deployment

The manual path is for an operator already authenticated to the Fly app.
Push the release first. If no pipeline starts, verify that absence on the
spindle, run the same checks locally (including the ARMv6 command below), and
deploy from a clean archive of the pushed revision so uncommitted work is not
built into the release:

```sh
flyctl auth whoami
release_src=$(mktemp -d)
git archive HEAD | tar -x -C "$release_src"
"$release_src/tools/fly-deploy.sh"
```

`tools/fly-deploy.sh` is also what CI runs, with `ZDS_PREBUILT=.ci-release`: the
x86_64 ReleaseSafe binaries `tools/ci-build.sh` compiled beside the tests and the
ARMv6 check. Run by hand, it builds them locally. Either way it uploads only
the binaries, in `Dockerfile.fly`'s runtime image, so Fly's builder does not
compile zds. `Dockerfile` still builds from source for
the published container image.

Do not create a second configuration path in shell history. `fly.toml` remains
the source of non-secret deployment configuration, and Fly secrets remain the
source of secret configuration.

### when no pipeline starts

A successful Git push alone is not a successful deployment. If
`tg pipeline status zat.dev/zds` shows no run for the pushed commit, query the
spindle directly with the repo DID and the full commit hash:

```sh
curl -fsSG https://spindle.zzstoatzz.io/xrpc/org.tangled.ci.queryPipelines \
  --data-urlencode 'repo=did:plc:6atzzmsifkfmfqmeqrl6jwzo' \
  --data-urlencode "commits=$(git rev-parse HEAD)" \
  --data-urlencode 'limit=5'
```

If the spindle has no pipeline either, use the manual path above.

## verification

Capture the current health response before deploying so there is a baseline:

```sh
curl -fsS https://pds.zat.dev/xrpc/_health | jq .
curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer |
  jq '{did,availableUserDomains,inviteCodeRequired}'
```

After deployment:

```sh
flyctl status --app zds-pds
flyctl logs --app zds-pds --no-tail
curl -fsS https://pds.zat.dev/xrpc/_health | jq .
curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer |
  jq '{did,availableUserDomains,inviteCodeRequired}'
curl -fsS https://pds.zat.dev/api/openapi.json >/dev/null
curl -fsS https://pds.zat.dev/account >/dev/null
```

Do not stop at an HTTP 200 from the health route when the change touched a
resident workflow. Exercise the affected path with a throwaway account or the
appropriate smoke client, then inspect recent logs for panics, repeated 5xx
responses, migration failures, proxy timeouts, or memory pressure. Never print
passwords, admin tokens, OAuth tokens, PLC keys, or mail credentials while
probing.

## rollback and recovery

Application rollback and data rollback are different operations. Re-deploying
an older image does not undo a SQLite migration, and restoring SQLite without
its corresponding blobstore can produce an incoherent account.

Before a risky storage migration, confirm that a current volume snapshot exists
and document the exact application revision and schema transition. If a release
fails without changing persistent state, deploy the last known-good source
revision through the same Fly path. If persistent state changed, stop and assess
the migration and volume snapshot together before rolling either side back.

For ordinary account operations after a healthy deployment, use the
[operator guide](https://tangled.org/zat.dev/zds/blob/main/docs/operations.md), [invite-code guide](https://tangled.org/zat.dev/zds/blob/main/docs/invite-codes.md), and
[account takedown runbook](https://tangled.org/zat.dev/zds/blob/main/docs/account-takedown-runbook.md).

## 32-bit and small boards

zds builds and runs on 32-bit ARM without native 64-bit atomics (ARMv6, e.g. a Raspberry
Pi 1 with 512 MB). Cross-compile from another machine:

```sh
zig build -Dtarget=arm-linux-musleabihf -Dcpu=arm1176jzf_s -Doptimize=ReleaseSafe
```

CI builds this target on every push. A complete Pi 1 recipe — Cloudflare Tunnel behind
CGNAT, systemd units, Wi-Fi watchdog — is Chad Miller's
[zds-pi-setup](https://tangled.org/chadtmiller.com/zds-pi-setup).

Set `ZDS_SERVER_DID` explicitly or leave it unset and let zds derive `did:web:<host>` from
`ZDS_PUBLIC_URL`.