# Deployment and operations

The website, APIs, SQLite index, ingestion worker, and multiplayer coordinator
run in one Node process.
Use one service instance with persistent local storage. For code ownership and
data flow, see [Architecture](architecture.md); public formats and endpoints
are in [records and APIs](reference/records.md).

## Local and production configuration

`npm run dev` starts the backend runtime with Vite on port 5173. Browser source
uses HMR; imported server modules restart through Node watch mode. `npm run
build` typechecks and builds with adapter-node; `npm start` or `npm run preview`
serves that build on port 3000. Builds do not open the database or run ingestion.

The development and start scripts load the ignored root `.env.local`. Compose
also reads that file; it is not copied into the image. Start from
[`.env.example`](../.env.example) for public settings. Only explicitly selected
public build settings enter client bundles; keep the Jetstream key server-side.

| Variable | Purpose and default |
| --- | --- |
| `SITE_URL` | Build-time full public URL, including subdirectory. Compose defaults to `http://127.0.0.1:3000/`. |
| `BASE_PATH` | Local build subpath checks when no public `SITE_URL` is supplied; use the same setting when running. |
| `HANDLE_RESOLVER` | Build-time browser handle resolver; defaults to `https://bsky.social`. |
| `HOST` / `PORT` | Node listener; defaults to `127.0.0.1:3000`, or `0.0.0.0:3000` in the image. |
| `DATABASE_PATH` | Defaults to `appview/data/appview.sqlite`, or `/data/appview.sqlite` in the image. |
| `INGESTION_ENABLED` | Defaults to `false`; enable explicitly for public PDS indexing. |
| `JETSTREAM_SERVICE` | Defaults to `https://jetstream.us-west.bsky.network`. |
| `JETSTREAM_API_KEY` | Required when ingestion is enabled. |
| `PUBLIC_ORIGIN` | Exact public origin, without a path, for API origin checks and secure Guest cookies. |
| `TRUSTED_PROXY_IPS` | Comma-separated exact proxy peer addresses allowed to supply `X-Real-IP`. |
| `HEALTHCHECK_PATH` | Optional container probe override; otherwise derived from `SITE_URL`. |

An explicitly relative `DATABASE_PATH` resolves from the working directory;
the default is anchored to the application. Parent directories are created
automatically. Changing Jetstream source or filters while retaining its database
is rejected because the durable cursor belongs to that source and filter.

For a public build:

```sh
SITE_URL=https://example.com/rad-cohesion/ npm run build
PUBLIC_ORIGIN=https://example.com npm start
```

`SITE_URL` sets the base path, SvelteKit request origin, OAuth metadata, canonical
URLs, and social previews. Rebuild when the public host or path changes. The
runtime `ORIGIN` variable is not used by the current adapter. Without
`SITE_URL`, local builds work but omit absolute canonical metadata. The social
image source is `src/assets/social-preview.png`; its emitted URL is hashed.

## Reverse proxy and container

```sh
docker compose up --build -d appview
docker compose logs -f appview
```

The service name remains `appview`, but the image serves the whole website.
Compose publishes host loopback port 3000 and mounts a named volume at `/data`.
The image pins its Node release in [Dockerfile](../Dockerfile), runs as the
unprivileged `node` user, and requires no separate API proxy or CORS setup.

For HTTPS hosting, set the public `SITE_URL` before building, set runtime
`PUBLIC_ORIGIN` to its origin, and forward the complete public prefix to Node
without stripping it. Configure the exact proxy peer address seen by Node in
`TRUSTED_PROXY_IPS`, and have the proxy overwrite `X-Real-IP`. The application
does not trust `X-Forwarded-For` or `X-Forwarded-Host`; unlisted peers are
rate-limited by their socket address.

The existing Oscillatory deployment uses `/rad-cohesion/` and host loopback
port 3010 because port 3000 belongs to the PDS. Its playbook and nginx template
live in the sibling `my-servers` repository. This repository owns the image and
Compose setup.

Health probes include the deployment prefix, for example
`/rad-cohesion/healthz`. The image derives this from `SITE_URL` unless
`HEALTHCHECK_PATH` overrides it. Docker's restart policy restarts exited
processes, not merely unhealthy ones. SIGTERM/SIGINT drain HTTP requests, stop
ingestion, and close SQLite.

### WebSocket forwarding

Multiplayer uses the same origin at `/api/multiplayer` beneath the public base
path. The reverse proxy must forward WebSocket upgrades as well as ordinary
HTTP requests. In the existing nginx location, retain the full path and proxy
headers above and include:

```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```

The endpoint checks browser origin and negotiates the
`rad-cohesion.multiplayer.v1` subprotocol. `PUBLIC_ORIGIN` and exact
`TRUSTED_PROXY_IPS` also apply to race connections and room-admission limits.
Native clients must supply the expected Origin header. Development shares the
Vite HTTP server while leaving its hot-reload upgrades alone.

Run one coordinator instance: rooms are ephemeral process memory, not SQLite
records, and are lost on deployment/restart. Shutdown closes race sockets before
draining HTTP. Multiplayer needs a live connection even when the static shell
is available offline. Existing single-player data and PDS indexing are separate.

Bot-enabled rooms use Node worker threads in the same deployment, with up to
three bot rooms per process by default and at most three bots per room. Set
`MULTIPLAYER_MAX_BOT_ROOMS` to `0`, `1`, `2` or `3` to disable bots or bound concurrent
bot rooms. Other values fail startup. Preparation
and simulation stay off the HTTP/coordinator thread. Adding a bot reports a
busy error when capacity is occupied; workers are released when their room
finishes or is abandoned. Bots require no extra service or credentials.

Each worker shares one prepared board, placement index, ID lookup and pair of
fingerprints across its bots. Different rooms still have separate workers and
heaps. In a local preparation-only benchmark at `k=10`, a minimal Node process
with one three-bot room used roughly 166 MiB RSS for an octahedron and 238 MiB
for an icosahedron, including a roughly 73 MiB parent-process baseline. Two
three-bot rooms used about 244 MiB and 390 MiB respectively. These are development
measurements, not memory limits: the full service, active gameplay, allocator
behavior and garbage collection need additional headroom. Adding a second or
third bot no longer duplicates the room's prepared lookup containers or hashing
work, but adding another room still adds another prepared context.

## OAuth and Lexicons

The public `oauth-client-metadata.json` URL is the browser OAuth client ID;
there is no client secret. Metadata and callback pages share the app's origin
and deployment base and are excluded from service-worker caching.

For local sign-in, use `http://127.0.0.1:5173/game/` and keep the same host and
port through the callback. The SDK uses the optional loopback client-ID
exception, so provider support is required. The localhost account dialog links
to the same page at `127.0.0.1`; click Sign in again there. Browser data does not
transfer between those hostnames. Local OAuth can publish directly to a real
PDS without a local PDS or public development deployment.

The two checked-in schemas are under [lexicons](../lexicons/). To make them
discoverable, publish each as a `com.atproto.lexicon.schema` record with its
NSID as the record key, then configure:

```text
_lexicon.radcohesion.oscillatory.net TXT "did=did:plc:<schema-owner>"
```

Checked-in JSON and builds do not publish schemas or change DNS. See the
[OAuth specification](https://atproto.com/specs/oauth#localhost-client-development)
and [Lexicon publishing guide](https://atproto.com/guides/publishing-lexicons).

## Ingestion and troubleshooting

Enable indexing in root `.env.local`:

```dotenv
INGESTION_ENABLED=true
JETSTREAM_API_KEY=your-key
```

The key is sent only to the configured HTTPS Jetstream service. Browser tests
disable ingestion. The archive reader prefetches one block with an 8 MiB
compressed-buffer target; decompressed size adds to memory use.

`GET /healthz` checks database availability and returns 503 after permanent
ingestion failure. `GET /status` reports phase, cursors, indexed counts,
pending author checks, retry information, and sanitized errors without the key.
Both paths include the configured base. Phases include `disabled`, `starting`,
`backfilling`, `live`, `retrying`, `failed`, `stopping`, and `stopped`.

| Status field | Meaning |
| --- | --- |
| `cursor` | Durable live-stream progress. |
| `discoveryCursor` | Separate initial archive-discovery progress. |
| `checkpointedAtMs` | Local time progress was saved. |
| `lastBatchAtMs` | Local time a batch was processed. |
| `lastEventAtMs` | Timestamp carried by the last newly processed event. |
| `eventLagMs` | Age of that event, clamped to zero; not a count of missing game records. |
| `retryAtMs` / `retryCount` | Scheduled retry time and cumulative retries in this worker lifetime. |

Times are Unix milliseconds. A sequence cursor cannot be converted to a date.
Empty batches retain the last event date. Filtered/no-op progress is saved at
most once every five seconds as batches arrive; replay after a crash is safe.

Initial discovery scans commit history for the two app collections, then checks
each candidate's current PDS. Bootstrap completion means every candidate has
had an initial attempt, not that all PDS checks succeeded. Failed checks remain
pending. Regular author checks continue while Jetstream is disconnected.

If a publication is missing, inspect status and logs for archive progress,
pending author checks, invalid-record rejection, or an inactive account. A PDS
failure preserves last-known state. Deactivation can take about an hour plus
queue delay to disappear; record refreshes run daily and after accepted commits.
Confirmed deletion removes records, while reactivation causes a fresh listing.

An expired live cursor triggers archive catch-up from the saved live cursor,
not a reset to zero. Catch-up progress and its fixed endpoint survive restart.
HTTP 429 from hosted Jetstream can indicate archive byte quota exhaustion;
respect the reported retry time. Explicit `Retry-After` delays are not capped,
while ordinary exponential backoff is. Do not reset the database or skip a
cursor to bypass a retry. See [Jetstream replay](https://bsky.network/docs/jetstream-replay/).

The install hook [patch-jetstream.mjs](../appview/tools/patch-jetstream.mjs)
patches SDK 1.0.0's empty optional sync timestamp handling and explicit
`Retry-After` cap. It runs on `npm ci` and in Docker, is idempotent, and rejects
unrecognized SDK sources. After `npm ci --ignore-scripts`, run it manually.
Review these patches when updating the SDK; preserve the database when
rebuilding to resume from the committed cursor.

## Database, backups, and upgrades

Use local storage, not NFS, for the `/data` volume. Recreating the container
preserves it; `docker compose down --volumes` deletes it. Guest submissions
are stored there in addition to the rebuildable PDS index.

For a stopped-service backup and upgrade:

```sh
docker compose stop appview
docker compose run --rm --no-deps --entrypoint tar appview -C /data -cf - . > appview-backup.tar
docker compose up --build -d appview
```

Keep a copy off-host. Restore with the service stopped into an empty volume
with its original ownership, using the application image matching the backup.
Do not copy only a live `.sqlite` file: committed changes may still be in its
WAL. Use SQLite's online backup API if uninterrupted backups become necessary.

Startup applies all pending SQL and migration bookkeeping in one transaction
before listening. Failures roll back the batch and prevent startup. Append
migrations; never edit, remove, or rename applied files. Rollback means restoring
both the old image and its matching database backup. Stop the old service before
schema upgrades. See [migration conventions](../appview/migrations/README.md)
for the local implementation contract and migration inventory.

All connections go through `appview/src/database.ts`: foreign keys on, 5-second
busy timeout, WAL journaling, `FULL` synchronous durability, trusted schema off,
double-quoted string literals and extension loading disabled, and defensive
mode enabled. Tables use `STRICT`, `NOT NULL`, and domain checks. Connection
settings are reapplied per connection; page/cache sizes, memory mapping, and
automatic WAL checkpointing retain SQLite defaults. Keep data changes and
ingestion cursors transactional, use bound query parameters, and handle lock
errors even with a busy timeout.

## Asset maintenance

Replace `src/assets/social-preview.png` to update link previews; rebuilding
changes the hashed image URL. Sharing services may retain cached previews.
Audio ships as lazy-loaded MP3s; the untracked WAV masters stay outside builds.
Regenerate them from the repository root when editing audio:

```sh
ffmpeg -y -i theme-2.wav -map_metadata -1 -codec:a libmp3lame -q:a 3 src/assets/track-1.mp3
ffmpeg -y -i theme.wav -map_metadata -1 -codec:a libmp3lame -q:a 3 src/assets/track-2.mp3
ffmpeg -y -i placement.wav -map_metadata -1 -codec:a libmp3lame -q:a 3 src/assets/placement.mp3
ffmpeg -y -i clear.wav -map_metadata -1 -codec:a libmp3lame -q:a 3 src/assets/clear.mp3
ffmpeg -y -i invalid-placement.wav -map_metadata -1 -codec:a libmp3lame -q:a 3 src/assets/invalid-placement.mp3
```