deployment #

zlay runs on an 8-vCPU Hetzner node (168.119.156.18; it also serves relay.waow.tech since 2026-10-18), managed via k3s. all deployment is orchestrated from the relay repo using just recipes.

the server builds with the zig on its PATH: /usr/local/bin/zig -> /opt/zig-0.17.0/zig since 2026-10-18 (0.17.0, checksum-verified from ziglang.org). /opt/zig-0.16.0 stays for building the 0.16 backport branches; repoint the symlink for those. zig 0.17 keeps fetched packages in the project-local zig-pkg/ (gitignored), which survives the recipe's git checkout --force; tangled's archive endpoint rate-limits bursts of fetches (HTTP 429) and a rejected fetch leaves an N-V-… directory there, so seeding /opt/zlay/zig-pkg/ from a workstation's verified zig-pkg/ is a valid shortcut.

build and deploy #

the preferred method builds natively on the server (fast, no cross-compilation):

just zlay-publish-remote

this SSHs into the server and:

1. git pull --ff-only in /opt/zlay
2. zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu
3. buildah bud -f Dockerfile.runtime . — thin runtime image with SHA tag
4. pushes to k3s containerd via buildah push → ctr images import
5. kubectl set image deployment/zlay -n zlay main=<sha-tagged-image> + kubectl rollout status

the runtime image (Dockerfile.runtime) is minimal: debian bookworm-slim + ca-certificates + the binary.

why not Docker build? #

the full Dockerfile exists for CI/standalone builds but is slow on Mac (cross-compilation + QEMU). zlay-publish-remote skips all of that by building on the target architecture.

build flags #

• -Dbackend=zio — production backend since 2026-08-05: cooperative fibers instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the Io.Threaded fallback, which is the rollback path (old images stay pinned in containerd; rollback is one kubectl set image). deploy as just zlay publish-remote ReleaseSafe <ref> zio. since 2026-10-30 the zon pins lalinsky/zio 3bbd74d: RELAY_IO_EXECUTORS executor threads (default 4, 0 = one per usable cpu) with work stealing, io_uring with an automatic epoll fallback. before that it was the single-scheduler fork zzstoatzz.io/zio 73b3e09 (last image atcr.io/zzstoatzz.io/zlay:ReleaseSafe-zio-28f893b). see handoffs/HANDOFF-2026-10-29-lalinsky-zio.md and, for the original adoption, handoffs/HANDOFF-2026-08-05-zio-canary.md.

zio triage and forensics #

if a zio build misbehaves: forensics before rollback — run scripts/zlay-forensics.sh in-pod and dump /metrics; rolling back deletes the pod, its logs, and the evidence (how canary #1's forensics were lost).

triage order: relay_seq first (flat ingest reads exactly like broken serving if you only probe the serving side), then zio_sched_executors (should equal RELAY_IO_EXECUTORS), then zio_sched_parks_total and zio_sched_drain_woken_total over two scrapes: parks climbing while ingest is flat means nothing is runnable (look upstream, not at the scheduler); drain_woken flat while frames are still arriving means cross-executor wakes are being lost. /metrics latency itself is the accept-path signal — it was the fork's starvation signature in 2026-08.

the liveness probe restarts a stalled pod. since 2026-08-22 /_healthz returns 503 once the process-wide ingest rate (relay_frames_received_total sampled every 10s) has been below RELAY_INGEST_STALL_THRESHOLD_FPS (50) for RELAY_INGEST_STALL_WINDOW_SEC (900s), after a startup grace of RELAY_INGEST_STALL_STARTUP_GRACE_SEC (900s). the threshold matches the ZlayIngestStalled rule and the window is longer than that rule's lead time, so the alert pages first; with the probe's failureThreshold 20 × 30s, kubelet kills the pod ~10 minutes after the 503 begins, i.e. ~25 minutes into a stall. the verdict is never per-worker — silence from a single host is normal, and judging hosts individually is what caused the 2026-08-19 outage.

this means a wedged pod no longer waits for you, so throw the kill switch before forensics — capture-before-restart is still the standing play:

# hold the pod up: /_healthz goes back to 200 immediately
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":false}'
# inspect (also shows rate, how long it has been below threshold, and the knobs)
curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ingest-stall
# widen instead of disabling: {"window_sec":3600} / {"threshold_fps":10}
# re-arm when done
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":true}'

settings made this way are not persisted: a restart returns to the env defaults, so a disabled detector cannot outlive the incident by accident. to change the defaults, set the RELAY_INGEST_STALL_* env vars in deploy/zlay-values.yaml. relay_ingest_stalled, relay_ingest_rate_fps, relay_ingest_below_threshold_seconds and relay_ingest_stall_enabled are on /metrics; when stalled is 1 the 503 body carries the rate and window.

the keepalive pinger detects half-open connections. the 2026-08-18 and 2026-08-27 wedges were mass half-open TCP: the network path died without delivering FIN/RST and nothing in the stack could tell a quiet peer from a dead one. each subscriber connection now pings its upstream after RELAY_WS_PING_INTERVAL_SEC (30s) of silence and reconnects after RELAY_WS_PING_MAX_FAILURES (4) unanswered pings. a ping counts as answered iff any frame (data or pong) arrives after its send time — never by comparing silence against the interval, which the first canary showed wrongly reaps healthy quiet hosts whenever scheduler oversleep exceeds the pong's rtt. quiet-but-alive hosts are never touched. under zio this runs as a per-connection pinger with the read loop as a cancelable task (SO_RCVTIMEO, which drives the Threaded path's heartbeat, never fires on an epoll-parked fiber). knobs are runtime-settable, not persisted, same contract as the stall detector:

curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ws-ping
# kill switch (reverts to pre-fix no-keepalive behavior):
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ws-ping -d '{"enabled":false}'
# retune: any subset of {"interval_sec":N,"max_failures":N}

on /metrics: relay_ws_pings_sent_total (baseline traffic — quiet hosts make this tick steadily), relay_ws_ping_timeout_closes_total (each is a connection declared half-open; a mass spike is the 08-18 signature being caught in the act) and relay_ws_ping_write_failures_total.

the replay window is 2 hours, by policy. RELAY_RETENTION_HOURS=2 in prod — at ~185 segments/hour (~53MB each, 10K events/file at ~520 seq/s) every retained hour costs ~8.5G on a 150G local-path disk shared with prometheus and postgres. deep replay is stream.waow.tech's archive, not zlay's job. RELAY_MAX_EVENTS_GB=30 is an overflow backstop only: keep it sized ABOVE the retention window (2h ≈ 17G) so the clock governs — when the byte cap binds instead, the effective window silently shrinks below the advertised one (this is exactly how 12h-configured retention was really ~10.5h before 2026-08-27). GC runs every 10 minutes and also sweeps orphaned segments that have no log_file_refs row.

• -Dtarget=x86_64-linux-gnu — production target, glibc. glibc malloc (per-thread arenas + madvise page-return + malloc_trim) is load-bearing for RSS at ~2,800 threads. NB: the old "musl breaks RocksDB / illegal instructions" claim was a zig 0.15 artifact and is retired — on 0.16 musl builds, links, and runs SIGILL-free at thread scale (canary, 2026-06). musl stays off prod only because the one static-musl allocator we validated as buildable — mimalloc — leaks RSS under our thread model (v2/v3/purge-forced/even under glibc). full matrix in musl-investigation.md.
• -Dcpu=baseline — required when building inside Docker/QEMU (not needed for zlay-publish-remote since it builds natively).
• -Doptimize=ReleaseSafe — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see incident-2026-03-04.md) — resolved by the frame pool moving heavy work off reader threads.

initial setup #

just zlay-init          # terraform init
just zlay-infra         # create Hetzner server with k3s
just zlay-kubeconfig    # pull kubeconfig (~2 min after creation)
just zlay-deploy        # full deploy: cert-manager, postgres, relay, monitoring

point DNS A record for ZLAY_DOMAIN at the server IP (just zlay-server-ip) before deploying.

environment variables #

set in .env in the relay repo:

variable	required	description
HCLOUD_TOKEN	yes	Hetzner Cloud API token
ZLAY_DOMAIN	yes	public domain (e.g. zlay.waow.tech)
ZLAY_ADMIN_PASSWORD	yes	bearer token for admin endpoints
ZLAY_POSTGRES_PASSWORD	yes	postgres password
LETSENCRYPT_EMAIL	yes	email for TLS certificates

operations #

just zlay-status        # nodes, pods, health
just zlay-logs          # tail relay logs
just zlay-health        # curl public health endpoint
just zlay-ssh           # ssh into server

infrastructure #

• server: Hetzner CPX41 — 16 vCPU (AMD), 32 GB RAM, 240 GB NVMe
• k3s: single-node kubernetes with traefik ingress
• cert-manager: automatic TLS via Let's Encrypt
• postgres: bitnami/postgresql helm chart (relay state, backfill progress)
• monitoring: prometheus + grafana via kube-prometheus-stack
• terraform: infra/zlay/ in the relay repo

memory tuning #

four changes brought steady-state memory from ~6.6 GiB down to ~1.1 GiB at ~2,250 connected hosts (ReleaseSafe):

shared TLS CA bundle. the biggest single win. websocket.zig's TLS client calls Bundle.rescan() per connection, loading the system CA certificates into a per-connection arena. with ~2,750 PDS connections, that's ~2,750 copies of the CA bundle in memory (~800 KB each = ~2.2 GiB). fix: load the bundle once in the slurper, pass it to all subscribers via config.ca_bundle. memory dropped from ~3.3 GiB to ~1.2 GiB (~65% reduction).

thread stack sizes. zig's default thread stack is 16 MB. with ~2,750 subscriber threads that maps 44 GB of virtual memory. all Thread.spawn calls use main.default_stack_size (8 MB). this is virtual memory — only touched pages count as RSS. 8 MB supports ReleaseSafe's TLS handshake path (~134 KiB peak stack).

c_allocator instead of GeneralPurposeAllocator. GPA is a debug allocator — it tracks per-allocation metadata and never returns freed small allocations to the OS. since zlay links glibc (build.zig:42), std.heap.c_allocator gives us glibc malloc with per-thread arenas, madvise-based page return, and production-grade fragmentation mitigation.

frame processing pool. reader threads (one per PDS) now only do TLS read, header decode, cursor tracking, and rate limiting — then queue raw frames to a shared pool of 16 workers. this dramatically reduced per-thread RSS in ReleaseSafe (from ~3.9 MiB to ~0.45 MiB) by keeping crypto, DB, and broadcast off reader thread stacks.

resource usage #

metric	value
memory	~1.1 GiB at ~2,250 hosts (ReleaseSafe), projected ~1.3 GiB steady state
CPU	~1.5 cores peak
requests	1 GiB memory, 1000m CPU
limits	8 GiB memory
PVC	20 GiB (events + RocksDB collection index)
postgres	~238 MiB

git push #

the zlay repo is hosted on tangled. pushing requires the tangled SSH key:

GIT_SSH_COMMAND="ssh -i ~/.ssh/tangled_ed25519 -o IdentitiesOnly=yes" git push

Commit CID migration (2026-10-18) #

account_repo.commit_data_cid is the MST root used for prevData continuity. It is not a repository head. commit_cid stores the signed commit CID, and listRepos.head and getLatestCommit.cid read that field. Both CIDs and the revision are updated atomically after verification; stale or equal revisions cannot overwrite them.

Startup adds commit_cid with an empty default, preserving existing revision/MST state, and creates a partial UID index for known commits. Legacy roots cannot be converted to signed commit CIDs. Such rows are omitted from listRepos; getLatestCommit returns RepoNotFound until a new verified commit or sync supplies the head. This reduces enumeration coverage during migration, including inactive repositories that may not emit another commit. A historical backfill must obtain the actual signed commit and match its revision before filling a legacy row; do not copy the MST root into the new column. The pre-rollout inventory was approximately 7.43 million rows. No historical backfill is part of this patch.

Rollback to a pre-migration binary leaves the additive column intact, but that binary does not maintain it as revisions change. Before returning to this version after such a rollback, clear or revalidate previously stored commit_cid values; otherwise old CIDs could be paired with newer revisions. Keep the old MST-root column for chain validation.

The regression suite requires DATABASE_URL for database-backed tests. Each test creates and drops its own database; use a disposable PostgreSQL instance with a database-creating role, not the production instance.

TLS trust updates #

All Debian image stages require ca-certificates >= 20250419~deb12u2 for 2026-10-21 eMudhra distrust. Raising the minimum changes the Docker build cache key and fails the build if the package repository cannot supply the fixed bundle. The shared WebSocket dependency also enforces distrust with caller-supplied CA bundles. The policy removes trust in the four affected root keys, also rejecting otherwise unaffected chains that depend on those roots.

After deploying a new image, run the runtime probe inside the replacement container (requires OpenSSL and coreutils timeout):

kubectl exec -i -n zlay deployment/zlay -c main -- sh < scripts/verify-runtime-tls.sh

It requires a working valid control and certificate-validation rejection of both direct and AAA cross-signed compromised chains. Network errors do not pass. This probes OpenSSL's runtime trust; shared transport regression tests cover application TLS. A process restart is required because the slurper loads its CA bundle once. Record the old image digest and new source revision before rollout, then check ingest/serving and the zio metrics above. Rolling back to an image with the old CA bundle reintroduces the vulnerability; prefer rebuilding the previous source with the patched bundle if rollback is needed.