zlay 0.16 migration — status and known issues #

last updated: 2026-04-04 stable production build: a931853 (zig 0.15) latest 0.16 build: b433403 (not yet deployed — includes all fixes through crash 8)

what happened #

zlay is an AT Protocol relay (~8,400 LOC, 15 source files). it crawls ~2,750 PDS instances, validates frames, persists to disk, and fans out to downstream consumers over WebSocket. the zig 0.16 migration rewrote all I/O to use std.Io primitives.

eight separate crashes/bugs were found and fixed during deployment attempts:

crash 1: SIGSEGV on startup (fixed in f996812) #

symptom: exit code 139 immediately after printing "io backend: Evented"

cause: the build auto-selected Io.Evented (io_uring on linux). evented futex wait/wake access fiber-local Thread.current() state. but zlay's frame pool (thread_pool.zig) spawns plain std.Thread workers — they don't have fiber state. any Io.Mutex operation from a plain thread segfaulted.

fix: force Backend = Io.Threaded in src/main.zig:55. threaded futex uses direct kernel syscalls that work from any execution context. all 0.16 API benefits are preserved; io.concurrent under Threaded spawns real OS threads.

future path: either migrate frame workers to io.concurrent, or give cross-boundary modules a dedicated Threaded sync_io for mutex operations. then Evented can be re-enabled for the network I/O layer.

crash 2: pool acquire panic under load (fixed in e5ed0d1) #

symptom: unreachable panic in Io.Event.waitTimeout after 30-60s of processing. stack: event_log.zig uidForDid → pg pool.zig acquire → Io.zig

cause: pg.zig pool used Io.Event + reset() as a connection-available signal. Io.Event.reset() has a stdlib invariant (Io.zig:1857): assumes no pending call to wait. with 16 frame workers contending for 5 connections: set() wakes all waiters, one calls reset(), others hit unreachable.

fix (pg.zig 5ce2355 on dev branch): replaced Io.Event with a monotonic u32 futex counter. release() increments counter + futexWake(1) (wake one). acquire() snapshots counter under mutex + futexWaitTimeout() with snapshot. no reset(), no single-waiter constraint. also fixed a pre-existing double-lock race in acquire and made zlay's pool size configurable via DB_POOL_SIZE env (default 20, was hardcoded 5).

crash 3: GPF in websocket client write (fixed in 0f11cfc) #

symptom: general protection exception in memcpy → Writer.zig → writeAll → websocket client.zig writeFrame. happens after 30-60s of normal processing.

cause: the websocket Client struct had no write serialization. three concurrent writers on the same client:

1. pingLoop — client.writeFrame(.ping, ...) every 30s (subscriber.zig:378)
2. readLoop auto-pong — self.writeFrame(.pong, ...) on upstream ping (inside the library, client.zig:272)
3. close path — client.writeFrame(.close, ...) on failure (subscriber.zig:383)

interleaved frame headers/payloads corrupt the shared stream/TLS writer state. the server-side Conn already had lock: Io.Mutex around all writes; the client was missing it.

fix (websocket.zig 0261b7d): added _write_lock: Io.Mutex to Client, acquired in writeFrame() around the two writeAll calls. header construction and payload masking happen outside the lock.

since zat also depends on websocket.zig, a new zat alpha (v0.3.0-alpha.9) was cut to resolve the diamond dependency.

crash 4: GPF in pingLoop after connection teardown (fixed in 6d6c832) #

symptom: GPF in memcpy → Writer.zig → writeAll → client.zig writeFrame every ~60-90s of processing. same stack trace as crash 3 but different root cause — the write lock from crash 3 was necessary but not sufficient.

cause: pingLoop runs as an io.concurrent task and sleeps in 1s increments. when readLoop returns (connection dies), the defer chain runs:

1. ping_future.cancel(self.io) — requests cancellation
2. client.deinit() — frees stream/TLS buffers

but pingLoop did self.io.sleep(...) catch {} — swallowing all errors including error.Canceled. so cancel() could not stop the task. deinit() then freed the client while pingLoop was still running. next iteration's writeFrame(.ping, ...) hit freed memory → GPF.

fix (zlay 6d6c832 + websocket.zig 104608b):

• io.sleep(...) catch {} → catch return — makes pingLoop cancellation-cooperative. cancel() can now stop the task before deinit() runs.
• added Client.isClosed() check before writeFrame — defense-in-depth against writes to dead connections.

fix 5: httpFallback dispatch for health probes (fixed in websocket.zig 4222f98) #

symptom: health probes on port 3000 return 400 instead of reaching the application handler. k8s rollout fails.

cause: the websocket server's _handleHandshake sent 400 on any non-upgrade HTTP request. there was no code path to detect "valid HTTP but not a websocket upgrade" and dispatch to Handler.httpFallback().

fix (websocket.zig 4222f98): intercepts MissingHeaders, InvalidConnection, and InvalidUpgrade errors from Handshake.parse(). for these (and only these), a standalone parseHttpRequest() re-parses the raw buffer and dispatches to H.httpFallback() if it exists (comptime hasFn check). other handshake errors (e.g. InvalidVersion) still get 400. 10 tests cover all request patterns.

crash 6: SIGSEGV — plain threads calling Evented Io.Mutex (fixed in 6674812) #

symptom: SIGSEGV at startup after switching to Io.Evented backend. crash in Uring.zig at Thread struct field offset from NULL.

cause: the resyncer was spawned via io.concurrent() which creates an Evented fiber. but resync work calls DiskPersist methods that take self.mutex.lockUncancelable(self.io) where self.io is pool_io (Threaded). Threaded futex from an Evented fiber dereferences Thread.current() which is a threadlocal only set on Uring-managed threads — NULL on Evented fibers.

fix: run resyncer on a plain std.Thread with pool_io, not as an Evented fiber. the thread checks shutdown_flag to exit.

fix 7: startup connection ramp throttling (fixed in f9bf515) #

symptom: event loop starvation during initial PDS connection burst (~2,750 simultaneous connects).

fix: throttled startup to connect in batches, preventing io_uring submission queue overflow.

crash 8: cross-Io heap corruption in GC + health checks (fixed in 2156d08 + b433403) #

symptom: steady-state heap corruption / SIGSEGV with zero downstream consumers. dmesg shows crash addresses in Uring.zig at Thread struct field offsets from NULL — same signature as crash 6 but in the GC and health check paths.

cause: two cross-Io violations active during steady-state operation:

1. GC loop ran as an Evented fiber (io.concurrent(gcLoop, ...)) but called dp.gc() which takes self.mutex.lockUncancelable(self.io) where self.io is pool_io (Threaded), and queries pg.Pool (also Threaded). Threaded futex from Evented fiber → NULL Thread.current() → heap corruption.

2. health check endpoints (/_readyz, /_health, /xrpc/_health) on both the metrics server and API router executed db.exec("SELECT 1") through pg.Pool (Threaded) from Evented HTTP handler context. same cross-Io violation.

fix:

• GC loop moved from io.concurrent() to std.Thread.spawn(.{}, gcLoop, .{&dp, pool_io}). the thread is joined during shutdown before dp.deinit() runs (dp is stack-owned).
• health checks replaced with isDbHealthy() — reads an atomic last_db_success timestamp set by Threaded workers after successful DB queries. safe from any Io.
• markDbSuccess() called from uidForDid() (every incoming event) and gc() (every 10 minutes).

fix 8: broadcaster double-destroy (fixed in 72ba680) #

symptom: use-after-free in broadcaster. broadcast() was destroying consumers that Handler.close() still referenced.

cause: broadcast() detected dead consumers (via alive atomic) and called consumer.shutdown() + self.allocator.destroy(consumer) inline. but the consumer's Handler.close() callback was still running and would later call removeConsumer() on the already-freed pointer.

fix: broadcast() now only does swapRemove + count decrement for dead consumers. removeConsumer() (called from Handler.close()) is the sole owner of shutdown() + destroy().

the cross-Io rule #

the single most important lesson from this migration:

Io.Mutex, Io.Condition, io.sleep(), and any pg.Pool operation must be called from the same Io type they were initialized with.

• Threaded futex on Evented fiber → dereferences NULL Thread.current() threadlocal → SIGSEGV or heap corruption (crashes 1, 6, 8)
• Evented futex on plain thread → same NULL deref in the other direction

safe cross-Io patterns: raw atomics (Value, fetchAdd, CAS), tryLock (non-blocking CAS, no futex), MPSC ring buffers with atomic spinlocks.

unsafe cross-Io patterns: Io.Mutex.lockUncancelable(wrong_io), Io.Condition, io.sleep(wrong_io, ...), pg.Pool queries from wrong context.

fix pattern: components that use Threaded resources (mutexes, pg.Pool) must run on plain std.Thread, not Evented io.concurrent(). examples: resyncer (439c678), GC loop (2156d08).

known remaining: XRPC and admin API handlers run on Evented fibers and access pg.Pool (Threaded) on external HTTP requests. not steady-state, but real. fixing requires either running API handlers on pool_io or making pg.Pool Io-agnostic.

stdlib patches #

six stdlib issues required patches or workarounds to run on the Evented (io_uring) backend. see docs/stdlib-patches.md for the full writeup: what's patched, what's worked around, and what upstream changes would let us drop each one.

summary:

#	issue	fix type
1	Uring networking stubs (error.NetworkDown)	patch (patches/uring-networking.patch)
2	DNS resolution (netLookup) missing	Threaded fallback in subscriber
3	ReleaseSafe GPF in fiber context	build with ReleaseFast
4	debug_io single-threaded default	std_options_debug_threaded_io override
5	Io.Event.reset() single-waiter panic	futex counter in pg.zig fork
6	cross-Io Mutex/futex incompatibility	Io segregation (plain threads for Threaded work)

where things live #

repos and their roles #

repo	location	what
zlay	tangled.org:zzstoatzz.io/zlay	the relay. this repo.
zat	tangled.org:zat.dev/zat	AT Protocol primitives (CBOR, CAR, firehose, DID, etc.)
websocket.zig	github.com/zzstoatzz/websocket.zig	fork of karlseguin/websocket.zig
pg.zig	github.com/zzstoatzz/pg.zig (dev branch)	fork of karlseguin/pg.zig
rocksdb-zig	github.com/zzstoatzz/rocksdb-zig	RocksDB bindings
relay ops	tangled.org:zzstoatzz.io/relay	helm chart, terraform, justfile

zlay source map #

file	LOC	role
main.zig	404	Io bootstrap, signal handlers, GC loop, MetricsServer
subscriber.zig	~1021	per-host PDS connection: connect, readLoop, pingLoop, backoff
slurper.zig	~663	multi-host crawl manager: host discovery, worker lifecycle
broadcaster.zig	~1319	downstream consumer management, broadcast, history, stats
frame_worker.zig	~311	per-frame processing: CBOR decode, validate, persist, broadcast
thread_pool.zig	~357	key-partitioned worker pool (plain std.Thread, not io.concurrent)
event_log.zig	~1446	disk persistence (indigo-compatible) + Postgres index
validator.zig	~1002	DID resolution, signature verification, validation cache
lru.zig	~289	generic LRU cache (used by validator + event_log)
ring_buffer.zig	~256	per-consumer ring buffer for broadcast fanout
collection_index.zig	~200	RocksDB collection index (listReposByCollection)
backfill.zig	~396	collection index backfill from source relay
resync.zig	~315	collection index resync on #sync events
cleaner.zig	~114	stale entry cleanup from collection index
api/router.zig	142	HTTP request dispatch (health, XRPC, admin)
api/xrpc.zig	—	XRPC endpoint handlers
api/admin.zig	—	admin endpoint handlers
api/http.zig	—	HTTP response helpers

concurrency architecture #

three layers, shaped by the cross-Io constraint:

Evented fibers (io.concurrent on Io.Evented)

• upstream PDS subscribers (read loops, ping loops)
• downstream consumer write loops
• DID resolver loops (validator)
• broadcast loop (drains queue, fans out to consumers)
• slurper coordination
• metrics server, backfill, cleaner

plain std.Thread with pool_io (Io.Threaded)

• GC loop — uses DiskPersist.mutex + pg.Pool (crash 8)
• resyncer — uses DiskPersist + HTTP client (crash 6)
• these MUST NOT run as Evented fibers (see "the cross-Io rule")

CPU-bound ordered processing → explicit std.Thread workers

• thread_pool.zig: workers[host_id % N] ensures per-key FIFO ordering
• frame_worker.zig: CBOR decode, DID resolution, signature verify, DB persist
• bounded backpressure: blocking submit when queue full → TCP backpressure
• uses Io.Mutex / Io.Condition with pool_io (Threaded futex)

dependency versions (current b433403) #

zat           v0.3.0-alpha.16 (tangled.org)
websocket.zig 80c6434         (github, master)
pg.zig        5ce2355         (github, dev branch)
rocksdb-zig   cdef67b         (github)
zig           0.16.0-dev.3059+42e33db9d (patched Uring networking)

key env vars #

var	default	what
RELAY_PORT	3000	WebSocket firehose + HTTP API
RELAY_METRICS_PORT	3001	internal metrics + health probes
FRAME_WORKERS	16	CPU worker threads for frame processing
FRAME_QUEUE_CAPACITY	4096	per-worker queue depth
DB_POOL_SIZE	20	Postgres connection pool size (new)
RELAY_UPSTREAM	bsky.network	seed host for PDS discovery
RELAY_DATA_DIR	data/events	disk persistence directory
RELAY_RETENTION_HOURS	72	event log retention
RELAY_MAX_EVENTS_GB	100	max disk usage
DATABASE_URL	postgres://relay:relay@localhost:5432/relay	Postgres connection

what needs to happen next #

1. deploy b433403 — all eight crashes/fixes are in. the steady-state heap corruption (cross-Io GC + health checks) is fixed. broadcaster double-destroy is fixed. health probes use atomic flag, not cross-Io DB query.

2. monitor after deploy — compare against 0.15 baseline:

   • thread count (2,903 on 0.15 — should be similar under Threaded)
   • memory (24.9 GiB VmSize, 1.44 GiB RSS on 0.15)
   • throughput, reconnect behavior, ConsumerTooSlow rate
   • verify zero crashes after extended run (hours). the GC cross-Io bug fired every 10 minutes, so 30+ minutes clean = strong signal.
   • watch /_health — should report healthy once uidForDid or gc() succeeds (within 10 minutes of startup at latest)

3. known issue — XRPC/admin cross-Io (not blocking deploy):

   • API handlers run on Evented fibers but some query pg.Pool (Threaded)
   • only triggered by external HTTP requests, not steady-state
   • fix: either run API handlers on pool_io, or make pg.Pool Io-agnostic

4. follow-up work:

   • investigate Evented backend viability (frame workers → io.concurrent?)
   • consider upstreaming the client write lock to karlseguin/websocket.zig
   • consider upstreaming Uring networking patch (zig#31723)
   • evaluate whether pg.Pool can be made Io-agnostic (would eliminate cross-Io issues)