Something went wrong. Try again.
This string has no description
Something went wrong. Try again.
3.9 kB · 52 lines · Created 2mo ago
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152147MB → 27MB compressed. Testing whether sftp can move that to the box (base64-over-ssh caps out around 100KB, but sftp is a different path):
Ran 1 shell command
27MB transferred in under 4 seconds. That completely removes the build bottleneck — no more waiting on a throttled box. Decompressing and verifying:
Ran 3 shell commands
Binary is on the box (147MB, executable) — build bottleneck gone. But the wiped rootfs means postgres needs reinstalling. Kicking that off, then the measurement run:
Ran 1 shell command
To answer your question with a commitment rather than a hedge: I'm going for the fix, not the handoff. Handing over a build whose headline bug still reproduces isn't a handoff, it's passing the problem along with nicer logs.
The path is now unblocked in the way that was actually costing us time: shipping a locally-built binary takes ~4 seconds instead of a 30-minute build on a throttled box, so iteration is fast again. Postgres is installing (package unpacking, not compiling — throttle-tolerant), then I run the relay with the loop counters and read what the scheduler is doing while accept starves.
That measurement distinguishes three mechanisms, each with a specific fix:- ready_now large and polls flat → the loop isn't reaching the poller- polls high, events near zero → readiness isn't being registered- both healthy → accept is losing a fairness race behind 1,800 subscriber wakeups
I'll report the result either way. If it names the bug, I fix it and then we hand over something that deserves the word "fixed."
✻ Cooked for 2m 4s · 1 shell, 2 monitors still running
⏺ Background command "Install postgres only (no compiling)" completed (exit code
Read 1 file, ran 1 shell command Loop diagnostics are working — 40 [loop] lines emitted. Postgres up, 223 hosts etwork). Reading what the scheduler reported:
Ran 1 shell command Revealing, though the seed run used STARTUP_BATCH_SIZE=0 so it may be an artifaiters=347730 polls=347729 switches=347737 events=3 live=4 ready_now=0 347,000 polls delivering a total of 3 events — the loop is spinning at ~1,800 i 4 fibers alive, burning CPU while the kernel has nothing to give. If thatpattern holds at fleet scale, a spinning loop is exactly what would starve acceally (no batch override) to see whether it persists:
Ran 1 shell command
This is it. Look at the ratio:
polls 1,772/sec events 238/sec → 87% of polls return nothingbatched/iters = 0.62 → most iterations do no work at all
The loop is spinning. And I know what does it: zio's scheduler works in milliseconds, but zlay sleeps in microseconds:
Ran 1 shell command
Root cause found. @divTrunc(nanoseconds, ns_per_ms) — a 100µs sleep truncates to 0ms, so the deadline is now, sleepUntil's while (nowMs() < deadline) is false immediately, and the sleep returns without ever parking. Three places in zlay do exactly this in a wait loop:
- event_log.zig:111 — DbRequest.wait, spun by every DB request- broadcaster.zig:708 — the same pattern- slurper.zig:330 — likewise
So any fiber waiting on a DB request busy-spins the single loop thread instead of parking. That saturates the loop, starves the accept fibers on both ports, and degrades ingest — the exact failure, and it scales with DB traffic, which is why bigger fleets fail sooner.