getting atproto records at every scale — measured #
all numbers measured 2026-08-17 against real infrastructure (stream.waow.tech = jetstream v2 instance w/ full-network archive; lightrail/slingshot indexes; real PDSes). same collections, same day.
| method | statusphere (1,654 repos) | lexidraw.scene (52 repos) | what you get |
|---|---|---|---|
jetstream v2 archive slice (collections + kinds=commit in the plan) |
17,537 rows · 4m12s · 497 MB | 615 rows · 15s · 27 MB | full event history: creates, updates, deletes, plus rows from repos no longer reachable |
| lightrail → listRecords per repo | 13,145 rows · 4m53s · 3.6 MB · 71 repos failed | — | current state only; misses unreachable repos + all deletion history |
| lightrail → getRepo CAR walk | (not run: ~1,654 full repos) | 49/52 repos · 26s · 622 MB | exact current state, but you download every creator's entire repo |
| single-repo CAR (PDS direct) | n/a — one repo only | n/a | unbeatable for one DID: the repo is the optimal layout of itself |
two sharp edges worth knowing #
kinds:["commit"]must be in the plan request, not just client-side filtering. the planner otherwise admits every marker-bearing block: for statusphere the plan is 140,705 blocks (~24 GB) without it vs 1,684 blocks (~500 MB) with it — an 83× difference. easy to miss: a client can filter markers locally while still fetching all their blocks.- low observed MB/s during backfill usually isn't bandwidth. the same ~2 MB/s shows up from a datacenter box one hop from the server — it's the block-fetch pipeline (concurrency × per-request RTT). more parallel getBlock requests raise it; total bytes stay fixed.
the structural limit: archive blocks are ~4,096 events in arrival order, so a sparse collection scattered across the network costs (blocks touched) × (~300 KB), regardless of how few rows you want. 17K statusphere rows live in 1,686 blocks → ~500 MB. no client fixes that; only smaller blocks, per-collection layout, or sub-block reads server-side would.
why you'd pay it anyway: the archive slice returned +33% more rows than listRecords — deleted and updated records, plus everything from the 71 repos (4%!) that were unresolvable or unreachable at fetch time — from one source, with zero per-PDS failure handling, and a gapless cutover into the live tail when you're done. that's the local-first sync story: history + push, not just state + poll.
rule of thumb
- one repo → fetch its CAR from the PDS. always.
- many repos, current state, bytes matter → lightrail + listRecords (accept the reachability holes)
- many repos, full history / deletes / unreachable repos / then go live → jetstream v2 slice with plan-level
kinds=commit - freshly-bootstrapped archives flatter per-DID slices (bootstrap lays repos out contiguously); the numbers decay as live arrival-order accumulates — benchmark against an aged archive before concluding
https://tangled.org/zat.dev/jetstream