# getting atproto records at every scale — measured

all numbers measured 2026-08-17 against real infrastructure (stream.waow.tech = jetstream v2 instance w/ full-network archive; lightrail/slingshot indexes; real PDSes). same collections, same day.

| method | statusphere (1,654 repos) | lexidraw.scene (52 repos) | what you get |
|---|---|---|---|
| **jetstream v2 archive slice** (collections + `kinds=commit` in the plan) | 17,537 rows · 4m12s · **497 MB** | 615 rows · 15s · 27 MB | full event history: creates, updates, deletes, plus rows from repos no longer reachable |
| **lightrail → listRecords per repo** | 13,145 rows · 4m53s · **3.6 MB** · 71 repos failed | — | current state only; misses unreachable repos + all deletion history |
| **lightrail → getRepo CAR walk** | (not run: ~1,654 full repos) | 49/52 repos · 26s · **622 MB** | exact current state, but you download every creator's *entire* repo |
| **single-repo CAR** (PDS direct) | n/a — one repo only | n/a | unbeatable for one DID: the repo is the optimal layout of itself |

## two sharp edges worth knowing

1. **`kinds:["commit"]` must be in the *plan request*, not just client-side filtering.** the planner otherwise admits every marker-bearing block: for statusphere the plan is 140,705 blocks (~24 GB) without it vs 1,684 blocks (~500 MB) with it — an 83× difference. easy to miss: a client can filter markers locally while still fetching all their blocks.
2. **low observed MB/s during backfill usually isn't bandwidth.** the same ~2 MB/s shows up from a datacenter box one hop from the server — it's the block-fetch pipeline (concurrency × per-request RTT). more parallel getBlock requests raise it; total bytes stay fixed.

**the structural limit**: archive blocks are ~4,096 events in arrival order, so a sparse collection scattered across the network costs (blocks touched) × (~300 KB), regardless of how few rows you want. 17K statusphere rows live in 1,686 blocks → ~500 MB. no client fixes that; only smaller blocks, per-collection layout, or sub-block reads server-side would.

**why you'd pay it anyway**: the archive slice returned **+33% more rows** than listRecords — deleted and updated records, plus everything from the 71 repos (4%!) that were unresolvable or unreachable at fetch time — from one source, with zero per-PDS failure handling, and a gapless cutover into the live tail when you're done. that's the local-first sync story: history + push, not just state + poll.

**rule of thumb**
- one repo → fetch its CAR from the PDS. always.
- many repos, current state, bytes matter → lightrail + listRecords (accept the reachability holes)
- many repos, full history / deletes / unreachable repos / then go live → jetstream v2 slice with plan-level `kinds=commit`
- freshly-bootstrapped archives flatter per-DID slices (bootstrap lays repos out contiguously); the numbers decay as live arrival-order accumulates — benchmark against an aged archive before concluding