---
title: stream
---

available in builds compiled with the `indexer_stream` feature. `/stream` is hydrant's ordered websocket event stream. each subscriber receives a full copy of the stream.

`/stream` is not wire- or protocol-compatible with [`tap`](../concepts/vs-tap.md). tap uses a sharded work queue, server-managed acknowledgements, and redelivery; hydrant uses broadcast delivery and client-managed cursors.

## GET /stream

subscribe to the ordered event stream.

### query parameters

| param | type | description |
| :--- | :--- | :--- |
| `cursor` | integer | inclusive `u64` event ID from which to replay durable record events. |
| `stream` | string | the stream `cursor` came from, see [`/stream/head`](#get-streamhead). when it isn't this one, hydrant sends a `StreamMismatch` error frame and closes the connection. |

if `cursor` is omitted, the connection starts at the current stream head and receives only new events. if it is present, hydrant replays durable record events whose IDs are greater than or equal to the cursor, up to the head observed when the connection starts, then switches to live delivery.

`cursor=0` replays all retained record events. IDs are monotonic but may have gaps: live-only identity and account events consume IDs without being persisted, and events that cannot be inflated are skipped.

### event envelope

events are JSON text frames with this envelope:

```json
{
  "id": 42,
  "type": "record",
  "record": {}
}
```

| field | type | description |
| :--- | :--- | :--- |
| `id` | integer | `u64` stream event ID. |
| `type` | string | `record`, `identity`, or `account`. |
| `record` | object | present when `type` is `record`. |
| `identity` | object | present when `type` is `identity`. |
| `account` | object | present when `type` is `account`. |

only the payload matching `type` is present.

### record

record events are durable and available through cursor replay.

```json
{
  "id": 42,
  "type": "record",
  "record": {
    "live": true,
    "did": "did:plc:abc123xyz",
    "rev": "3kpjxabc123",
    "collection": "app.bsky.feed.post",
    "rkey": "3kpjxabc123",
    "action": "create",
    "record": {
      "text": "hello, world!",
      "$type": "app.bsky.feed.post",
      "createdAt": "2026-05-27T12:00:00.000Z"
    },
    "cid": "bafyreihy..."
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `live` | boolean | `true` for live ingestion; `false` for backfill. |
| `did` | string | repository DID. |
| `rev` | string | repository revision TID. |
| `collection` | string | collection NSID. |
| `rkey` | string | record key. |
| `action` | string | `create`, `update`, or `delete`. |
| `record` | object | decoded DAG-CBOR record, its keys in the order the record's CBOR has them (shorter first), not sorted; omitted when no record content is stored. |
| `cid` | string | CID of `record`; omitted when no record content is stored. |

`record` and `cid` are omitted for deletes and when hydrant is configured not to store record content, including `HYDRANT_ONLY_INDEX_LINKS=true`.

a redaction stops a body from being served once it completes, except in events already queued for a reader at that moment.

### identity

identity events are live-only and are not available through cursor replay.

```json
{
  "id": 43,
  "type": "identity",
  "identity": {
    "did": "did:plc:abc123xyz",
    "handle": "user.bsky.social"
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `did` | string | repository DID. |
| `handle` | string | current handle; omitted when unavailable. |

### account

account events are live-only and are not available through cursor replay.

```json
{
  "id": 44,
  "type": "account",
  "account": {
    "did": "did:plc:abc123xyz",
    "active": false,
    "status": "deactivated"
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `did` | string | repository DID. |
| `active` | boolean | whether the repository is active. |
| `status` | string | account status; omitted when unavailable. |

### websocket behavior

- the server sends an empty ping every 30 seconds and responds to client ping frames with pong frames.
- client text and binary frames are not accepted; hydrant closes the connection when it receives one.
- when `/stream` is in `HYDRANT_THROTTLED_STREAMS` (it isn't by default), a replay from a cursor runs at most `HYDRANT_STREAM_REPLAY_SPEED` times the bandwidth the live stream would take (10 by default), or `HYDRANT_STREAM_REPLAY_FLOOR_BYTES` a second (about 80 Mbit/s) when that's more. the live tail never waits on it.
- a subscriber that falls behind the live broadcast catches up from the database at its own pace. `record` events are never skipped, but `identity` and `account` events it fell behind on are not replayed.
- if hydrant can't read the next events from the database, it closes the connection rather than skip them. reconnect with the last `id` you received as the cursor.
- when hydrant stops, it closes the connection with close code `1012` (service restart). reconnect the same way. under systemd socket activation, the reconnect waits for the next hydrant instead of being refused.
- if sending blocks for `HYDRANT_STREAM_SEND_TIMEOUT` (30 seconds by default), hydrant sends an error text frame and closes the connection:

```json
{
  "type": "error",
  "error": "ConsumerTooSlow",
  "message": "stream socket send blocked for at least 30 seconds"
}
```

## GET /stream/head

get the ID of the newest durable record event, and the stream it belongs to.

```json
{ "id": 289455, "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71" }
```

`id` is `null` while no events are stored. `stream` is the id of this database's stream, new for every database, so two nodes, or one that was wiped, never share one. an event id only means something on the stream it came from. identity and account events consume IDs without being persisted, so this is the newest ID a cursor replay can deliver, not the ID counter.

a consumer can snapshot the head before subscribing with `cursor=0` and treat the replay as caught up once it has seen a frame with `id` greater than or equal to the snapshot. keep an idle-timeout fallback: if the head event itself cannot be inflated, it is skipped and never delivered.

## chunks

with `HYDRANT_STREAM_CHUNK_SPAN` set, hydrant also seals the stream into immutable files a subscriber starting from nothing can read instead of replaying `/stream` from 0: they're compressed, they cost hydrant a file read instead of rendering every event again for every subscriber, and a cache can hold them.

a chunk covers the ids `first` through `last`. like a replay, it holds only record events, never `identity` or `account` ones, and every one of its range. reading the chunks in order, then `/stream` from the id after the last one, delivers exactly what `/stream?cursor=0` does.

### chunk format

a chunk is a [STAR](https://dasl.ing/star.html) (DASL's streaming archive, from its [pull 54](https://tangled.org/dasl.ing/dasl.ing/pulls/54)), compressed with zstd. decompressed, it is

```
2a 6c 01 00 | varint len | meta | ( varint 8 | id | varint len | event )*
```

- the four magic bytes, then the header: an unsigned varint (LEB128, no superfluous trailing `00` byte, at most 9 bytes) holding the length of `meta`, a DRISL map:

  | key | value |
  | :--- | :--- |
  | `$type` | `"systems.gaze.hydrant.streamChunk"` |
  | `stream` | the stream id, as in `/stream/head` |
  | `first` | the first id the chunk covers |
  | `last` | the last id the chunk covers |

- then one pair per event, in id order, each id between `first` and `last` and greater than the one before. the key is 8 bytes, the id big-endian. the value is a DRISL map of the event's `/stream` frame fields, with the record as hydrant stores it:

  | key | DRISL | json frame |
  | :--- | :--- | :--- |
  | `live` | bool | `record.live` |
  | `did` | text | `record.did` |
  | `rev` | text | `record.rev` |
  | `collection` | text | `record.collection` |
  | `rkey` | text | `record.rkey` |
  | `action` | text | `record.action` |
  | `record` | byte string, absent when there's no body | `record.record` |
  | `cid` | tag 42 cid link, absent for a delete and when it's the cidv1 (dag-cbor, sha2-256) of `record`'s bytes | `record.cid`, as a base32 string |

a missing `cid` next to a `record` is the cidv1 of the bytes as they are, not of a re-encode of them. the archive ends after the last pair, with nothing after it. STAR can't tell an archive cut short at a pair boundary from a whole one, so check a chunk against its sha256 before trusting it.

the compressed file is several zstd frames back to back, which decompress as one stream: one for the magic and header, then one or more holding the pairs. that is how a merge works: a new header frame, then each chunk's frames after its own header, as they are.

`record` holds the record's dag-cbor bytes. the json frame has them as json: maps keep their cbor key order, a tag 42 link becomes `{"$link": "<base32 cid>"}`, bytes become `{"$bytes": "<padded base64>"}`, ints, strings, bools, nulls and arrays map as themselves. a body this can't express faithfully (a float, an int past 64 bits, a map key `$link` or `$bytes`, keys out of dag-cbor order or repeated, a `blob` map with more or other fields than `$type`, `ref`, `mimeType` and `size`) is written the way jacquard's `RawData` serializes it, and one that isn't dag-cbor at all leaves `record` out of the json frame. the json frame is then

```
{"id":<id>,"type":"record","record":{"live":..,"did":..,"rev":..,"collection":..,"rkey":..,"action":..[,"record":<body>][,"cid":"<cid>"]}}
```

with no whitespace, which is byte for byte what `/stream` sends.

new chunks are one span each. hydrant merges the newest 4 chunks of the same size into one, up to 16 spans, so the list grows by one chunk every 16 spans and a subscriber can fetch several at once. it builds them on a low priority thread, and while ingest is behind it seals one span every 30 seconds, so catching up on a whole stream (turning chunks on, or after a wipe) waits for ingest.

a redaction or an erase reaches every chunk holding the repository's events. those chunks leave the list until hydrant has built them again from the event rows, with the bodies gone like a replay would have them. the stream itself has no redaction event, so a subscriber that already read a chunk, like one already connected to `/stream`, still holds what it read.

### GET /stream/manifest

the chunks to read, in order.

```json
{
  "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71",
  "generation": 41,
  "through": 917503,
  "chunks": [
    { "first": 786432, "last": 851967, "events": 47512, "bytes": 5104330, "sha256": "9f2c..." },
    { "first": 851968, "last": 917503, "events": 61664, "bytes": 11420553, "sha256": "41ab..." }
  ]
}
```

(the list is shortened here.) the chunks run from id 0 through `through` with no holes, which can end before the newest chunk while one is being rebuilt. read them, then `/stream?cursor={through + 1}&stream={stream}`. if a chunk can't be read, the frames read so far are still a valid place to go on from: continue with `/stream` from the id after the last frame read. `generation` changes whenever the list does. `404` when chunks are off.

### GET /stream/chunk/{sha256}

the chunk's bytes as stored, `application/zstd`, whose sha256 is the name and the `ETag`. a chunk never changes, so it can be cached for good, and a `Range` request gets part of it. a merge leaves the chunks it replaced readable for 15 minutes, for subscribers part way through an older list. a chunk a redaction or erase reached is `404` at once.