stream
jetstream v2 in zig stream.waow.tech
Star
Fork
Watch
stream docs grafana-dashboard.md
14 kB · 266 lines
Markdown
at main

Jetstream V2 and Stream Grafana dashboards #

Stream reuses the upstream dashboard. The reviewed artifact is:

  • upstream commit: f29815c391fc2644f8a3dd36b899fb3697dd1ea6 (upstream's repository, not this one — the path below is upstream's too)
  • path: contrib/grafana/jetstream.json
  • SHA-256: ef6033c182d8c3f9d88af7d91e61126724efbe72e6b940984c9de12e9530cd91

Empty panels during bootstrap are expected, not broken #

Every metric named in this file has been checked against a live instance, but a panel only has data once the code path behind it has run at least once. During bootstrap that leaves three absent from the metrics endpoint:

metric appears when
jetstream_getblock_requests_total serving ungates; getBlock returns 503 until steady state
jetstream_orchestrator_phase_transitions_total the lifecycle first leaves bootstrap
jetstream_orchestrator_state_duration_seconds same — needs a completed phase

Verified on experiment 6 at 54 h elapsed: max_over_time(jetstream_orchestrator_phase[54h]) was still 1, so no transition had ever occurred and those series could not exist. Do not read their absence as a wiring fault.

Also: histograms have no bare series. A *_duration_seconds or *_wait_seconds metric exists only as _bucket, _count and _sum, so grepping the metrics endpoint for the bare name finds nothing while the histogram is present and working. Nine of the names in this file behave that way. Query histogram_quantile(...) or rate(..._sum[5m]) / rate(..._count[5m]), and when checking whether a metric exists, look for the suffixed series.

The exact reviewed bytes are vendored at deploy/grafana/jetstream-upstream.json; deployment does not depend on the network or a moving branch. Stream provisions deploy/grafana/stream.json (SHA-256 d00582d478cdf0c8b00d348dd953c79651c4961e161592bf5eee9f389b3b51d9). It is deterministically generated by tools/adapt_dashboard.py from the pinned source. The generator refuses a changed source checksum, and the tests prove every upstream non-runtime panel remains structurally identical. Stream adds two explicit final rows. The first displays Stream's current dispatch-batch counters. Those counters are not whole-network progress, and the 2026-07-24 audit found that their durability boundary is not equivalent to upstream's per-repository completion batcher. The second covers Stream's scheduler, which has no upstream Go equivalent: ticket boundaries, unsettled stage gaps, current fatal state, and first-failure counters partitioned by stage.

just dashboard-test verifies both checksums, regeneration, and the contract parser. just dashboard-contract <scrape-file-or-localhost-url> extracts every metric family referenced by the Stream dashboard's PromQL and compares it with the candidate's exposition, including histogram child series. It exits non-zero and prints every missing family. This is a schema-presence check, not evidence that a producer has the upstream semantic boundary or remains correct across restart.

Reuse boundary #

Protocol and lifecycle rows are reused verbatim: overview, live firehose, ingest, backfill/retry, serving, storage, integrity, cutover/merge, and timestamp import. Structural reuse does not establish semantic parity. The current producer audit is in semantic-parity.md, and any family whose underlying behavior is blocked must be treated as diagnostic rather than equivalent.

The Go runtime row is the sole adaptation. Stream does not synthesize go_* series. It retains upstream's same-semantics standard process measurements for start time, cumulative CPU, current RSS, and file descriptors; adds current virtual memory and OS thread count; and replaces Go GC and per-process network panels with cumulative page-fault and context-switch counters. Values come from getrusage plus Linux procfs or Darwin proc_pidinfo. Unavailable measurements are omitted, never reported as zero. just process-metrics-contract launches the production binary with only loopback dependencies and validates types, values, invariants, monotonicity, and the absence of go_* series across independent HTTP scrapes. The same receipt proves real startup RocksDB reads, point writes, and batch commits populate the canonical store histogram. A real RocksDB unit receipt covers point-read ok/notfound, durable set/delete, and atomic batch_commit boundaries; registered error series remain zero until RocksDB reports one. just http-metrics-contract seeds a real sealed archive and exercises the ReleaseSafe binary through ordinary HTTP, Range/conditional getBlock requests, and a real WebSocket upgrade/close. It proves the same public-mux middleware boundary and xrpc/ label as upstream, all four getBlock outcomes, exact histogram counts, and full-200-only served-byte accounting. Upstream's debug /metrics and /healthz endpoints and unmatched public requests remain outside the HTTP histogram rather than manufacturing extra handler labels. The pinned official-client archive receipt also proves the resident manifest: startup loads exactly the sealed-segment count, archive-to-live replay produces real block-index hits, unknown-index unit coverage produces misses, and refcounted readers survive a verified metadata refresh. Compaction and timestamp rewrite tests prove the resident checksum/index is refreshed before the rewritten generation is published to later serving work.

Implemented canonical producers #

The following dashboard families already have non-placeholder producers:

  • process_start_time_seconds
  • process_cpu_seconds_total
  • process_resident_memory_bytes
  • process_virtual_memory_bytes
  • process_open_fds
  • process_max_fds
  • process_minor_page_faults_total
  • process_major_page_faults_total
  • process_voluntary_context_switches_total
  • process_involuntary_context_switches_total
  • stream_process_threads
  • stream_pipeline_ticket
  • stream_pipeline_stage_gap
  • stream_pipeline_fatal
  • stream_pipeline_failures_total
  • jetstream_build_info
  • jetstream_livestream_events_received_total
  • jetstream_livestream_decode_errors_total
  • jetstream_livestream_reconnects_total
  • jetstream_livestream_sequence_gaps_total
  • jetstream_livestream_sequence_gap_missed_seqs_total
  • jetstream_livestream_unknown_events_total
  • jetstream_livestream_stream_error_frames_total
  • jetstream_livestream_last_seen_upstream_event_timestamp_seconds
  • jetstream_livestream_replayed_account_events_dropped_total
  • jetstream_livestream_replayed_identity_events_dropped_total
  • jetstream_ingest_events_appended_total (counted in the shared archive writer, so live, bootstrap, retry, and resync producers all contribute)
  • jetstream_ingest_blocks_flushed_total
  • jetstream_ingest_segments_rotated_total
  • jetstream_ingest_append_errors_total
  • jetstream_ingest_active_segment_bytes
  • jetstream_ingest_readable_log_bytes
  • jetstream_ingest_readable_log_pinned_bytes
  • jetstream_ingest_readable_log_pinned_overrun_bytes
  • jetstream_ingest_dropped_events_total
  • jetstream_segment_seal_duration_seconds
  • jetstream_data_dir_free_bytes
  • jetstream_http_request_duration_seconds
  • jetstream_getblock_requests_total
  • jetstream_getblock_served_bytes_total
  • jetstream_getblock_duration_seconds
  • jetstream_manifest_segments_loaded
  • jetstream_manifest_block_index_cache_hits_total
  • jetstream_manifest_block_index_cache_misses_total
  • jetstream_manifest_block_index_load_seconds
  • jetstream_store_op_duration_seconds
  • jetstream_orchestrator_phase
  • jetstream_orchestrator_phase_transitions_total
  • jetstream_orchestrator_state_duration_seconds
  • jetstream_orchestrator_merge_events_kept_total
  • jetstream_orchestrator_merge_events_dropped_total
  • jetstream_orchestrator_merge_segments_consumed_total
  • jetstream_orchestrator_merge_did_lookups_total
  • jetstream_orchestrator_merge_repo_revs_updated_total
  • jetstream_orchestrator_merge_dids_discovered_post_bootstrap_total
  • jetstream_verifier_failures_total
  • jetstream_backfill_discovered_total
  • jetstream_backfill_completed_total
  • jetstream_backfill_failed_total
  • jetstream_backfill_active_flips_total
  • jetstream_backfill_on_fail_store_errors_total
  • jetstream_backfill_handle_repo_duration_seconds
  • jetstream_backfill_progress_completed
  • jetstream_backfill_completion_queued_total
  • jetstream_backfill_completion_queue_depth
  • jetstream_backfill_completion_durable_batches_total
  • jetstream_backfill_completion_durable_repos_total
  • jetstream_backfill_completion_stage_errors_total
  • jetstream_backfill_completion_queue_wait_seconds
  • jetstream_backfill_forced_checkpoint_flushes_total
  • jetstream_backfill_failed_repo_retry_passes_total
  • jetstream_backfill_failed_repo_retry_candidates_total
  • jetstream_backfill_failed_repo_retry_attempts_total
  • jetstream_backfill_failed_repo_retry_succeeded_total
  • jetstream_backfill_failed_repo_retry_failed_total
  • jetstream_backfill_failed_repo_retry_skipped_host_parked_total
  • jetstream_subscribe_subscribers
  • jetstream_subscribe_events_sent_total
  • jetstream_subscribe_bytes_sent_total
  • jetstream_subscribe_bytes_encoded_total
  • jetstream_subscribe_events_filtered_total
  • jetstream_subscribe_events_oversize_total
  • jetstream_subscribe_events_skipped_total
  • jetstream_subscribe_encode_errors_total
  • jetstream_subscribe_options_updates_total
  • jetstream_subscribe_options_update_errors_total
  • jetstream_subscribe_cursor_requests_total
  • jetstream_subscribe_cursor_resolve_seconds
  • jetstream_subscribe_hot_reads_total
  • jetstream_subscribe_cold_reads_total
  • jetstream_subscribe_adversarial_drops_total
  • jetstream_subscribe_clean_disconnects_total
  • jetstream_import_jobs_total
  • jetstream_import_job_duration_seconds
  • jetstream_import_phase
  • jetstream_import_rows_parsed_total
  • jetstream_import_rows_rejected_total
  • jetstream_import_rows_mutated_total
  • jetstream_import_rows_matched_total
  • jetstream_import_rows_corrupt_offset_total
  • jetstream_import_dids_matched_total
  • jetstream_import_segments_examined_total
  • jetstream_import_segments_patched_total
  • jetstream_import_bytes_rewritten_total
  • jetstream_compaction_passes_total
  • jetstream_compaction_pass_duration_seconds
  • jetstream_compaction_passes_early_total
  • jetstream_compaction_tombstones_collected_total
  • jetstream_compaction_tombstone_set_entries
  • jetstream_compaction_tombstone_set_bytes
  • jetstream_compaction_segments_examined_total
  • jetstream_compaction_segments_rewritten_total
  • jetstream_compaction_segments_skipped_clean_total
  • jetstream_compaction_manifest_reconciled_total
  • jetstream_compaction_rows_dropped_total
  • jetstream_compaction_bytes_rewritten_total
  • jetstream_compaction_watermark_seq
  • jetstream_compaction_watermark_lag_seconds

The Stream-only backfill chart plots absolute repository counts across the current run over time: discovered, processed, and durably committed. It deliberately does not present a whole-network percentage while listRepos is still discovering the denominator. Upstream requests are pages of 1,000 entries; the production default accumulates roughly 100,000 entries into a shuffled dispatch/checkpoint batch before advancing durable progress.

The backfill completion families are measured at the writer durability boundary, independently of the dispatch/checkpoint batch. A successful CAR records its final archive sequence; ordinary writer fsyncs commit every covered repository state, and the terminal dispatch drain refuses to advance the listRepos cursor if any queued completion is not covered. Queue and durable batch/repository metrics therefore describe the same boundaries as upstream. The Prometheus counters themselves are process-local; restart-stable progress must be reconstructed from durable repository-state counts.

Compaction metrics are emitted at the actual fold, worker rewrite, manifest refresh, watermark commit, live-set, and timer boundaries. The production offline receipt reports required=104 present=104 missing=0. That result means only that every queried family is exposed; it does not establish producer semantics or close the deployment gate.

Dashboard inventory (source of truth: deploy/grafana/) #

uid title audience notes
stream-glance stream / at a glance operator 4 rows: pipeline alive (ingest rate, upstream freshness, reconnects+gaps) / consumers served (subscribers, delivery, archive bytes) / margins+faults (disk free, watermark lag, flat-zero fault rates) / memory-allocator (rss vs malloc live, free-held, arena heaps — zlay lesson 8)
stream-consumers stream / archive consumers operator per-key throughput, cumulative, 429s, bucket headroom, configured rates
stream-public-full Stream / Jetstream V2 public adapted upstream dashboard (see top of this file)

To edit: change the JSON in deploy/grafana/, then POST it —

python3 -c 'import json; d=json.load(open("deploy/grafana/<file>.json")); \
  print(json.dumps({"dashboard": d, "overwrite": True}))' > /tmp/post.json
# box: admin password in /opt/stream-experiment/grafana-password.txt
# POST /tmp/post.json to http://localhost:3000/api/dashboards/db inside
# the grafana container with basic auth (busybox wget needs an explicit
# "Authorization: Basic <base64>" header; URL-embedded creds are ignored)

Edits made in the Grafana UI are not durable state — anything worth keeping gets exported back into deploy/grafana/.

Panels must only name series that exist on the live instance: a panel over an unshipped metric renders "No data" and reads as an outage. Series shipping with the next stream deploy — import their panels after it, not before: admission counters (stream_subscribe_capacity_rejects_total, stream_subscribe_cold_contended_total) and the all-arena malloc gauges (stream_malloc_system_bytes, stream_malloc_free_total_bytes, stream_malloc_arena_heaps — the glance memory row depends on these).