otelview-clickhouse
ClickHouse remote storage for traces, logs and metrics — Jaeger v2 and otelview compatible
Star
Fork
Watch
7 1 0
Files
.github
benches
examples
migrations
proto
scripts
src
systemd
tests
.dockerignore
.gitignore
build.rs
Cargo.lock
Cargo.toml
compose.yml
Dockerfile
flake.lock
flake.nix
install.sh
LICENSE
README.md
README.md

otelview-clickhouse #

ci nix

ClickHouse remote storage for all three OpenTelemetry signals — traces, logs and metrics — implemented in Rust with Tonic and the official clickhouse client. A single listener is a complete remote backend for an otelview instance and a Jaeger v2 remote trace storage at the same time.

One gRPC listener provides:

  • OTLP TraceService, LogsService and MetricsService Export for writes
  • Jaeger jaeger.storage.v2.TraceReader for trace queries
  • Jaeger jaeger.storage.v2.DependencyReader for the dependency graph
  • otelview otelview.storage.v1.LogReader for log search
  • otelview otelview.storage.v1.MetricReader for metric listing and series
  • otelview otelview.storage.v1.Diagnostics for storage counters

The original OTLP payloads (resource, scope, span / log record / metric data point) are stored as protobuf, so reads stream back exactly what was ingested — histogram buckets, exemplars, quantiles and schema URLs included. Alongside each payload sit the columns the readers filter on, flattened for search.

Table of Contents #

Install #

A release binary #

The installer detects macOS ARM64, Linux x86_64 or Linux ARM64 and verifies the download against the release checksums:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://raw.githubusercontent.com/tsirysndr/otelview-clickhouse/main/install.sh | sh

A root Linux install also places the unit at /etc/systemd/system/otelview-clickhouse.service and writes a private environment file. Install system-wide, point it at your ClickHouse, and start it:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://raw.githubusercontent.com/tsirysndr/otelview-clickhouse/main/install.sh | sudo sh
sudo editor /etc/otelview-clickhouse/env
sudo systemctl enable --now otelview-clickhouse
sudo systemctl status otelview-clickhouse

The unit is installed but deliberately not enabled: CLICKHOUSE_URL defaults to http://localhost:8123, which is right often enough to be dangerous and wrong often enough to be worth a look first.

Set OTELVIEW_CLICKHOUSE_VERSION=v0.1.0 to pin a tag, INSTALL_DIR to change the binary destination, or INSTALL_SYSTEMD=0 to skip the unit.

Tags matching v* build Darwin ARM64, Linux x86_64 and Linux ARM64; the release workflow publishes all three archives and SHA256SUMS to the GitHub release.

With Docker #

compose.yml brings up ClickHouse and this server together:

docker compose up -d --wait

For a whole working deployment — ClickHouse, this server, otelview and its web UI, plus a script that seeds realistic demo telemetry — see examples/:

cd examples && docker compose up -d --wait && ./seed.py
# then open http://localhost:4319

With Nix #

The flake builds the server with crane for aarch64-darwin, x86_64-linux and aarch64-linux, with pre-built artifacts served from the otelview Cachix cache (advertised through the flake's nixConfig, so Nix offers it automatically):

# Run without installing
nix run github:tsirysndr/otelview-clickhouse

# Install into your profile
nix profile install github:tsirysndr/otelview-clickhouse

# Or from a checkout
nix build .#otelview-clickhouse
./result/bin/otelview-clickhouse

To enable the cache permanently — required when your user is not in trusted-users — use Cachix directly:

cachix use otelview

Or add it to /etc/nix/nix.conf (or nix.settings on NixOS):

extra-substituters = https://otelview.cachix.org
extra-trusted-public-keys = otelview.cachix.org-1:+Twrf64f2rg+cTAYU2MikV/hGMpHxnHK6l6yLvrseP4=

nix develop drops into a shell with the Rust toolchain, protoc and the tools the test suites expect. ClickHouse itself is not in the shell — it speaks HTTP, so docker compose up -d clickhouse is what provides it.

From source #

cargo build --release
./target/release/otelview-clickhouse

Needs protoc on the path for the tonic codegen.

Run #

Nothing is required but a reachable ClickHouse; the database, tables, indexes and materialized views are created at startup.

export CLICKHOUSE_URL='http://localhost:8123'
otelview-clickhouse

After connecting and applying the schema, startup prints a banner with the configured gRPC address and the services sharing its port.

Configuration:

Variable Default Meaning
CLICKHOUSE_URL http://localhost:8123 Primary HTTP endpoint (reads + writes)
CLICKHOUSE_READ_URL unset Optional read-only replica; see below
CLICKHOUSE_DATABASE otelview Database holding the tables
CLICKHOUSE_USER / CLICKHOUSE_PASSWORD default / empty Credentials
LISTEN_ADDR 0.0.0.0:17271 Combined OTLP, Jaeger and otelview gRPC address
MAX_SEARCH_DEPTH 1000 Upper bound for trace and log searches
ASYNC_INSERT false Let ClickHouse coalesce concurrent exports server-side (why it is off)
QUERY_TIMEOUT_SECS 30 Server-side ceiling on one query
RETENTION unset Delete telemetry older than this: 36h, 7d, 2w, 1mo (months are 30 days). Unset keeps everything
RETENTION_SWEEP_INTERVAL 1h How often the retention sweep runs
RUST_LOG otelview_clickhouse=info Log filter

Every gRPC request emits a start event, a completion event with busy/idle timings, and an error event when the RPC fails. Request payloads are not logged.

Use as an otelview backend #

storage:
  backend: remote
  remote:
    endpoint: "http://127.0.0.1:17271"

Use as a Jaeger backend #

Point the Jaeger v2 remote-storage backend at port 17271. The backend also implements the standard OTLP writer on that same endpoint, as the Jaeger storage v2 contract requires.

Read/write split #

Set CLICKHOUSE_READ_URL to a replica and every query (trace search, log search, metric series, stats, dependency graph) runs against it while OTLP writes and migrations stay on CLICKHOUSE_URL. The read client always runs with readonly = 2, so it cannot write even when the URL is the primary.

Schema #

Three fact tables, one per signal, and three small name indexes that keep the UI's listings off them.

Table Engine Sorting key
spans ReplacingMergeTree(inserted_at) (service_name, start_time_unix_nano, trace_id, span_id)
logs MergeTree (service_name, time_unix_nano)
metric_points MergeTree (metric_name, service_name, time_unix_nano)
span_names AggregatingMergeTree (service_name, operation_name, span_kind)
log_service_names AggregatingMergeTree service_name
metric_names AggregatingMergeTree (metric_name, service_name)

All three fact tables are partitioned by day and carry ttl_only_drop_parts = 1. The migrations are in migrations/ for operators who manage DDL separately; each file explains the choices it makes.

How the queries are kept fast #

The design notes live next to the code, but the shape of it:

Sorting keys follow the query, not the data model. service_name leads the span and log keys because a service-scoped search — the common one — then reads one contiguous range already ordered by time. metric_name leads the metric key because no reader ever asks for "all metrics in a window"; every metric query names its metric.

Skip indexes carry the access paths the key cannot. A minmax index on the time column is unusually effective here: rows within a service's key range are already time-ordered, so every granule has a tight range and an unfiltered time window still prunes nearly all of them. Trace-id lookups use a bloom filter; attribute search probes mapKeys and mapValues bloom filters separately, so a granule survives only if it holds both the key and the value.

No FINAL. spans is a ReplacingMergeTree, which gives re-exported spans last-write-wins semantics — but only after a merge. Forcing that at read time would merge every part on every read. Instead each reader collapses duplicates with the aggregate its shape needs anyway: argMax(col, inserted_at) for whole rows, min/max where duplicates cannot change the answer, uniqExact where it is counting. Same answer, a fraction of the work, and correct immediately rather than after a merge.

The listings are O(names), not O(telemetry). GetServices, GetOperations and ListMetrics run on every page load. Answered from the fact tables they cost a full column scan: ClickHouse has no loose index scan, so there is no way to skip along a sorting key taking one row per distinct value. Materialized views instead maintain one row per name as data arrives, and the listings read those. Over the 4M-row dataset below:

Listing From the fact table From the name index
GetServices 1,000,000 rows / 985 KB 1,600 rows / 2.7 KB
GetOperations 324,000 rows / 706 KB 1,600 rows / 15.2 KB
LogReader/GetServices 1,000,000 rows / 978 KB 20 rows / 308 B
ListMetrics 2,000,000 rows / 22.9 MB 1,050 rows / 18.9 KB
MetricReader/GetServices 2,000,000 rows / 1.9 MB 1,050 rows / 1.3 KB

The left column grows with ingest; the right column does not. The benchmark runs both and asserts they return identical answers, so the index cannot quietly drift from the table it is built from.

Because a view only sees inserts made after it exists, migrate backfills each index once from its fact table — so an existing deployment that upgrades into these tables does not lose its listings. The index records names as they were seen, so a name that stops appearing stays listed until its retention TTL expires it.

Downsampling happens in ClickHouse. FindMetrics cuts the window into max_points equal buckets and keeps each series' earliest point per bucket — one streaming aggregation, no window functions. Shipping every raw point and thinning afterwards moves megabytes per chart.

The dependency graph makes one pass, with no join. The obvious shape is a self-join of spans on (trace_id, parent_span_id) = (trace_id, span_id), which is what ClickHouse is worst at: it builds a hash table over the whole window before emitting a row. Instead each trace is folded into arrays and a span_id -> service map, and every span's parent is a map lookup.

Free-text log search reads one column. A search_text column materialises body, severity text and attribute values lowercased at write time, with an n-gram bloom filter over it — the only skip index that accelerates an arbitrary substring. Matching the three columns separately as ILIKE ... OR ILIKE ... would read three columns and prune nothing, because ClickHouse can only prune an OR when every branch is indexable.

The obvious index here is tokenbf_v1, and it is the wrong one: measured on ClickHouse 26.8 it prunes nothing at all for LIKE — 79 of 79 granules survive even for a needle present in no row — because it only serves hasToken and friends. Switching to hasToken would have meant whole-word matching and a search box that cannot find checkout by typing check. ngrambf_v1 keeps exact substring semantics and actually prunes: a rare term reads 33k rows instead of 940k.

Storage counters are metadata reads. GetStats sums rows from system.parts rather than running count() over three large tables. On spans that counts rows a merge has not yet collapsed, so a span re-exported before its next merge is counted twice — an approximation a status tile can live with, and one that converges.

Ingest is one request per export. Rows stream out as RowBinary in a single HTTP request — there is no bind-parameter ceiling to chunk around, and splitting an export would only create more parts to merge.

async_insert is off, against ClickHouse's own default. It exists to coalesce many small inserts, and an OTLP export is already one well-sized request, so the async buffer only adds a queue hop and up to async_insert_busy_timeout_max_ms of waiting per call. Measured both ways on this write path:

Export shape async_insert=1 async_insert=0
collector batches, 2000 rows × 8 writers 60,839 rows/s 228,242 rows/s
SDK-sized batches, 50 rows × 64 writers 13,199 rows/s 13,898 rows/s

Turn it on if ingest is spread over enough small writers that part count, rather than throughput, is the constraint — it does keep part count down. Note that the setting has to be sent explicitly in both directions: ClickHouse has defaulted it to 1 since 25.x, so omitting it does not mean off.

Retention #

RETENTION sets how long a row may live. Neither mechanism deletes a row:

  • A periodic sweep. Since the fact tables are partitioned by day, a day entirely past the cutoff goes with DROP PARTITION — a metadata operation, instant whatever the partition holds. This clears the bulk.
  • A table TTL, applied at startup, for the boundary day and anything the sweep has not reached. With ttl_only_drop_parts = 1 ClickHouse drops whole expired parts during ordinary merges, and it keeps working whether or not this process is running.

Notably absent is a DELETE for the rows inside the boundary day. That is the obvious way to make expiry exact, and it was the first design here, but a lightweight delete is a mutation: measured on this schema it costs one to three seconds per table even when it matches nothing, and it leaves a _row_exists mask that every later read pays for until a merge clears it. Run hourly across three tables, forever, that is a great deal of work to buy sub-day precision on a window measured in days. Dropping it took the sweep from 6.1 s to 0.36 s, and an idle pass from 3.9 s to 13 ms.

So the guarantee is deliberately coarse: a row may outlive its retention by up to the remainder of its day, in exchange for never rewriting a part.

Benchmarks #

One seeded run of every storage path, on an Apple M4 laptop against ClickHouse 26.8 in Docker. Dataset: 1M spans across 200k traces, 1M logs, 2M metric points — 4M rows over a two-week window, 20 services, 200 operations, 50 metrics, attributes on every row. 111 MB on disk compressed, 1.63 GB uncompressed (14.9x).

docker compose up -d clickhouse --wait
cargo bench --bench clickhouse -- 1000000

Every row goes in through the real OTLP write path, so the payloads are genuine encoded protobuf. Every read goes through the same Store methods the gRPC handlers call. The rows and bytes each query touched are read back out of ClickHouse's own system.query_log rather than estimated, and the read phase waits for the seed's merges to finish first, so it measures query cost rather than merge contention.

Ingest #

Eight concurrent exporters, 2000 rows per export.

Signal Rows Wall clock Throughput
spans 1,000,000 4.8 s 206k rows/s
logs 1,000,000 6.3 s 158k rows/s
metric points 2,000,000 8.3 s 240k rows/s

Reads #

30 iterations each after a warm-up. "Rows read" is what ClickHouse scanned to answer the query, against 1M spans / 1M logs / 2M points.

Query p50 p95 rows read bytes read returned
GetServices 1.6 ms 2.4 ms 1,600 2.7 KB 20 services
GetOperations, one service 1.9 ms 3.1 ms 1,600 16.9 KB 10 operations
LogReader/GetServices 0.8 ms 1.3 ms 20 308 B 20 services
MetricReader/GetServices 1.5 ms 2.2 ms 1,050 1.3 KB 20 services
ListMetrics 3.7 ms 6.0 ms 1,050 37.4 KB 50 metrics
GetStats 5.8 ms 8.4 ms 141 6.4 KB —
FindTraceIDs, no service, 1h window 3.2 ms 5.2 ms 22,315 762.7 KB 20 traces
FindTraceIDs, service + 1h window 4.8 ms 8.0 ms 10,192 358.6 KB 20 traces
FindTraceIDs, attribute + 1h window 12.7 ms 28.6 ms 22,315 3.50 MB 20 traces
GetTraces by id 6.7 ms 12.1 ms 16,384 5.53 MB 5 spans
GetTraces by id + time window 7.8 ms 9.8 ms 16,384 5.53 MB 5 spans
FindTraceSummaries, service + 1h window 14.4 ms 33.1 ms 50,662 4.80 MB 20 summaries
GetDependencies, 1h window 10.5 ms 14.9 ms 22,315 1.03 MB 20 edges
FindLogs, service + 1h window 4.1 ms 7.4 ms 25,459 747.9 KB 100 records
FindLogs, trace correlation 6.7 ms 16.3 ms 16,384 2.20 MB 1 record
FindLogs, search for a rare term 8.0 ms 13.7 ms 32,768 4.40 MB 1 record
FindLogs, search for a common term 24.0 ms 39.4 ms 870,273 108.17 MB 100 records
FindMetrics, one service (10 series) 18.5 ms 26.2 ms 229,376 28.17 MB 2,000 points
FindMetrics, all services (200 series) 56.4 ms 122.4 ms 292,644 38.91 MB 40,000 points

Two rows are worth reading twice. The common-term log search scans nearly the whole table, and correctly so: tenant-421 occurs in one record in a thousand, uniformly, so it is genuinely present in almost every granule and no index could prune it. The rare-term search on the same column and the same query shape reads 33k rows instead of 870k, which is the n-gram filter doing its job. And FindMetrics scales with the number of series, not the number of points, because max_points is a per-series budget — a chart should pass service_name.

Retention #

Rows before Rows removed Wall clock Rate
4,800,000 1,857,385 0.36 s 5.1M rows/s

A pass with nothing left to expire takes 13 ms, which matters more: that is the one that runs every hour forever.

Caveats #

A laptop under Docker is not a server, and the virtualised filesystem makes tails noisy — p95 is routinely two to three times p50 and moves run to run, while p50 and the rows/bytes columns are stable. The rows and bytes are the numbers to compare against another design; the milliseconds are indicative. The dataset is uniform, which is pessimistic for skip indexes: real telemetry clusters by service and time, so granules prune better than they do here.

Test #

cargo test            # unit tests only; no database needed

The store-level suite in tests/integration.rs exercises span upsert and search, attribute containment, log search and severity/trace-correlation filters, metric downsampling and listing, the name-index backfill, stats, the read-only client and the retention sweep — against a real ClickHouse when TEST_CLICKHOUSE_URL is set, and skips itself otherwise:

docker compose up -d clickhouse --wait
TEST_CLICKHOUSE_URL=http://localhost:8123 cargo test

tests/e2e.rs touches no Rust API at all: telemetry goes in through the OTLP Export RPCs and comes back out through the Jaeger and otelview readers, over real gRPC, against the server container talking to the ClickHouse container. scripts/e2e.sh brings the stack up with docker compose, runs both suites and tears it down:

scripts/e2e.sh              # build, run, tear down
KEEP_UP=1 scripts/e2e.sh    # leave the stack running afterwards

cargo bench --bench clickhouse seeds a dataset and times every path against a real ClickHouse; see Benchmarks. It takes a span count as its first argument.

The ci workflow runs the unit and integration suites against a ClickHouse started from compose.yml, plus clippy and rustfmt, and the compose end-to-end run as a separate job.

License #

MIT