ClickHouse remote storage for traces, logs and metrics — Jaeger v2 and otelview compatible
README.md

A complete otelview deployment on ClickHouse #

Three containers and a seed script: ClickHouse, this storage server, and otelview itself with its web UI.

cd examples
docker compose up -d --wait
./seed.py

Then open http://localhost:4319.

seed.py needs nothing but Python 3 — standard library only.

What is running #

seed.py ──OTLP/HTTP:4318──► otelview ──gRPC:17271──► otelview-clickhouse ──HTTP:8123──► ClickHouse
                               │
                               └── web UI :4319, reading back down the same chain

otelview holds no data here. storage.backend: remote in otelview.yaml sends every write and every query to otelview-clickhouse, so what the UI shows is what ClickHouse holds — delete the otelview container and the telemetry survives. Its own status endpoint confirms the arrangement:

$ curl -sS localhost:4319/api/stats
{"spans":630,"logs":148,"metric_points":60,"services":8,
 "backend":"remote (http://otelview-clickhouse:17271) → clickhouse"}
Port Service What for
4317 otelview OTLP gRPC — point an SDK or collector here
4318 otelview OTLP HTTP — what seed.py uses
4319 otelview Web UI and REST API
17271 otelview-clickhouse Jaeger v2 + otelview reader APIs, and OTLP writes
8123 ClickHouse HTTP, for querying the tables yourself
9000 ClickHouse Native protocol, for clickhouse-client

The storage server is built from this repo, not pulled, so the example exercises the code sitting next to it. The first up therefore compiles it.

The demo data #

seed.py simulates a small storefront, backfilling an hour of history:

gateway ─► checkout ─┬─► inventory ─► postgres
                     └─► payments  ─► stripe-api
gateway ─► catalog  ─► redis

Eight services, four levels deep, so the dependency graph is a graph rather than a star. About 8% of checkouts fail at the payment step with a status on the span and an ERROR log carrying the same trace id — which is what makes the log ↔ trace jump in the UI worth trying. Metrics cover all three shapes otelview charts: a monotonic sum, a gauge and a latency histogram.

./seed.py                      # an hour of history
./seed.py --live               # then keep emitting, ~a trace every 2s
./seed.py --minutes 180        # more history
./seed.py --rate 60            # denser traffic
./seed.py --seed 42            # reproducible run

Things worth clicking once it is seeded: the dependency map; a failed checkout trace and the ERROR log attached to it; the http.server.duration histogram; and a free-text log search for declined.

Poking at the storage directly #

The point of this backend is that the data is in plain ClickHouse tables, so you can ask it anything:

# Slowest checkouts in the last hour
curl -sS "localhost:8123/?database=otelview" --data-binary "
  SELECT operation_name, round(duration_nano/1e6, 1) AS ms, hex(trace_id) AS trace
  FROM spans
  WHERE service_name = 'gateway' AND start_time_unix_nano > toUnixTimestamp(now() - INTERVAL 1 HOUR) * 1000000000
  ORDER BY duration_nano DESC LIMIT 5 FORMAT PrettyCompact"

# Error rate per service
curl -sS "localhost:8123/?database=otelview" --data-binary "
  SELECT service_name, count() AS spans, countIf(status_code = 2) AS errors,
         round(100 * errors / spans, 1) AS pct
  FROM spans GROUP BY service_name ORDER BY pct DESC FORMAT PrettyCompact"

# What the listings read — one row per name, not per span
curl -sS "localhost:8123/?database=otelview" --data-binary "
  SELECT * FROM span_names ORDER BY service_name, operation_name FORMAT PrettyCompact"

Or with a client:

docker compose exec clickhouse clickhouse-client --database otelview

The schema is six tables: spans, logs and metric_points hold the telemetry, and span_names, log_service_names and metric_names are the name indexes that keep the UI's dropdowns off the fact tables. See the main README.

Resetting #

docker compose down -v      # -v also drops the ClickHouse volume

Without -v the data survives a restart, which is useful for checking that it really is ClickHouse holding it.

Notes #

"storage: memory" in the UI #

That means you are not looking at this stack. Check what the API says:

$ curl -sS localhost:4319/api/stats
{... "backend":"remote (http://otelview-clickhouse:17271) → clickhouse"}

If it says memory, something else owns the port — most likely another otelview you already had running. The confusing part is that both can be "listening on 4319" at once without either failing to start, because a process bound to 127.0.0.1:4319 and a container published on *:4319 do not collide: they are different addresses. Your browser then picks one by whichever way it resolves localhost, and curl may well pick the other.

lsof -nP -iTCP:4319 -sTCP:LISTEN      # who is actually listening
curl -sS http://127.0.0.1:4319/api/stats   # IPv4
curl -sS "http://[::1]:4319/api/stats"     # IPv6 — can be a different server

Stop the other one and restart this stack. The same applies to the root compose.yml, which publishes 8123 and 17271 for the development stack — docker compose down in the repo root first.

ClickHouse here runs with CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT and a passwordless default user, which is what makes it reachable from the other containers at all. That is fine for a demo on your laptop; do not copy it onto a network you do not own.