Performance checks #
test/perf/run.py runs daemon scenarios against recorded
provider responses and produces measurements and comparison results. The runner
measures memory, HTTP traffic, and disk writes. These checks describe the test
workload, not production performance.
Commands #
A full replay requires Linux /proc and derives gates. The default is five
repetitions per scenario, with repetitions concurrent unless --jobs limits
them.
nix develop -c python3 test/perf/run.py --mode replay --runs 5 --out /tmp/perf/results.json
nix develop -c python3 test/perf/compare.py /tmp/perf/results.json --baseline /tmp/perf/baseline.json
--scenario NAME selects a diagnostic subset and does not derive gates. Record
mode requires --runs 1 and --url. It records responses instead of measuring.
run.py defines the arguments and exit status.
scenarios.py defines the workloads.
Measurements #
measurement.py discovers the daemon process
tree, samples memory and write counters, and gathers BEAM checkpoints. Kernel
and process measurements depend on Linux process information.
transport.py proxies daemon and provider traffic
and tracks request and response bytes. The recordings replay response bytes, not
upstream packet timing.
Replay compares request digests with the recording and reports stale turns. A
completed scenario also checks for unused recorded responses. A missing or
surplus response fails replay. See request_digests and Recordings in
transport.py.
Metric definitions #
run.py derives the metric families below.
budgets.toml defines each gate, its unit,
ceiling, and tolerance floor. Keep workload and calibration rationale with those
entries.
| Family | Meaning and interpretation |
|---|---|
| Repeated samples | summary reports the median and median absolute deviation, or MAD. A single sample cannot estimate noise. |
| BEAM retained memory | beam_kept subtracts unused process-heap space and binaries not referenced by a process from major-GC total memory. It is not process RSS or USS. |
| Growth | robust_slope uses the median of pairwise slopes. Turn and kernel-cell metrics omit their configured warm-up region. Warm-up retention is reported separately. |
| Linux process memory | measurement.tree reads PSS and private-memory counters from smaps_rollup. Anonymous PSS and the private estimate include SwapPss, so swapping does not make owned memory disappear. This is not plain resident USS. |
| Kernel and pool memory | Kernel startup, retained cell growth, fork sharing, and pool observations have different scopes. Use the named gate's definition rather than summing tiles. Some pool comparisons are diagnostic rather than gated. |
| Physical disk writes | measurement.Writes totals write_bytes minus cancelled_write_bytes over the tracked process tree. Reaped child counters and last observations account for exited processes. |
| Per-file write activity | The preloaded write shim records bytes returned by intercepted write calls. It misses some writes and measures calls rather than physical page writeback. It identifies files, not an additive breakdown of physical disk totals. |
| Disk windows | write_window applies workload-specific windows. Long-run disk gates exclude warm-up, and per-turn physical writes use a median because WAL checkpoints are lumpy. Other reports can exclude database backfill when measuring append growth. |
| HTTP bytes | transport.py counts request and response bytes through the local proxy. Replay preserves response bytes, not provider latency or upstream packet timing. |
The scenario definitions set warm-up and scaled idle periods. A scaled idle experiment represents its configured duration, not an equal amount of wall-clock production observation. Read the metric and window together before comparing runs.
Runtime admission measurements #
admission_measure_test.py uses the
shared real-daemon harness to burst 24 sessions through concurrent opens,
composition observations and stops. Its
probe uses public runtime
entrypoints. Run the same fixture normally and under CPU/disk contention:
ALBEDO_ADMISSION_REPORT=/tmp/admission.jsonl nix develop -c python3 test/e2e/run.py "$PWD/test/manual/admission_measure_test.py"
ALBEDO_ADMISSION_REPORT=/tmp/admission-stress.jsonl nix develop -c python3 test/manual/stress.py python3 test/e2e/run.py "$PWD/test/manual/admission_measure_test.py"
runtime/measurement enables
JSONL samples only when ALBEDO_ADMISSION_MEASURE names an output file. The
fixture sets this on its private harness daemon. Samples include per-class queue
wait and service duration, oldest queued age, active workers, runtime and
reservation-owner mailbox counts, process/table bytes, serialized queued bytes,
and reservation/refusal/cancellation counts. Started and finished events share a
worker ticket. Finished events distinguish confirmed_stop, failed_stop,
success, error, uncertain, and worker_death. For confirmed_stop, queue
plus service duration measures worker confirmation latency. The burst summary
records success, error, crash, and timeout counts per public operation, and the
fixture fails when any operation does not succeed. Queue age starts when the
runtime first samples a queued request; reservation age includes producer and
deferred waiting.
Keep reports under /tmp, with the revision, working diff, environment and
command used. JSONL file writes add measurement overhead, and one burst does not
establish production latency or justify scheduling changes. Producer byte
budgets count captured heaps and backing binaries conservatively; serialized
queued bytes measure a different representation. The
runtime contract defines the bounds
these measurements observe.
Record provider responses #
Direct record mode is part of the runner, not the CI report gate. It requires
--runs 1 and --url. Recording contacts the selected upstream and produces
artifacts that need review before sharing.
The intended container workflow is
record_container.py, recorded as running
on valefar. It requires Linux, Docker, the selected local image, the bonsai
bridge setup, and its build tools. It prefers runsc when available and
otherwise uses runc. It serializes access through /tmp/bonsai-gpu.lock and
runs the workload without a container network, with a dedicated bridge to the
provider.
nix develop -c python3 test/perf/record_container.py --out /tmp/new-perf-recordings
The output directory must not exist. --scenario can select recordings. The
script's isolation probe checks selected network and host-file access before
recording. Neither the host name nor a chosen container runtime alone proves
isolation. Inspect the script's prerequisites before running it on another host.
Comparison #
A full replay derives metrics in run.py.
compare.py applies ceilings and, when a baseline is
supplied, regression tolerances from
budgets.toml. Byte metrics use 1% of the
baseline. Disk metrics use the larger of three times baseline noise and the
configured floor. Other metrics also include 5% of the baseline in that maximum.
A result over its ceiling fails even without a baseline.
The allocator peak from
albedo_perf_probe sums each allocator
instance's high-water mark. Those maxima need not occur together, so this is an
upper bound on simultaneous allocated memory. Its comparison floor accounts for
variation between repeated batches on unchanged production code. Keep the
calibration runs and environment beside its entry in budgets.toml.
Output safety #
safety.py scans output for supplied environment
secrets and recognizable credential patterns before writing JSON. This is a
guard, not proof that a recording contains no sensitive data. Recording contacts
the supplied upstream and can fail. Inspect the generated artifact before
sharing it.
See performance reporting for CI baseline selection, publication, external deployment observations, and report interpretation. The report comparison picker does not replace the gate's per-metric baseline.