small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
albedo robot-docs perf.md
10.0 kB · 155 lines
Markdown
at main

Performance checks #

test/perf/run.py runs daemon scenarios against recorded provider responses and produces measurements and comparison results. The runner measures memory, HTTP traffic, and disk writes. These checks describe the test workload, not production performance.

Commands #

A full replay requires Linux /proc and derives gates. The default is five repetitions per scenario, with repetitions concurrent unless --jobs limits them.

nix develop -c python3 test/perf/run.py --mode replay --runs 5 --out /tmp/perf/results.json
nix develop -c python3 test/perf/compare.py /tmp/perf/results.json --baseline /tmp/perf/baseline.json

--scenario NAME selects a diagnostic subset and does not derive gates. Record mode requires --runs 1 and --url. It records responses instead of measuring. run.py defines the arguments and exit status. scenarios.py defines the workloads.

Measurements #

measurement.py discovers the daemon process tree, samples memory and write counters, and gathers BEAM checkpoints. Kernel and process measurements depend on Linux process information. transport.py proxies daemon and provider traffic and tracks request and response bytes. The recordings replay response bytes, not upstream packet timing.

Replay compares request digests with the recording and reports stale turns. A completed scenario also checks for unused recorded responses. A missing or surplus response fails replay. See request_digests and Recordings in transport.py.

Metric definitions #

run.py derives the metric families below. budgets.toml defines each gate, its unit, ceiling, and tolerance floor. Keep workload and calibration rationale with those entries.

Family Meaning and interpretation
Repeated samples summary reports the median and median absolute deviation, or MAD. A single sample cannot estimate noise.
BEAM retained memory beam_kept subtracts unused process-heap space and binaries not referenced by a process from major-GC total memory. It is not process RSS or USS.
Growth robust_slope uses the median of pairwise slopes. Turn and kernel-cell metrics omit their configured warm-up region. Warm-up retention is reported separately.
Linux process memory measurement.tree reads PSS and private-memory counters from smaps_rollup. Anonymous PSS and the private estimate include SwapPss, so swapping does not make owned memory disappear. This is not plain resident USS.
Kernel and pool memory Kernel startup, retained cell growth, fork sharing, and pool observations have different scopes. Use the named gate's definition rather than summing tiles. Some pool comparisons are diagnostic rather than gated.
Physical disk writes measurement.Writes totals write_bytes minus cancelled_write_bytes over the tracked process tree. Reaped child counters and last observations account for exited processes.
Per-file write activity The preloaded write shim records bytes returned by intercepted write calls. It misses some writes and measures calls rather than physical page writeback. It identifies files, not an additive breakdown of physical disk totals.
Disk windows write_window applies workload-specific windows. Long-run disk gates exclude warm-up, and per-turn physical writes use a median because WAL checkpoints are lumpy. Other reports can exclude database backfill when measuring append growth.
HTTP bytes transport.py counts request and response bytes through the local proxy. Replay preserves response bytes, not provider latency or upstream packet timing.

The scenario definitions set warm-up and scaled idle periods. A scaled idle experiment represents its configured duration, not an equal amount of wall-clock production observation. Read the metric and window together before comparing runs.

Runtime admission measurements #

admission_measure_test.py uses the shared real-daemon harness to burst 24 sessions through concurrent opens, composition observations and stops. Its probe uses public runtime entrypoints. Run the same fixture normally and under CPU/disk contention:

ALBEDO_ADMISSION_REPORT=/tmp/admission.jsonl nix develop -c python3 test/e2e/run.py "$PWD/test/manual/admission_measure_test.py"
ALBEDO_ADMISSION_REPORT=/tmp/admission-stress.jsonl nix develop -c python3 test/manual/stress.py python3 test/e2e/run.py "$PWD/test/manual/admission_measure_test.py"

runtime/measurement enables JSONL samples only when ALBEDO_ADMISSION_MEASURE names an output file. The fixture sets this on its private harness daemon. Samples include per-class queue wait and service duration, oldest queued age, active workers, runtime and reservation-owner mailbox counts, process/table bytes, serialized queued bytes, and reservation/refusal/cancellation counts. Started and finished events share a worker ticket. Finished events distinguish confirmed_stop, failed_stop, success, error, uncertain, and worker_death. For confirmed_stop, queue plus service duration measures worker confirmation latency. The burst summary records success, error, crash, and timeout counts per public operation, and the fixture fails when any operation does not succeed. Queue age starts when the runtime first samples a queued request; reservation age includes producer and deferred waiting.

Keep reports under /tmp, with the revision, working diff, environment and command used. JSONL file writes add measurement overhead, and one burst does not establish production latency or justify scheduling changes. Producer byte budgets count captured heaps and backing binaries conservatively; serialized queued bytes measure a different representation. The runtime contract defines the bounds these measurements observe.

Record provider responses #

Direct record mode is part of the runner, not the CI report gate. It requires --runs 1 and --url. Recording contacts the selected upstream and produces artifacts that need review before sharing.

The intended container workflow is record_container.py, recorded as running on valefar. It requires Linux, Docker, the selected local image, the bonsai bridge setup, and its build tools. It prefers runsc when available and otherwise uses runc. It serializes access through /tmp/bonsai-gpu.lock and runs the workload without a container network, with a dedicated bridge to the provider.

nix develop -c python3 test/perf/record_container.py --out /tmp/new-perf-recordings

The output directory must not exist. --scenario can select recordings. The script's isolation probe checks selected network and host-file access before recording. Neither the host name nor a chosen container runtime alone proves isolation. Inspect the script's prerequisites before running it on another host.

Comparison #

A full replay derives metrics in run.py. compare.py applies ceilings and, when a baseline is supplied, regression tolerances from budgets.toml. Byte metrics use 1% of the baseline. Disk metrics use the larger of three times baseline noise and the configured floor. Other metrics also include 5% of the baseline in that maximum. A result over its ceiling fails even without a baseline.

The allocator peak from albedo_perf_probe sums each allocator instance's high-water mark. Those maxima need not occur together, so this is an upper bound on simultaneous allocated memory. Its comparison floor accounts for variation between repeated batches on unchanged production code. Keep the calibration runs and environment beside its entry in budgets.toml.

Output safety #

safety.py scans output for supplied environment secrets and recognizable credential patterns before writing JSON. This is a guard, not proof that a recording contains no sensitive data. Recording contacts the supplied upstream and can fail. Inspect the generated artifact before sharing it.

See performance reporting for CI baseline selection, publication, external deployment observations, and report interpretation. The report comparison picker does not replace the gate's per-metric baseline.