Performance reporting #
The performance workflow measures Albedo's memory, HTTP traffic, and disk writes using recorded bonsai responses. Performance checks defines the measurements and replay commands. This guide covers CI, publication, and interpretation.
Run the CI gate #
The workflow declares pushes and pull requests to main, plus manual runs.
ci.py runs the performance and report tests, builds the CLI and a
snapshot daemon, and replays each scenario with five fresh-home samples. It
invokes measurement once, without retrying a failed measurement. CI reporting
uses replay. The runner also supports recording.
The orchestration has separate outcomes:
- Tests, build, isolation probes, baseline lookup, or timeouts can fail before a completed result exists.
- A failed replay can leave a partial result. A failure before its first completed scenario can leave no result. CI does not publish results without gated metrics.
- A completed result is printed as JSON and the site is built even without publishing credentials. This keeps fork-PR logs useful when secrets are unavailable.
- Without
WISP_APP_PASSWORD, publication is skipped. Public baseline reads still require network access. - With credentials, CI publishes before returning the gate verdict. Measurement failure, a failed verdict, or publication failure makes the job fail. A green measurement can therefore have a failed job because publication failed.
The deployment setup uses WISP_APP_PASSWORD, an app password for okami.mom,
under repository Settings → Pipelines → Secrets. It authorizes record writes
and site deployment. Repository secret settings and external scheduler behavior
are deployment configuration, not properties proved by this code.
Isolate measurement from publication #
Model-controlled code runs as perf-agent through
sudo -n -u perf-agent env -i. isolation.py owns the
environment allowlist and creates a mode-0700 directory under /tmp with
private home and temporary directories.
Before starting measurement, Sandbox.open checks that the sudo invocation
works, that the UID can read test/perf/run.py, and that it cannot read
nonempty environ or mem data for the supplied CI process and parent PIDs.
Tests and the CLI build happen before those probes. The probes do not inspect
every CI process, every checkout file, or the complete sudo policy.
After copying the result, CI kills processes under the measurement UID, checks
that none remain, and removes its directory before starting publication. Cleanup
failure prevents publication. This ordering matters because
publish.py passes the password in wispctl's argv. The UID must
be dedicated to this runner's measurement work. Concurrent measurements sharing
it would interfere with cleanup.
The recorded infrastructure setup provisions the spindle guest's user and sudo
rule through ark ec75712. Retain that deployment reference when investigating
the host. The local probes check the current host's selected boundaries, not
that external revision or every possible process-visibility channel.
Select a baseline #
ci.baseline reads public records, unshallows the clone when needed,
excludes the candidate SHA, and selects main-push ancestors. It orders them by
commit instant, then record creation time.
records.baseline selects each metric independently. A composite
baseline can draw metrics from different commits. A metric is eligible when:
- Its verdict is
pass, with no comparison baseline or a value at least as high as that baseline. - Its value is lower than its comparison baseline and the next newer encountered
result also reports a lower value. This includes
passresults whose drop was inside the noise floor, as well asimprovedresults. The older of those consecutive lower results becomes eligible. accepted.tomlnames its commit and exact measured value, with an acceptance reason.
A single low measurement must not tighten the baseline and fail later normal measurements. The baseline tests cover that noise defense, per-metric selection, and exact-value acceptance.
An unaccepted regression does not become a baseline automatically. A later fix
can still pass. Acceptance changes baseline eligibility, not the absolute
ceilings in budgets.toml.
CI logs each selected metric's source and passes the composite to the runner.
Compared metrics retain base_commit. With no eligible ancestor baseline, only
budgets apply. Regressed metrics carry suggested accept entries, printed in CI
and shown on the run page. Review the measurement and add a reason before
accepting it.
Publish records and shared assets #
records.py writes pet.nekomimi.albedo.perf.run records in
okami.mom's ATProto repository. The record key starts with the commit SHA.
Reruns try suffixed keys such as <sha>-2. Record creation uses explicit keys
and relies on the PDS rejecting collisions.
summary stores numeric summaries as JSON text because ATProto has no
floating-point scalar type. result is a structured blob reference to the full
results JSON. committedAt orders commit history, while url identifies the
report page. A typical URL is
https://sites.wisp.place/okami.mom/albedo-perf/<sha7>/, with the rerun suffix
when applicable.
Publication uploads the result and claims the record before updating the site. A later deployment failure does not roll back the record. Inspect the record and deployment state before retrying publication. There is no need to repeat measurement just because publication failed.
The shared albedo-perf site is deployed with wispctl --spa. Its hosting
setup serves index.html for run paths, and the app resolves the route. Per-run
records do not require per-run assets or a growing deployment manifest.
publish.should_deploy permits changed assets only for push or
manual events on main. The publisher hashes served assets into version.json,
compares the hosted version, and waits for the new version after deployment. PRs
and other branches do not satisfy that predicate. This is a local deployment
rule, not a substitute for credential or hosting permissions.
The code has no deployment lock or commit-freshness check. Two eligible jobs can race, and an older job can deploy after a newer one unless external orchestration prevents it.
Earlier runs used separate albedo-perf-<sha7> sites with data/run.json. The
deployment history records those legacy sites as still present, with records
readable by the shared app. Before removing one, verify its current existence,
ownership, and replacement record. The removal command is:
wispctl site delete okami.mom --site albedo-perf-<sha7> --yes
That command deletes a hosted site. It is not part of normal report publication.
Find a report #
routes.js defines the routes and ambiguity handling:
/lists fetched runs newest first, 40 per displayed page, with main-push and failure markers and links to PDS records./<sha-prefix>/opens a run, including a rerun suffix when present. An ambiguous commit prefix shows choices./main/selects the newest main-push commit among fetched records.
The browser loads up to 20 record pages of 100 records each. All routes and
charts resolve against that fetched set. Records outside it can be absent even
when addressed by full SHA. Record keys are SHA-based, so omitted records are
not necessarily the oldest. /main/ and history do not guarantee complete
repository history. The truncation message marks this limit explicitly.
The app reads summaries for comparisons and fetches the selected run's result blob. Records of retired live runs stay in their own collection and are not read by this app.
Interpret a report #
app.js renders the gate verdict, metric tiles, scenario details,
comparisons, and history. Its metric definitions are canonical for which tiles
appear, including conditional kernel, forked-kernel, and disk metrics. Tiles use
decimal units, show budget bars, and expose exact values and noise in tooltips.
Sparklines appear once at least three fetched main runs are available. The
per-turn tile uses retained BEAM growth as its gate and process-memory growth as
a secondary observation. Scenario cards show retained memory, per-turn peaks,
and request bytes, with raw values available separately.
The recorded gate verdict uses the composite per-metric baseline. The comparison
picker instead selects another run's summary. Its default is a listed main
record matching the result's top-level baseCommit, when available. That record
can be a rerun with different values and cannot represent every component of a
composite baseline. Changing the picker changes displayed comparisons, not the
gate decision.
The change function controls displayed movement:
- A single-sample comparison needs at least a 5% move.
- With zero noise, any nonzero change counts.
- Otherwise the change must exceed three times the larger noise and reach at least 1%.
"What moved" uses that rule. Red arrows indicate a failed gate metric. The history chart uses fetched main records, metric tabs, a three-MAD noise band, and nearby budgets. It does not recover records omitted by the listing cap.
render.py pins Apache ECharts 6.1.0 from jsDelivr with an SRI
hash. The recorded provenance is the dist/echarts.min.js npm-tarball asset,
checked against the registry's SHA-512. That provenance records the original
verification, not a new check of current CDN bytes. Charts use canvas and the
built-in default and dark themes, following prefers-color-scheme. Without
JavaScript, the page explains that it is required.
Browser caching and deployment observations #
The app bypasses the browser HTTP cache for PDS fetches and uses its own Cache
API storage. DID resolution is cached for one day, record listings for one
minute, and content-addressed blobs until newer blobs displace them from the
12-entry cache. Those constants and cache-failure behavior live in
app.js.
The recorded deployment observation used reference PDS 0.5.38 behind Cloudflare.
From sites.wisp.place, both listRecords and getBlob returned
Access-Control-Allow-Origin: * and no rate-limit headers. The recorded
reference-PDS limit, when enabled, was 3,000 requests per five minutes per IP.
These are observations of that deployment, not guarantees made by the report
application.
The cache-bypass rationale comes from a Chrome observation: the PDS sent CORS
headers only with an Origin request and Vary: Origin, while Chrome reused an
answer cached for another origin and the fetch failed. Keep that rationale when
changing caching. It does not establish identical behavior across browser or PDS
versions.
With one record page, a cold run view needs three data requests: DID resolution, listing, and blob retrieval. Fresh working caches can reduce those to zero. These counts exclude HTML, assets, and CDN requests. More record pages, expired or unavailable caches, and an explicit PDS endpoint change the count.
Preview locally #
Run from the repository root:
python3 test/perf/report/render.py /tmp/albedo-perf-site --serve 8765
Open http://127.0.0.1:8765/. The local server uses the same SPA fallback. By
default, the page reads real public records. --did and --pds select another
repository or a local mock PDS. Local serving does not itself publish records or
deploy assets.
Publish an existing result #
With wispctl on PATH and WISP_APP_PASSWORD in the environment:
python3 test/perf/report/publish.py replay /path/to/replay-results.json
This writes a blob and record. Shared-site deployment depends on the event and ref embedded in the result. Inspect that metadata before running it. Incomplete results without gated metrics are refused.
To update only shared assets for the main deployment:
python3 test/perf/report/publish.py site --event push --ref refs/heads/main
Verify changes #
The report tests cover baseline selection, publication guards, safe generated content, and routing. The measurement contract tests cover metric comparisons and output-secret scanning. Use those owners when changing behavior. Do not add tests that assert this document's wording.