small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
README.md

Performance reporting #

The performance workflow measures Albedo's memory, HTTP traffic, and disk writes using recorded bonsai responses. Performance checks defines the measurements and replay commands. This guide covers CI, publication, and interpretation.

Run the CI gate #

The workflow declares pushes and pull requests to main, plus manual runs. ci.py runs the performance and report tests, builds the CLI and a snapshot daemon, and replays each scenario with five fresh-home samples. It invokes measurement once, without retrying a failed measurement. CI reporting uses replay. The runner also supports recording.

The orchestration has separate outcomes:

  • Tests, build, isolation probes, baseline lookup, or timeouts can fail before a completed result exists.
  • A failed replay can leave a partial result. A failure before its first completed scenario can leave no result. CI does not publish results without gated metrics.
  • A completed result is printed as JSON and the site is built even without publishing credentials. This keeps fork-PR logs useful when secrets are unavailable.
  • Without WISP_APP_PASSWORD, publication is skipped. Public baseline reads still require network access.
  • With credentials, CI publishes before returning the gate verdict. Measurement failure, a failed verdict, or publication failure makes the job fail. A green measurement can therefore have a failed job because publication failed.

The deployment setup uses WISP_APP_PASSWORD, an app password for okami.mom, under repository Settings → Pipelines → Secrets. It authorizes record writes and site deployment. Repository secret settings and external scheduler behavior are deployment configuration, not properties proved by this code.

Isolate measurement from publication #

Model-controlled code runs as perf-agent through sudo -n -u perf-agent env -i. isolation.py owns the environment allowlist and creates a mode-0700 directory under /tmp with private home and temporary directories.

Before starting measurement, Sandbox.open checks that the sudo invocation works, that the UID can read test/perf/run.py, and that it cannot read nonempty environ or mem data for the supplied CI process and parent PIDs. Tests and the CLI build happen before those probes. The probes do not inspect every CI process, every checkout file, or the complete sudo policy.

After copying the result, CI kills processes under the measurement UID, checks that none remain, and removes its directory before starting publication. Cleanup failure prevents publication. This ordering matters because publish.py passes the password in wispctl's argv. The UID must be dedicated to this runner's measurement work. Concurrent measurements sharing it would interfere with cleanup.

The recorded infrastructure setup provisions the spindle guest's user and sudo rule through ark ec75712. Retain that deployment reference when investigating the host. The local probes check the current host's selected boundaries, not that external revision or every possible process-visibility channel.

Select a baseline #

ci.baseline reads public records, unshallows the clone when needed, excludes the candidate SHA, and selects main-push ancestors. It orders them by commit instant, then record creation time.

records.baseline selects each metric independently. A composite baseline can draw metrics from different commits. A metric is eligible when:

  • Its verdict is pass, with no comparison baseline or a value at least as high as that baseline.
  • Its value is lower than its comparison baseline and the next newer encountered result also reports a lower value. This includes pass results whose drop was inside the noise floor, as well as improved results. The older of those consecutive lower results becomes eligible.
  • accepted.toml names its commit and exact measured value, with an acceptance reason.

A single low measurement must not tighten the baseline and fail later normal measurements. The baseline tests cover that noise defense, per-metric selection, and exact-value acceptance.

An unaccepted regression does not become a baseline automatically. A later fix can still pass. Acceptance changes baseline eligibility, not the absolute ceilings in budgets.toml.

CI logs each selected metric's source and passes the composite to the runner. Compared metrics retain base_commit. With no eligible ancestor baseline, only budgets apply. Regressed metrics carry suggested accept entries, printed in CI and shown on the run page. Review the measurement and add a reason before accepting it.

Publish records and shared assets #

records.py writes pet.nekomimi.albedo.perf.run records in okami.mom's ATProto repository. The record key starts with the commit SHA. Reruns try suffixed keys such as <sha>-2. Record creation uses explicit keys and relies on the PDS rejecting collisions.

summary stores numeric summaries as JSON text because ATProto has no floating-point scalar type. result is a structured blob reference to the full results JSON. committedAt orders commit history, while url identifies the report page. A typical URL is https://sites.wisp.place/okami.mom/albedo-perf/<sha7>/, with the rerun suffix when applicable.

Publication uploads the result and claims the record before updating the site. A later deployment failure does not roll back the record. Inspect the record and deployment state before retrying publication. There is no need to repeat measurement just because publication failed.

The shared albedo-perf site is deployed with wispctl --spa. Its hosting setup serves index.html for run paths, and the app resolves the route. Per-run records do not require per-run assets or a growing deployment manifest.

publish.should_deploy permits changed assets only for push or manual events on main. The publisher hashes served assets into version.json, compares the hosted version, and waits for the new version after deployment. PRs and other branches do not satisfy that predicate. This is a local deployment rule, not a substitute for credential or hosting permissions.

The code has no deployment lock or commit-freshness check. Two eligible jobs can race, and an older job can deploy after a newer one unless external orchestration prevents it.

Earlier runs used separate albedo-perf-<sha7> sites with data/run.json. The deployment history records those legacy sites as still present, with records readable by the shared app. Before removing one, verify its current existence, ownership, and replacement record. The removal command is:

wispctl site delete okami.mom --site albedo-perf-<sha7> --yes

That command deletes a hosted site. It is not part of normal report publication.

Find a report #

routes.js defines the routes and ambiguity handling:

  • / lists fetched runs newest first, 40 per displayed page, with main-push and failure markers and links to PDS records.
  • /<sha-prefix>/ opens a run, including a rerun suffix when present. An ambiguous commit prefix shows choices.
  • /main/ selects the newest main-push commit among fetched records.

The browser loads up to 20 record pages of 100 records each. All routes and charts resolve against that fetched set. Records outside it can be absent even when addressed by full SHA. Record keys are SHA-based, so omitted records are not necessarily the oldest. /main/ and history do not guarantee complete repository history. The truncation message marks this limit explicitly.

The app reads summaries for comparisons and fetches the selected run's result blob. Records of retired live runs stay in their own collection and are not read by this app.

Interpret a report #

app.js renders the gate verdict, metric tiles, scenario details, comparisons, and history. Its metric definitions are canonical for which tiles appear, including conditional kernel, forked-kernel, and disk metrics. Tiles use decimal units, show budget bars, and expose exact values and noise in tooltips. Sparklines appear once at least three fetched main runs are available. The per-turn tile uses retained BEAM growth as its gate and process-memory growth as a secondary observation. Scenario cards show retained memory, per-turn peaks, and request bytes, with raw values available separately.

The recorded gate verdict uses the composite per-metric baseline. The comparison picker instead selects another run's summary. Its default is a listed main record matching the result's top-level baseCommit, when available. That record can be a rerun with different values and cannot represent every component of a composite baseline. Changing the picker changes displayed comparisons, not the gate decision.

The change function controls displayed movement:

  • A single-sample comparison needs at least a 5% move.
  • With zero noise, any nonzero change counts.
  • Otherwise the change must exceed three times the larger noise and reach at least 1%.

"What moved" uses that rule. Red arrows indicate a failed gate metric. The history chart uses fetched main records, metric tabs, a three-MAD noise band, and nearby budgets. It does not recover records omitted by the listing cap.

render.py pins Apache ECharts 6.1.0 from jsDelivr with an SRI hash. The recorded provenance is the dist/echarts.min.js npm-tarball asset, checked against the registry's SHA-512. That provenance records the original verification, not a new check of current CDN bytes. Charts use canvas and the built-in default and dark themes, following prefers-color-scheme. Without JavaScript, the page explains that it is required.

Browser caching and deployment observations #

The app bypasses the browser HTTP cache for PDS fetches and uses its own Cache API storage. DID resolution is cached for one day, record listings for one minute, and content-addressed blobs until newer blobs displace them from the 12-entry cache. Those constants and cache-failure behavior live in app.js.

The recorded deployment observation used reference PDS 0.5.38 behind Cloudflare. From sites.wisp.place, both listRecords and getBlob returned Access-Control-Allow-Origin: * and no rate-limit headers. The recorded reference-PDS limit, when enabled, was 3,000 requests per five minutes per IP. These are observations of that deployment, not guarantees made by the report application.

The cache-bypass rationale comes from a Chrome observation: the PDS sent CORS headers only with an Origin request and Vary: Origin, while Chrome reused an answer cached for another origin and the fetch failed. Keep that rationale when changing caching. It does not establish identical behavior across browser or PDS versions.

With one record page, a cold run view needs three data requests: DID resolution, listing, and blob retrieval. Fresh working caches can reduce those to zero. These counts exclude HTML, assets, and CDN requests. More record pages, expired or unavailable caches, and an explicit PDS endpoint change the count.

Preview locally #

Run from the repository root:

python3 test/perf/report/render.py /tmp/albedo-perf-site --serve 8765

Open http://127.0.0.1:8765/. The local server uses the same SPA fallback. By default, the page reads real public records. --did and --pds select another repository or a local mock PDS. Local serving does not itself publish records or deploy assets.

Publish an existing result #

With wispctl on PATH and WISP_APP_PASSWORD in the environment:

python3 test/perf/report/publish.py replay /path/to/replay-results.json

This writes a blob and record. Shared-site deployment depends on the event and ref embedded in the result. Inspect that metadata before running it. Incomplete results without gated metrics are refused.

To update only shared assets for the main deployment:

python3 test/perf/report/publish.py site --event push --ref refs/heads/main

Verify changes #

The report tests cover baseline selection, publication guards, safe generated content, and routing. The measurement contract tests cover metric comparisons and output-secret scanning. Use those owners when changing behavior. Do not add tests that assert this document's wording.