small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
albedo robot-docs cache-warming.md
3.5 kB · 69 lines
Markdown
at main

Cache warming #

The warm managed extension repeats a session's last sent provider request while the idle session waits for work. It uses a reduced output budget and a background call. This spends provider usage and does not guarantee a cache hit. warm/extension.gleam owns the worker and its scheduling decisions.

Eligibility and settings #

wanted asks Session.awaited() whether work that can wake the session is running. The captured call also needs a usable TTL and enough cached input. The count is cached input tokens plus cache creation tokens. Missing usage counts as zero. warm.minCachedTokens defaults to 1024 and is read when planning a ping. See warm/extension.gleam and the extension architecture.

A request with budgeted thinking cannot be warmed: Anthropic keeps the thinking budget below max_tokens, so a one-token ping cannot carry it, and dropping the thinking would change the cached prefix. The anthropic-api provider refuses such a ping before sending it. The refusal ends warming for that idle stretch and shows as one failed background request. Adaptive thinking has no budget and warms like any other request.

Explicit cache marks supply their shortest TTL. Without marks, the first clock tier of a matching Refresh or Fixed entry supplies it. See the TTL table for matching and layer precedence.

Scheduling and stopping #

A successful CallSent replaces the stored call and resets the ping count. An uncancelled TurnEnded schedules a ping if the plan and awaited work permit it. Stirred invalidates older scheduled ticks, and Compacted clears the call. A restored call can schedule a ping when the worker has no call already. step and due implement these transitions.

For TTLs over ten seconds, the interval is the smaller of 90% of the TTL and the TTL minus ten seconds. Shorter TTLs use 90%. The clock starts at request start or response finish according to the table entry. A tick after latest_ms is discarded. The cap is max(0, floor(write / read) - 1) pings, using default multipliers of 1.0 for writes and 0.1 for reads when necessary. A failed call or a ping without cached input after a cached original call ends the sequence. These are estimates based on table inputs, not a billing guarantee. See plan, resolution, interval_ms, cap, and cache_lost in the warmer source.

Background requests and restoration #

ping copies the captured request, sets patient to false, and sets max_output_tokens to 1 for Chat Completions or 16 for Responses. It sends the original prefix identity through Session.call. A refused or failed call does not schedule another ping. See warm/extension.gleam and provider requests.

The worker keeps its captured call in memory. session_last_call.gleam rebuilds a candidate after restart and checks its identity before restoration. A changed request can therefore prevent restoration rather than warming a different prefix.