the detached kernel #

a session's python kernel (priv/python/albedo_kernel.py, which runs albedo_cells.py as __main__ so its bytecode comes from the cache) is its own process, detached from the daemon: it outlives a dropped connection and a daemon restart, keeping its namespace and its background jobs. the daemon connects to it directly, or through a bridge when it runs on another host, and every message between the two travels through a small session layer that neither loses nor repeats a message across a reconnect.

frame vocabulary #

The daemon sends execute, reply, interrupt, shutdown, snapshot, restore, job_stop, job_list and job_read. The kernel sends ready, startup_error, call, done, job_start, job, trace, cleanup, cells and jobs; job stop and job view replies repeat the request type. The remote plugin also sends invoke, introspect and release, and consumes invoked, introspected and mirror.

test/harness/kernel_frames_test.py compares these names with the protocol TypedDicts and source literals without booting a kernel. Its explicit baseline records the missing TypedDicts for the cells and jobs observations. It checks names, not payload fields or delivery behavior; dynamically selected names need an extractor update.

host calls #

albedo_python.erl decodes the Python call frame once and passes its method and native arguments to host.Route. Routes return Result(Json, #(code, message)); rpc.handle dispatches by namespace and rpc.reply encodes the answer once. The port owner embeds that answer in the reply frame, encoding the call id separately. Host failure and restart replies use the same rpc.reply shape. A frame without a string method or an args field is refused with invalid / invalid host request; argument validation stays in each route. The Python JSON boundary is unchanged.

tool bindings #

Cells also see the common standard-library modules (asyncio base64 collections datetime functools hashlib itertools json math os re shutil sys textwrap time, and Path) without importing them; these live in the same session-private builtins, so they are not saved as user variables. import albedo and from albedo import files answer the injected bindings (albedo_cells.Bindings).

A cell may not block the kernel with time.sleep of a second or more: albedo_shell refuses it through the same audit hook that refuses raw subprocess calls, and says to start the work with run() and do other useful work while it runs, and when nothing else is left to give the user a status report first, then wait with await asyncio.sleep(n). Job timeouts are seconds, at most a day (albedo_api.check_timeout): a job outlives the cell that started it, so it is not held to the cell's hour.

The kernel keeps injected helpers in cell globals and in a session-private copy of Python's builtins. An assignment such as files = [...] shadows the helper; del files reveals the original binding again, including inside functions and later cells. The process-wide builtins module is unchanged. Plugin instances are owned by this kernel's API, not module-level singletons. This fallback is built once after plugins load and is not serialized with user state.

images #

show_image(source, *sources) returns PNG, JPEG, or WebP bytes or paths with the cell result. albedo_capture.Capture owns the four-image and 5 MiB attachment budgets. Overflow is saved losslessly under the session's output directory (or in a temporary directory without one); the returned Text names each file and says to use show_image(path) in another cell. Output-directory files share its 14-day retention. An image over 5 MiB cannot fit even in an empty cell: it raises with its size and advice to downscale or crop, without reading a large file path into memory. Invalid, cut-short, or over-edge attachments still raise. No image decoder is required, so overflow is not tiled.

browser gestures #

priv/python/albedo_plugins/browser/page.py owns page actions and their lock. Use await page.click((x, y)) for viewport coordinates and await page.drag(start, end, steps=20) for left-button drags. Endpoints may be observed nodes, references, or viewport (x, y) points; node endpoints must both fit in the viewport. Drag steps are 1–1000, and mouse release runs even if a move fails. Document-level mouse/pointer listeners do not need capture. Use await page.click_text(text) to click a unique button or link whose accessible name contains the case-sensitive text. Zero or multiple matches raise SelectionError rather than choosing one.

await page.evaluate(source) accepts expressions, zero-argument functions, and statement bodies such as localStorage.clear(); return 1. It selects the syntax before running the source, so a runtime exception never causes a retry. Promises are awaited. This escape hatch still bypasses action locking.

await page.screenshot() returns the PNG's path. Given path= it writes there, as the cell's own change. Without one it files a uniquely named PNG under $ALBEDO_HOME/cache/browser-screenshots/ (browser/paths.py), never the working tree, outside the cell's file-change trace, and first removes PNGs there older than seven days.

spawned browsers #

priv/python/albedo_plugins/browser/launch.py owns browser discovery and spawned processes. browser.spawn() uses a temporary profile and headless mode by default; await b.close() stops the owned process and removes that profile. Connecting to an external browser never takes ownership of its process.

A spawned Chrome never outlives its kernel. A normal exit closes it from atexit; for a kernel that dies any other way (SIGKILL, a crash) each spawn starts browser/watchdog.py in its own session, reading a pipe only the kernel holds. close() writes a byte to it first and it exits; end of input without that byte ends Chrome's process group (albedo_proc.terminate) and removes its lock and temporary profile. It also exits once Chrome ends on its own.

Automatic headless launches prefer chrome-headless-shell on PATH, then $ALBEDO_HOME/browsers/chrome-headless-shell (~/.albedo by default), before regular Chrome/Chromium. The Nix dev shell and packaged daemon put the shell on PATH, using nixpkgs' pinned Chrome for Testing binary without the Playwright runtime. Full Chrome's headless launches on macOS can leave duplicate recent-app Dock tiles even after they exit. Outside Nix, install the official Chrome for Testing shell with its support files and symlink its executable at that home path; discovery resolves the symlink so the shell can find its resources. No browser is downloaded by spawn().

executable=, ALBEDO_BROWSER_EXECUTABLE, PRIME_BROWSER_EXECUTABLE, and CHROME_PATH still override discovery. headless=False skips automatic shell selection. An absent shell falls back to Chrome/Chromium; a selected browser's startup failure is reported, not retried with another executable. Dock settings and the user's normal Chrome profile are never changed.

background job observations #

GET /sessions/{id} exposes kernel.live_job_count and kernel.running_jobs without starting a kernel. Unknown observations are null. The list contains up to 100 live jobs ordered by ID, with id, nullable pid, command (up to 4096 scalars), and started_at: wall-clock milliseconds when the port owner recorded the job's group, zero when unknown (a restored link older than the field). Jobs the kernel's remote connections run (remote.connect(...).run) are listed beside the local ones, with no pid (they own no group on this machine) and the wall-clock time the connection first saw them. The remote plugin reports them as {"type": "jobs", "live": n, "awaited": m, "items": [{id, command, service, started}]}, m the ones not started as services; the port owner keeps the latest frame (and a reattach's hello carries it as external_jobs). A kernel from before items reports counts only, and the count then exceeds the list by the awaited ones it could not name. kernel.awaited_jobs is the one definition of which listed jobs will wake the session (not service); awaits_jobs (the cache warmer, resumed kernels) and the session's still-running summary both ask it, so an extension never reads service itself. Stopping a remote job through /jobs sends a job_stop frame down the bridge to the local kernel ({"type": "job_stop", "id": "<job id>"}); if the kernel is unreachable (port := none), stopping fails immediately without queueing. The local kernel forwards the stop to the remote kernel (or, for degraded fallback ssh jobs, stops the job by killing the local ssh client). The port owner waits up to 15 seconds for a confirmation frame ({"type": "job_stop", "id": "<job id>", "ok": true|false}); if unconfirmed after 15 seconds, the stop fails. The TUI uses these observations for its idle-job status, animation, and sidebar, and calls a job background only once it has run three seconds (jobGrace), so work a cell starts and awaits never flashes through the chrome; /jobs lists every live job regardless of age.

A watching session does not poll for them. It registers a closure with albedo_python:watch/2 (the kernel's pid in the albedo_kernel_watch registry) and the port owner runs it after any frame that changes what an observation reply would carry, comparing a cheap facts tuple so a frame that changed nothing wakes nobody. The session then re-observes and publishes a kernel stream event only when the facts differ (sessions). A dropped kernel handle is forgotten with albedo_python:unwatch/1. The Erlang port owner truncates commands through text_scalars.take, which scans only the retained prefix and copies it. Do not convert the entire command to a codepoint list to take a bounded preview; see runtime.

/jobs reads GET /extensions/run/sessions/{id}/jobs, whose page declares the stop action. POST /extensions/run/sessions/{id}/jobs/{job_id}/stop supervises that job's recorded process group and reports failure if its stop is unconfirmed. A missing job returns 404. After response loss, read the collection before confirming another attempt; stopping is never automatically retried. Job rows carry no resource; the stop action binds the row id.

job book and output #

GET /extensions/run/sessions/{id}/jobs/recent lists what the kernel's run plugin still knows of the jobs it started, newest first, up to 100: id, command, state (running or exited), exit_code, timed_out, service, started_at and ended_at (wall-clock milliseconds), and bytes written. The kernel keeps the 64 newest finished jobs with their output, so a finished job stays listed that long (and only while its kernel lives); a plugin's own internal jobs are not listed. A session with no kernel has none, and neither does one running another python bundle or protocol than the daemon (stale bundle or protocol, until its session is idle and it is swapped): a kernel that old ends its link on a frame it does not know, so the daemon never sends it job_list or job_read; the list is empty and a read is 503. This is not the live observation above: that one comes from the port owner and covers remote jobs, this one from the kernel and covers finished ones.

GET /extensions/run/sessions/{id}/jobs/{job_id}/output?offset=&limit= reads a page of one job's output: {job, offset, next_offset, text}. Offsets are bytes and pages are cut on characters, so a viewer that continues from next_offset loses none of what a job wrote, even where a pipe read split a character; a running job's half-written last character waits for the next page. limit is 4 to 65,536 bytes (default 16,384). Without offset the page is the end of the output, which is where a viewer opens on a long-running job. A page inside output the kernel did not keep (past the 1 MiB it holds in memory and the 16 MiB it spills) is 409 output_lost; the end of the output is always readable. An unknown job is 404 job_not_found.

Both are asked of the kernel as job_list and job_read frames (the request carries a fresh id and the reply repeats it, ok with the fields or code and error), answered by the handler each plugin registers with api.on_job_view; the run plugin's reads bypass Job.read, so watching a job never withdraws the wake its result is owed. The port owner waits at most 10 s (ASK_TIMEOUT) and fails the request, and does not monitor the caller. Each frame goes through the outbox like any other, so a viewer polling every second costs one write and one delete a second while it is open.

A job that finishes with its result unread wakes the session. The model reads the full notice with the job's handle; the transcript keeps the wake's one-line display (job finished (exit_code=0, ran 1.2s): …) as a note from job.

retained output #

Every cell and job owns one output channel (albedo_capture.Capture), kept for the 16 newest cells and 64 newest jobs plus native. A channel holds the first 1 MiB as written and, once more than that has arrived, the last 64 KiB in a separate tail buffer that exists only from then on; seen counts everything. Job and remote output enter as bytes (write_bytes), never decoded and re-encoded, so a UTF-8 character split across two pipe reads survives and a late job.pipe() replays the exact bytes while seen still fits in the retained start. Text readers (output.read, tail(), previews) decode a window on demand. output.read returns at most 64 KiB per call and raises for a larger limit rather than shortening it, since a caller that asked for everything and got a silent prefix builds files from it; the rest is paged with offset. A single print longer than what could be retained is not encoded whole when it is ASCII: the buffers see both ends and the middle only counts (unless a spill file takes it, below).

job output #

priv/python/albedo_output.py owns JobOutput, shared by local jobs and SSH command-mode jobs. head(n) and tail(n) are previews: up to 64 KiB, with lines= taking precedence. Zero returns empty. An out-of-range request or a character/line request the retained window cannot satisfy raises with job.read() / job.save(path) as the recovery, never silently clamps. A line cut at a window boundary is not counted as a complete line.

After completion, job.read() returns complete UTF-8 text up to 1 MiB; larger whole-text requests raise with the pagination call to bound allocations and remote result frames. job.read(offset=..., limit=...) reads byte-offset pages up to 64 KiB, including while running, and transparently reads spill files. output.read(id, offset=..., limit=...) uses the same disk-backed paging for jobs and cells. Non-UTF-8 bytes are ignored at page boundaries, as in existing previews; whole text reads decode with replacement.

job.save(path) streams the exact bytes into a temporary file in 64 KiB chunks, atomically replaces the destination, and returns its absolute path. It never loads the whole spill file into memory. Running jobs, output lost past the disk cap, or unavailable spill files refuse rather than saving partial output; failed saves leave the destination untouched and remove the temporary file. Pages crossing lost output also refuse instead of returning a short page or false EOF. The error names the retained prefix and its pagination call. For artifacts beyond the disk cap, attach .pipe("tee", path) before awaiting; a late pipe cannot recover dropped bytes.

remote.connect() mirrors the same 64 KiB tail window, coalesced at the existing interval. Larger reads and saves use the ordinary remote method transport. Remote spill files and saves live on the remote host; degraded SSH command-mode jobs spill and save locally, where their capture lives.

spilled output #

albedo_capture.Capture owns spilling and byte reads. Cells spill past the 64 KiB preview, jobs past the 1 MiB in-memory prefix. Both retain up to SPILL_LIMIT (16 MiB) on disk. The first overflowing write opens a private spill file with all preceding bytes; subsequent writes append without buffering, so pages can read live output. Job completion closes the writer. Memory still holds only the first 1 MiB and rolling 64 KiB tail, irrespective of disk output size.

Cell results name their spill file, seen, and whether all output is retained (Capture.spill_note). The directory sits beside the run directories: $ALBEDO_HOME/output locally, ~/.albedo-remote/output remotely, and albedo-output in the temp dir for a kernel over stdio. Spill files survive output channel eviction and are pruned after SPILL_KEEP (14 days) on kernel boot. A spill creation or write failure does not interrupt the command: in-memory previews remain available, but pages beyond the retained prefix and whole-output saves refuse. Missing or shortened spill files raise visibly on reads and saves.

memory cap #

The kernel's physical footprint (albedo_memory.footprint: the resident set from /proc/self/statm, or Darwin's ri_phys_footprint through libproc, the figure vmmap and Activity Monitor show) is capped at ALBEDO_KERNEL_MEMORY_BYTES, 256 MiB by default, which the daemon passes through to local kernels. A thread samples it every 100 ms while cells run and interrupts, with reason memory, every running cell whose threshold the footprint passed; the cell ends interrupted with a note naming what is held and the cap, and the namespace stays. A cell's threshold is the cap, or 8 MiB over the footprint it started at when the kernel was already past the cap (a large variable left behind), so del big; gc.collect() runs instead of being interrupted at once. It is a sampled guard, not an allocator limit: one allocation larger than the cap succeeds and is caught by the next sample, and a cell that catches KeyboardInterrupt is told once. Jobs are separate processes and are not counted.

Finished results are kept by cell id (the newest 16) for execute replays. Their base64 image text is capped at 8 MiB across all of them, newest first; an older result past the budget replays without its images and says so in its output.

background cells #

The kernel returns a python tool result with status=backgrounded after 60 seconds (ALBEDO_CELL_BACKGROUND_SECONDS overrides the threshold, in seconds). The result includes output so far, the cell id, and instructions to do useful work or report status and end the turn, without polling or sleeping. This is not a final outcome: the journal stays started until cells.finish records it. cells.info/read/trace remain available; traces appear when execution finishes. The original timeout_ms is carried to the kernel and still interrupts the cell at its deadline, even after the daemon's original execute wait has ended.

New cells can run while a detached cell awaits. They share the namespace, so avoid modifying variables the detached cell uses. Python output, images, and activity traces use the task's context; native fd output goes to native when multiple cells run, since it cannot be attributed safely. A synchronous cell can return its tool result early, but other cells must wait until it yields or finishes. Background and deadline timers run outside the asyncio loop so a synchronous cell cannot prevent its deadline from firing.

At most 16 cells run concurrently. Running captures are protected from eviction; completed output rolls out under the existing 16-capture limit. await cells.cancel(id) requests cancellation and suppresses that cell's wake. Explicit kernel reset/replacement ends its cells; a daemon restart merely reattaches. Internal cells count frames, replayed in the hello, keep idle reaping and automatic stale-kernel replacement from ending a running cell without pretending the cells are process jobs in the public API.

Completion is committed before cells.completed sends a wake. The transcript gets a one-line cell finished (status=ok, ran 90.0s): <first code line> from cell, while the model gets the full notice with the cell id separately. Unread completions share a wake and retry every two seconds while the session is busy; there is no model polling. A wake that parks while the kernel is being replaced starts its turn on the replacement, together with the upgrade notice. The final expression is appended to retained output under [result], so output.read(id) can read it after a background completion. Reading finished output, source, status, or trace retires a pending wake. Reading partial output while the cell still runs does not retire its eventual completion. A read racing with the completion journal also retires the wake.

processes #

• kernel: started in its own session and process group, stdio not tied to anyone. it listens on kernel.sock in its run directory, $ALBEDO_HOME/run/<kernel id>/ (0700; the socket is bound relative to the directory so a long temp home stays under the ~104-byte sun_path limit). kernel.log there catches what it writes before its own output capture starts.
• relay (in albedo_python.erl): how the port owner reaches a local kernel. an erlang process owns a gen_tcp connection to kernel.sock ({packet, 4}) and passes frames to the port owner as a bridge port would, so no python process stays beside a local kernel. a fresh boot first runs albedo_bridge.py launch <run_dir> <modules-json> (token as its one stdin line), which starts the kernel and exits once the socket answers. the relay announces nothing, so the port owner hashes the bundle itself, the same way albedo_bundle.digest() does. a refused or missing socket reads as exit 3. a run directory whose socket path does not fit sun_path (104 bytes) keeps the bridge.
• bridge (priv/python/albedo_bridge.py): what the daemon's erlang port runs for a remote kernel (the same argv over ssh, see remote kernels below) or a run directory too deep for the relay, with the same {packet, 4} stdio framing the kernel used to have. start <run_dir> <modules-json> starts the kernel beside it (the token goes over the kernel's stdin, never argv) and waits for its socket; attach <run_dir> connects to a running one. it announces its own bundle hash ({"bridge": {"bundle"}}), then copies bytes both ways until either side closes. exit status 3 means no kernel is there. killing it never touches the kernel.
• port owner (albedo_python.erl): one erlang process per kernel, as before. it owns the job groups the kernel reports and the albedo_signal.py termination ladder, aimed at the pid/pgid/leader the kernel declares in its hello.

a kernel run with plain albedo_kernel.py <modules> (no --run) keeps the old raw stdio protocol and dies with its pipe: the remote plugin's targets use that.

handshake #

the daemon writes {"attach": {"kernel", "token", "ack", "grace"}} first. the kernel checks the token (kept in the daemon's sqlite), bumps its epoch, cuts off any older connection (newest attach wins), and answers {"hello": {protocol, bundle, epoch, ack, pid, pgid, leader, ready, jobs, external, awaited_external, external_jobs, dropped}}, or {"refused": reason}. protocol (1) and the hello reports the running cells count. bundle is albedo_bundle.digest(), the same content hash the remote plugin stages under; a hello whose bundle or protocol differs from the bridge's (or, for a local kernel, the port owner's own hash) makes the kernel stale (see version skew). jobs (live job_start frames) and external let a fresh port owner take over job ownership and live-job observations.

session layer #

after the handshake every frame is {"seq": n, "ack": m, "frame": {...}}, or a bare {"ack": m}. ack is cumulative: everything up to it arrived.

• kernel side (priv/python/albedo_link.py): an in-memory outbox of unacknowledged frames, resent after each hello. mirror (per handle), trace (per cell) and jobs coalesce latest-wins; everything else must be delivered. bounded at 4096 frames / 64 MiB: coalescing frames go first, then the oldest, counted in the next hello's dropped. a daemon frame at or below the last seq applied is a replay and is dropped; each applied frame is acknowledged at once.
• daemon side (python/link.gleam, tables from the python extension's kernel_links schema migration): kernel_links (one row per session: kernel id, token, run dir, workspace, modules, pid/pgid/leader, epoch, last seq sent, the job groups it owns, rewritten as they change, and the $ALBEDO_HOME of the daemon that booted it), kernel_outbox keyed (session, kernel, seq), bounded at 2048 frames / 64 MiB per kernel, and kernel_calls. the port owner persists each frame before writing it, deletes what the kernel acknowledges, and acknowledges kernel frames within 100 ms. its own last-seen seq lives in memory: after a daemon restart it accepts everything the kernel resends, and the dedupe rules below make that safe.

dedupe:

• host call id: a call already running in this daemon is ignored (its reply goes out through the outbox); one answered already is ignored the same way; one an earlier daemon started without answering is answered {ok: false, code: "unknown"}, never run again. a call's row goes once the kernel acknowledges its reply.
• execute, by cell id: a finished cell (the newest 16) resends its done; a queued or running one is ignored. callers must not reuse cell ids.
• interrupt, reply, release are idempotent already. an interrupt for a cell still queued behind the loop is kept and cancels the cell as it starts; one for a cell the kernel no longer holds is dropped.
• job_stop: the port owner asks the local kernel to stop a remote background job with {"type": "job_stop", "id": "<job id>"}. The kernel answers with {"type": "job_stop", "id": "<job id>", "ok": true} or {"type": "job_stop", "id": "<job id>", "ok": false, "error": "<lowercase why>"} exactly once per request. The port owner does not queue while the kernel is unreachable (port := none), and gives up with an unconfirmed error after 15 s. The request is delivered like any other frame, so one sent while the kernel was briefly out of reach can still land after a reattach and end the job later: "unconfirmed" means the outcome is unknown, not that the stop failed.
• job_list, job_read: the port owner asks the kernel for its job book or one page of a job's output (see job book and output). The kernel answers each with the same type and id; a reply after the owner's deadline is dropped.
• what the kernel resends after a restart: job wakes are host calls (jobs.completed), so the call ledger covers them; job_start, job, and jobs only update ownership and observations, which converge; a done nobody waits for rewrites the same outcome; traces save by cell id.

the runtime scheduler #

harness/runtime/boot.gleam owns every worker the runtime actor starts. Booting, attaching, swapping a stale kernel at open, an explicit upgrade, recomposing for a reload (proposed or desired), and a composition read all queue as a BootRequest and run through boot_next, which admits them into boot_slots workers, one class never starving the other. A handler that queues a request ends in |> boot.boot_next in serve; nothing spawns a kernel worker outside start. Each worker computes off the actor, then delivers its message or, when the runtime stopped meanwhile, releases what it made (work). runtime/state names the generation lifecycle a request belongs to: admit opens one, current_waiters says whether a result is still for it, generation_over ends it and replays what waited.

drops #

• the connection drops (the relay's socket closes, or the bridge dies): the port owner reattaches with backoff (50 ms doubling to 2 s, 40 tries); exit 3 or a refusal means the kernel is gone and the old abandon path reaps it. a port owner attaching after a restart starts from the record's pid/pgid/leader and groups, so a kernel that died hard while nobody was attached still has its process group and its jobs' groups ended. the leader token (start time, from /proc or Darwin's sysctl) keeps a reused pid from being signalled. the cell keeps running meanwhile.
• the cell deadline passes, or an interrupt goes unanswered for 2 s, while detached: the interrupt waits in the outbox and the caller gets kernel.Detached, which leaves the journaled cell started (the model is told to check cells.info). a done that arrives for a cell nobody waits for any more is journaled through the cells.finish host route.
• the daemon crashes: its connection closes (a bridge sees EOF and exits); the kernel waits.
• graceful shutdown: runtime.detach_kernels first, so closing sessions and stopping the runtime let kernels go instead of ending them.
• daemon start: first link.disown forgets, without touching them, the records another daemon wrote: a row naming another home, or an older row without one whose local run directory is outside this home (an older remote row can't be told apart and stays). a daemon booted on a copy of albedo.sqlite would otherwise attach to the original's kernels with the copied token, replay their outboxes into them, and stop their process groups when it swapped them.
• daemon start: runtime.resume_kernels reattaches every recorded kernel in the background, one at a time through the boot slots, so late results and job wakes arrive without waiting for the session. no actor is loaded for an idle session yet, so a wake finds no registered closure: albedo_wakes then calls the loader the registry installs (on_missing), which starts the session as a Lookup would, and delivers to it; only a session that cannot start answers unavailable, which the kernel does not retry. a session whose kernel came back running a job that is not a service is started at once, idle, so its cache warming resumes (robot-docs/cache-warming.md). a session that asks for its kernel meanwhile waits for that attach, and gets a fresh boot if there was nothing to attach to. an attached kernel's runtime.origin is Resumed, and the session keeps its namespace without the disk restore. a recorded kernel whose module set no longer fits is kept and marked stale; one in another workspace is stopped and replaced. when jobs or background cells are still running, the session's first turn on a Resumed kernel carries a model note that the kernel reattached with its namespace, naming those jobs (id and command) and cells (id and first line), so a live handle is not mistaken for one the restart killed. a session with history is told even when nothing runs ("Nothing was running"), so it need not probe its variables. an origin is handed out once: later opens of the same kernel get Kept, so no notice repeats and no live cell is journaled as ended. the note is built from the port owner's observation and the cells journal, never from a new kernel frame, so a reattached kernel on an older bundle still gets it.

a recorded kernel found gone (killed while the daemon was down, or dead when its session asks) boots a Fresh replacement. the dead record's job groups carry id and command, so link.lose keeps the awaited ones in kernel_lost before the record is forgotten, and open reads them before the replacement's record clears them (Fresh(lost_jobs)). this is needed because a starting daemon reattaches in the background and finds the kernel dead before its session asks. adopting a Fresh kernel in a session with history ends the journal's unfinished cells as interrupted (nothing of the old kernel runs) and the model note names them with the lost jobs: "Jobs stopped by the restart: id (command). Cells ended by the restart: id (first line).", after the restored-variables or reset sentence. a reset with nothing to name keeps the bare reset notice. deleting a session, and /cd which deletes its namespace, clear kernel_lost; the model's own reset names nothing.

explicit drops still end the kernel at once: reset, release by the idle sweep, /cd, session close, extension reloads (the replacement boots beside the live kernel, never attached to it), and the old kernel after a swap.

version skew #

a kernel is stale when it runs another bundle than the bridge that reached it (for a local kernel, the bundle the port owner hashed), speaks another protocol (both read from the hello), or booted with another module set than its session now has (found at reattach). kernel.stale answers the reason and whether the swap was forced. GET /sessions/{id} returns the native kernel observation with instance_id, build, state, stage, stale, and live_job_count. Unknown identities and counts are null. Reading this resource does not prepare or replace a kernel. stage reports remote bundle staging while the owner is booting.

At the next idle moment, session_namespace.ready lets the runtime replace a stale kernel when no live jobs keep it. The runtime does the work off its actor:

1. Snapshot the old namespace to namespace.state in its run directory.
2. Boot a replacement beside it. The replacement's identities, groups and outbox belong to kernel_stages; the old kernel_links row stays authoritative.
3. Check that the replacement is attached and current, then restore its namespace.
4. Mark the staged replacement ready, stop the old kernel and its jobs, and confirm that the old durable ownership record was removed.
5. Publish the replacement in kernel_links and adopt its actual handle.

Boot, validation or restore failure stops the staged candidate and keeps the old kernel. A failed old-kernel stop reports the actual failure and attempts to stop the staged candidate. Shutdown may already have affected jobs; the report cannot promise to restore those external effects. Failed cleanup retains the relevant durable identities for a later verified stop. No failed stop is reported as a successful swap.

On restart, storage publishes a ready candidate if its old link is absent. The daemon then supervises abandoned staged candidates before admitting work. Staged owners use their immutable kernel ID throughout publication, so replies and ownership updates continue to reach the same namespace.

The adopted kernel has origin Upgraded. The next model input names the carried variables and definitions, the jobs the swap stopped, and the cells it ended, and says retained output from before the upgrade is gone while the cells journal keeps every cell's source and recorded result; the transcript records the upgrade. A swap ends the old kernel, so cells it was still running can never deliver their own result: they are journaled interrupted (a forced /kernel upgrade or a protocol-skew swap can end them; the automatic swap waits, because a kernel running a synchronous cell answers Busy to the snapshot and upgradable counts running cells). A background upgrade that cannot proceed keeps a usable old kernel and tries at a later idle moment. A lost old kernel is reported as lost.

The idle moment a stale kernel waits for is a real caller: a turn, a wake, a compaction. Nothing parks at a turn's end just because the queue is empty, and an arriving kernel nobody is waiting for is adopted when the held one is gone — its notice still reaches the model — rather than dropped silently. A pending cell-completion wake is such a caller: it parks while the kernel is unavailable, the swap runs under it, and its turn starts on the replacement carrying the upgrade notice, so the wake is never lost to the swap.

Live jobs keep an older bundle or module set until they end. POST /sessions/{id}/kernel/upgrade is an explicit idle-session action that can stop those jobs. Its response waits for the session to adopt the replacement and reports actual old/new identities and builds, observed stopped job IDs, warnings, and failure. An empty unattached session remains unchanged. There is no automatic retry after a lost HTTP response; read the session kernel first. /kernel upgrade uses this same operation, and /kernel reads the observation.

a kernel booted now that is already stale says this daemon and its own python bundle disagree (a protocol bumped on one side only, a partial install): its boot fails with that reason (albedo_python:out_of_step/1), so a turn waits blocked on it instead of swapping one fresh kernel for another forever.

protocol skew does not wait for jobs: a kernel on another protocol cannot be supervised. its outbox is dropped, not replayed; the swap asks for a snapshot and a shutdown through the frozen frames, carries nothing if the snapshot fails, and the stop's signal ladder ends the recorded process group either way.

lifetime #

a kernel with nothing attached for its grace period reaps its job groups (plugin cleanup) and exits, unless a job or cell is still live. the grace is ALBEDO_KERNEL_GRACE_SECONDS read by the daemon (default 3600) and sent in every attach; before its first attach a kernel waits 60 s. a kernel the daemon reattached at start is attached, so the grace never runs out for it: the maintenance sweep saves and stops one whose session has no loaded actor, no live job or cell, and no turn within ALBEDO_IDLE_SECONDS. a kernel also exits within a second of its run directory disappearing, which is how test homes clean up after themselves. the e2e harness sets a 30 s grace, the gleam test home 20 s.

remote kernels #

a session whose workspace is [user@]host:/path (workspaces.md) runs its kernel on that host. nothing above the bridge changes: the outbox, reattach with backoff and resume_kernels treat an ssh drop as a bridge that exited.

• one ssh layer: priv/python/albedo_ssh.py builds the argv (BatchMode, ConnectTimeout=10, StrictHostKeyChecking=accept-new, ControlMaster=auto, a ControlPath and ControlPersist), finds an agent socket, wraps commands in the remote login shell and stages the bundle. the model's remote plugin imports it, so the daemon and remote.connect() share masters and the staged bundle.
• aliases: albedo_ssh.alias(host, ssh_options, environment) names a host this kernel process reaches without ~/.ssh/config: its -o values go before albedo's defaults (ssh keeps the first value it is given) and its environment reaches ssh and the ProxyCommand it spawns, which is how a token stays out of every command line. it is per process, so the daemon's own remote kernels never see one. colab.md uses it.
• the user's masters: when the user's ssh config names a ControlPath for the host (ssh -G, which reads no network), albedo uses that path and its ControlPersist (600 when it says no), so a master the user opened in a terminal carries albedo, and the master albedo opened carries the user's own ssh, colmena and git in run jobs: one sign-in (one hardware-key touch) per host. with none configured it is /tmp/albedo-ssh-cm/%C for 600 s.
• clean stdout: every remote command is /bin/sh -c '<outer>', quoted with shlex in albedo_ssh.in_login_shell only. <outer> writes the command to a temporary file, keeps the real stdout and stderr on fds 3 and 4, sends the login shell's own output to /dev/null, and asks $SHELL -l -c "/bin/sh <path>": whatever the login profile prints (a motd, a greeting) is dropped, the command gets the real fds back, and the bridge's frames, the remote plugin's kernel and the probe's lines stay clean. the shell is only ever asked for that one quote-free command, so a login shell that is not POSIX (nu, fish, tcsh) still runs it; the redirections and the command itself stay POSIX, and stdin is never redirected (the bridge's frames travel on it). the script unlinks itself before it runs and the shell execs the login shell, so no process is left whose command line holds it (a test finds a host's bridge by its command line).
• fixed command lines: the probe (and, with no network, albedo_ssh.py commands <target> <home>) answers the complete remote commands the daemon runs: bridge (albedo_bridge.py --frame), signal (albedo_signal.py -) and remove. their inputs go on stdin, never in the command line, so nothing in erlang quotes for a shell: the bridge's first frame is {"bridge": {"argv": [start|attach, run_dir, modules?], "cwd": path}} (exit 4 when cwd is no folder), the ladder reads its request line, remove the run directory.
• probe (albedo_ssh.py probe <target>, run by albedo_ssh.erl): one command over the master prints os, arch, home, cores, the python3 version and whether ~/.albedo-remote/<digest[:16]> is staged; python older than 3.11 is unsupported, a missing bundle is staged (tar over the same stream, unpacked beside the target and moved into place, so a bundle directory that exists is complete). ssh's own failure (exit 255) is needs_auth when its words say a person could get in (permission denied, host key verification, passphrase, keyboard-interactive), otherwise unreachable. answers are cached in the daemon, ready ones for a minute and failures for five seconds; a second caller joins a probe in flight. albedo/harness/ssh.gleam is the gleam face.
• boot: kernel.place probes (or reuses the cached probe) before the bridge starts, so a warmed host boots fast. the port owner runs the bridge command over ssh. a fresh boot's exit 4 reads as "not a folder there", 255 as an ssh failure. run directories live under the remote home, ~/.albedo-remote/run/<kernel id>, recorded absolute.
• a host out of reach is not a kernel gone: only an attach the bridge ends with exit 3 (nothing at the run directory) or a refused attach proves a remote kernel gone. ssh failing (255), any other exit, or a startup timeout while attaching is a dropped connection: the session gets the kernel at once, reattaching, and the port owner keeps attaching in the background, backing off to every 30 s, until the kernel's grace (plus a minute) has surely passed since it lost the connection; then it forgets the record quietly, without reaching for the host. when the probe itself fails at boot, a recorded kernel is attached with commands rebuilt from its run directory (which names the home), so a wifi blip during a daemon restart keeps the namespace. a turn in a session with a kernel on record goes through while its host is unreachable or needs a sign-in, waiting like any reattach. only a fresh boot needs the host now.
• ownership: the pids a remote kernel declares are that host's. the albedo_signal.py ladder for its kernel and job groups runs there over ssh (the signal command, 15 s), never locally; if ssh can't reach the host the targets are left to the kernel's own grace exit, which reaps its jobs on the host, and the error says so. only the bridge (a local ssh client) is ever signalled here. the run directory goes with rm -rf over ssh. os_pid answers nothing for a remote kernel, so the reaper's ps never reads an unrelated local pid.
• turns: a session that holds a kernel (attached, or reattaching through its outbox) never probes on a turn. one that must boot probes for up to 3 s: still warming is left to the boot, which waits for the probe; needs_auth, unreachable or unsupported leave the turn admitted and waiting, blocked with the probe's words as its receipt's blockingReason, retried every 15 s like any blocked input (operations.md).
• the model's remote calls (albedo_plugins/remote.py on the owner side, serve_invoke in the remote kernel): one call per frame, answered by one reference. a result that can cross is inlined; anything else is kept live (LIVE, 64 entries, identity-stable handles) so methods call the real object there. awaiting a reference sends one await frame: the remote side awaits the object only when it is awaitable (a job); a settled non-awaitable object -- a spawned browser handle -- is the answer itself, back as the same reference, so an await can never strand a live process with no handle. arguments cross with the same markers as targets and values: {"__ref__": id} for a live reference, {"pending": id} for a call still in flight (resolved in order), {"__bytes__": b64} for byte data, decoded remotely by albedo_values, so rem.run(..., stdin=b"...") and byte-sized results work. rem.write takes text or bytes; either travels on the stdin of one ssh command, never in the command line. a connection that dies stops the jobs it was mirroring as far as this side can tell: its live jobs leave the /jobs list, and every one no reference here awaited or read wakes the session once, naming the job and "connection lost, exit status unknown" (a busy session is retried, as for a local wake). a reference on a dead connection raises RemoteLost from poll()/exit_code instead of reporting None, which reads as a job that never ends.
• the model gets a context line (python extension, place.gleam): "your python kernel and run jobs execute on chernobog (Linux aarch64)…", with the remote home.

Host probes #

GET /hosts?target={host} reads cached SSH probe state. POST /hosts/{host}/probe starts or joins a bounded probe and returns 202. The client polls the read until it reaches a terminal state. Probe state includes safe diagnostics and authentication instructions when operator sign-in is needed. The same probe supports remote folder gathering. See the HTTP contract for the wire format and workspaces for folder browsing.

signing in #

BatchMode stays on for the daemon. when a remote session's waiting turn is blocked, the chat asks GET /hosts?target={host}; if that says needs_auth and the daemon shares the tui's machine, the chat offers ctrl+l: it runs ssh -M -fN -o ControlPath=<control_path> -o ControlPersist=600 <host> through tea.ExecProcess, so ssh asks in the terminal, then warms the host, and the waiting turn starts on the daemon's next try; the daemon rides that master from then on. with a daemon elsewhere the notice says to run ssh <host> on its machine, which carries the daemon when the user's ssh config multiplexes that host. the offer goes once the turn starts or goes away, or the session moves.

not yet #

• profile output a remote login shell prints reaches the daemon's own stderr log on every remote command.