the detached kernel #

a session's python kernel (priv/python/albedo_kernel.py, which runs albedo_cells.py as __main__ so its bytecode comes from the cache) is its own process, detached from the daemon: it outlives a dropped connection and a daemon restart, keeping its namespace and its background jobs. the daemon connects to it directly, or through a bridge when it runs on another host, and every message between the two travels through a small session layer that neither loses nor repeats a message across a reconnect.

process names #

Local Python helpers launch through role aliases in $ALBEDO_HOME/bin: albedo-kernel (session cells), albedo-bridge (socket transport), albedo-spare (unused Linux kernel), albedo-zygote (fork parent), albedo-signal (process-group stops), albedo-discord (bot), albedo-pty (terminal), albedo-ssh (host probes), albedo-verifier (agent gate), albedo-hook (hook commands), albedo-watchdog (browser cleanup), and albedo-colab (websocket proxy).

Remote bridges put kernel aliases in ~/.albedo-remote/bin, beside the staged bundles and run directories, without setting ALBEDO_HOME on the remote host.

Aliases are atomically refreshed when Python changes. If they cannot be written, launch uses the original interpreter. Arguments and script paths are unchanged. A macOS framework Python's bin/python3 is a stub that re-execs Resources/Python.app/Contents/MacOS/Python, which would show as Python, so its alias points at that inner executable instead (not inside a venv, whose stub tells the interpreter where the venv is). sys.executable still names the stub there; sys.orig_argv[0] names the alias. Linux forks rename themselves to albedo-spare, then albedo-kernel on adoption. macOS spares start as albedo-kernel: they are kernels waiting for a session. An albedo_spare.py/albedo-kernel process with ppid 1 is usually an adopted detached kernel awaiting reattach, not an unused spare. If the Linux zygote path is explicitly enabled on macOS, its children retain albedo-zygote; macOS cannot rename an existing process this way.

frame vocabulary #

The daemon sends execute, reply, interrupt, shutdown, snapshot, restore, job_stop, job_list and job_read. The kernel sends ready, startup_error, call, done, job_start, job, trace, cleanup, cells and jobs; job stop and job view replies repeat the request type. The remote plugin also sends invoke, introspect and release, and consumes invoked, introspected and mirror. Beside the sequenced frames, the owner writes {"ping": n} and the link's reader thread answers {"pong": n}, like {"ack": n} a control message of the connection, never persisted or replayed (wedges).

test/harness/kernel_frames_test.py compares these names with the protocol TypedDicts and source literals without booting a kernel. Its explicit baseline records the missing TypedDicts for the cells and jobs observations. It checks names, not payload fields or delivery behavior; dynamically selected names need an extractor update.

host calls #

albedo_python.erl decodes the Python call frame once and passes its method and native arguments to host.Route. Routes return Result(Json, #(code, message)); rpc.handle dispatches by namespace and rpc.reply encodes the answer once. The port owner embeds that answer in the reply frame, encoding the call id separately. Host failure and restart replies use the same rpc.reply shape. A frame without a string method or an args field is refused with invalid / invalid host request; argument validation stays in each route. The Python JSON boundary is unchanged.

tool bindings #

python is the model's only tool: every other capability, including MCP servers (an ACP editor's too) and the lcm/transcript retrievals, is a binding in this namespace, contributed as a Python module by its extension (ToolPlugin.python_modules or Managed.python_modules, routes beside it). Each rule the model reads is stated once, by the extension that owns the behaviour; test/e2e/prompt_test.py fails when a composed request repeats a sentence or carries another tool.

Cells also see the common standard-library modules (asyncio base64 collections datetime functools hashlib itertools json math os re shutil sys textwrap time, and Path) without importing them; these live in the same session-private builtins, so they are not saved as user variables. import albedo and from albedo import files answer the injected bindings (albedo_cells.Bindings).

A cell may not block the kernel with time.sleep of a second or more: albedo_shell refuses it through the same audit hook that refuses raw subprocess calls, and says to start the work with run() and do other useful work while it runs, and when nothing else is left to give the user a status report first, then wait with await asyncio.sleep(n). Job timeouts are seconds, at most a day (albedo_api.check_timeout): a job outlives the cell that started it, so it is not held to the cell's hour.

The kernel keeps injected helpers in cell globals and in a session-private copy of Python's builtins. Injected tool names (files, run, cells, output, etc.) are locked against assignment and deletion in cell globals — both at cell compile time (before any line in the cell runs) and on NAMESPACE at runtime — while local variables inside functions or comprehensions and from albedo import files remain allowed. Preloaded standard-library modules in session builtins may still be shadowed and recovered with del. The process-wide builtins module is unchanged. sys.modules["__main__"] is a ModuleType backed by NAMESPACE (CellMain), so import __main__ and pickle resolve cell definitions instead of albedo_cells. Plugin instances are owned by this kernel's API, not module-level singletons.

file-change traces #

Every session records bounded unified diffs. ACP sessions additionally retain whole-file before/after text in cell_traces for editor diff rendering and replay: a small edit can store roughly twice the file size, up to 256 KiB of snapshot text per trace, until session deletion. Ordinary sessions do not produce or store these snapshots.

A cell the model ran also records calls, the injected bindings its code calls as dotted names (files.find, run, agents.self.spawn), sorted, at most 64, read from its syntax tree (albedo_cells.called_bindings) rather than from what ran. The loop compares them between steps (sessions).

images #

show_image(source, *sources) returns PNG, JPEG, or WebP bytes or paths with the cell result. albedo_capture.Capture owns the four-image and 5 MiB attachment budgets. Overflow is saved losslessly under the session's output directory (or in a temporary directory without one); the returned Text names each file and says to use show_image(path) in another cell. Output-directory files share its 14-day retention. An image over 5 MiB cannot fit even in an empty cell: it raises with its size and advice to downscale or crop, without reading a large file path into memory. Invalid, cut-short, or over-edge attachments still raise. No image decoder is required, so overflow is not tiled.

browser gestures #

priv/python/albedo_plugins/browser/page.py owns page actions and their lock. Use await page.click((x, y)) for viewport coordinates and await page.drag(start, end, steps=20) for left-button drags. Endpoints may be observed nodes, references, or viewport (x, y) points; node endpoints must both fit in the viewport. Drag steps are 1–1000, and mouse release runs even if a move fails. Document-level mouse/pointer listeners do not need capture. Use await page.click_text(text) to click a unique button or link whose accessible name contains the case-sensitive text. Zero or multiple matches raise SelectionError rather than choosing one.

await page.evaluate(source) accepts expressions, zero-argument functions, and statement bodies such as localStorage.clear(); return 1. It selects the syntax before running the source, so a runtime exception never causes a retry. Promises are awaited. This escape hatch still bypasses action locking.

await page.screenshot() returns the PNG's path. Given path= it writes there, as the cell's own change. Without one it files a uniquely named PNG under $ALBEDO_HOME/cache/browser-screenshots/ (browser/paths.py), never the working tree, outside the cell's file-change trace, and first removes PNGs there older than seven days.

prerendering #

Chrome refuses speculation-rules prerendering (Preload.prerenderStatusUpdated says PrerenderingDisabledByDevTools) while a client is attached to the page target itself, which is how b.new_page() and b.attach(target_id) attach. prerender=True on either attaches through the page's tab target instead (Browser._attach_target in browser/session.py): the tab session auto-attaches each prerendered page, and the page initialises it like an iframe session and replays its recorded settings on it. When a click or in-page navigation activates one, Chrome detaches the old page and the tab shows the prerendered target; the same Page handle follows it (Page._activate), so page.target_id changes and earlier snapshots and event scopes go stale. page.wait_for keeps polling across that swap. page.goto (a CDP Page.navigate) never activates a prerender; Chrome discards it and loads the page again.

spawned browsers #

priv/python/albedo_plugins/browser/launch.py owns browser discovery and spawned processes. browser.spawn() uses a temporary profile and headless mode by default; await b.close() stops the owned process and removes that profile. Connecting to an external browser never takes ownership of its process.

A spawned Chrome never outlives its kernel. A normal exit closes it from atexit; for a kernel that dies any other way (SIGKILL, a crash) each spawn starts browser/watchdog.py in its own session, reading a pipe only the kernel holds. close() writes a byte to it first and it exits; end of input without that byte ends Chrome's process group (albedo_proc.terminate) and removes its lock and temporary profile. It also exits once Chrome ends on its own.

Automatic headless launches prefer chrome-headless-shell on PATH, then $ALBEDO_HOME/browsers/chrome-headless-shell (~/.albedo by default), before regular Chrome/Chromium. The Nix dev shell and packaged daemon put the shell on PATH, using nixpkgs' pinned Chrome for Testing binary without the Playwright runtime. Full Chrome's headless launches on macOS can leave duplicate recent-app Dock tiles even after they exit. Outside Nix, install the official Chrome for Testing shell with its support files and symlink its executable at that home path; discovery resolves the symlink so the shell can find its resources. No browser is downloaded by spawn().

executable=, ALBEDO_BROWSER_EXECUTABLE, PRIME_BROWSER_EXECUTABLE, and CHROME_PATH still override discovery. headless=False skips automatic shell selection. An absent shell falls back to Chrome/Chromium; a selected browser's startup failure is reported, not retried with another executable. Dock settings and the user's normal Chrome profile are never changed.

Human shell output #

await output.read_shell(command_id, offset=0, limit=4000) is a read-only host call scoped to this session. It returns {text, next_offset} for a human's ! command without executing or stopping anything. Follow next_offset until null; offsets count retained sanitized UTF-8 bytes, pages do not split codepoints, and limits are 4–65536 bytes. The daemon retains at most 16 MiB per command. This is separate from the kernel cell/job channels read by output.read.

background job observations #

GET /sessions/{id} exposes kernel.live_job_count and kernel.running_jobs without starting a kernel. Unknown observations are null. The list contains up to 100 live jobs ordered by ID, with id, nullable pid, command (up to 4096 scalars), and started_at: wall-clock milliseconds when the port owner recorded the job's group, zero when unknown (a restored link older than the field). Jobs the kernel's remote connections run (remote.connect(...).run) are listed beside the local ones, with no pid (they own no group on this machine) and the wall-clock time the connection first saw them. The remote plugin reports them as {"type": "jobs", "live": n, "awaited": m, "items": [{id, command, service, started}]}, m the ones not started as services; the port owner keeps the latest frame (and a reattach's hello carries it as external_jobs). A kernel from before items reports counts only, and the count then exceeds the list by the awaited ones it could not name. kernel.awaited_jobs is the one definition of which listed jobs will wake the session (not service); awaits_jobs (the cache warmer, resumed kernels) and the session's still-running summary both ask it, so an extension never reads service itself. Stopping a remote job through /jobs sends a job_stop frame down the bridge to the local kernel ({"type": "job_stop", "id": "<job id>"}); if the kernel is unreachable (port := none), stopping fails immediately without queueing. The local kernel forwards the stop to the remote kernel (or, for degraded fallback ssh jobs, stops the job by killing the local ssh client). Fallback jobs observe client exit independently of output EOF: a descendant may retain the pipe after the client exits. Normal completion and stop drain output for up to one second, then close the local transport; a deadline closes it once the client exits. Neither waits for the remote command, whose fate remains unknown. The port owner waits up to 15 seconds for a confirmation frame ({"type": "job_stop", "id": "<job id>", "ok": true|false}); if unconfirmed after 15 seconds, the stop fails. The TUI uses these observations for its idle-job status, animation, and sidebar, and calls a job background only once it has run three seconds (jobGrace), so work a cell starts and awaits never flashes through the chrome; /jobs lists every live job regardless of age.

Observation replies have a last-known cache in the kernel port owner. The owner publishes the complete observation facts before returning the kernel handle and after every facts-changing frame, including bridge loss and reattachment. The protected table publishes the observation, current target PID, and idle live-work count (including cells) together, so the reaper's Idle check never synchronously queries a blocked port owner. The cache is exposed through a protected ETS table whose tid is kept in the owner's process dictionary, so a daemon read does not send a request to a port owner that may be synchronously persisting a host call. The table therefore disappears with an abnormal owner without a central cleanup registry. Cached facts are deliberately last-known while a write is in flight; an owner-death check preserves the ordinary lost-kernel error, and the cache is removed when that owner ends. A missing initial cache falls back to the bounded observation call during startup.

A watching session does not poll for them. It registers a closure with albedo_python:watch/2 (the kernel's pid in the albedo_kernel_watch registry) and the port owner runs it after any frame that changes what an observation reply would carry, comparing a cheap facts tuple so a frame that changed nothing wakes nobody. The session then re-observes and publishes a kernel stream event only when the facts differ (sessions). A dropped kernel handle is forgotten with albedo_python:unwatch/1. The port owner hibernates after 100 ms without a message, compacting its live state and dropping garbage from frame decoding and host calls. The next message wakes it immediately; cells, jobs, deadlines, and reattach timers stay live. The local socket relay uses the same quiet-period hibernation, retaining only its owner monitor and socket rather than its last forwarded frame's heap.

The Erlang port owner truncates commands through text_scalars.take, which scans only the retained prefix and copies it. Do not convert the entire command to a codepoint list to take a bounded preview; see runtime.

/jobs reads GET /extensions/run/sessions/{id}/jobs, whose page declares the stop action. POST /extensions/run/sessions/{id}/jobs/{job_id}/stop supervises that job's recorded process group and reports failure if its stop is unconfirmed. A missing job returns 404. After response loss, read the collection before confirming another attempt; stopping is never automatically retried. Job rows carry no resource; the stop action binds the row id.

job book and output #

GET /extensions/run/sessions/{id}/jobs/recent lists what the kernel's run plugin still knows of the jobs it started, newest first, up to 100: id, command, state (running or exited), exit_code, timed_out, service, started_at and ended_at (wall-clock milliseconds), and bytes written. The kernel keeps the 64 newest finished jobs with their output, so a finished job stays listed that long (and only while its kernel lives); a plugin's own internal jobs are not listed. A session with no kernel has none, and neither does one running another python bundle or protocol than the daemon (stale bundle or protocol, until its session is idle and it is swapped): a kernel that old ends its link on a frame it does not know, so the daemon never sends it job_list or job_read; the list is empty and a read is 503. This is not the live observation above: that one comes from the port owner and covers remote jobs, this one from the kernel and covers finished ones.

GET /extensions/run/sessions/{id}/jobs/{job_id}/output?offset=&limit= reads a page of one job's output: {job, offset, next_offset, text}. Offsets are bytes and pages are cut on characters, so a viewer that continues from next_offset loses none of what a job wrote, even where a pipe read split a character; a running job's half-written last character waits for the next page. limit is 4 to 65,536 bytes (default 16,384). Without offset the page is the end of the output, which is where a viewer opens on a long-running job. A page inside output the kernel did not keep (past the 1 MiB it holds in memory and the 16 MiB it spills) is 409 output_lost; the end of the output is always readable. An unknown job is 404 job_not_found.

Both are asked of the kernel as job_list and job_read frames (the request carries a fresh id and the reply repeats it, ok with the fields or code and error), answered by the handler each plugin registers with api.on_job_view; the run plugin's reads bypass Job.read, so watching a job never withdraws the wake its result is owed. The port owner waits at most 10 s (ASK_TIMEOUT) and fails the request, and does not monitor the caller. Each frame goes through the outbox like any other, so a viewer polling every second costs one write and one delete a second while it is open.

A job that finishes with its result unread wakes the session. The model reads the full notice with the job's handle; the transcript keeps the wake's one-line display (job finished (exit_code=0, ran 1.2s): …) as a note from job.

retained output #

Every cell and job owns one output channel (albedo_capture.Capture), kept for the 16 newest cells and 64 newest jobs plus native. A channel holds the first 1 MiB as written and, once more than that has arrived, the last 64 KiB in a separate tail buffer that exists only from then on; seen counts everything. Job and remote output enter as bytes (write_bytes), never decoded and re-encoded, so a UTF-8 character split across two pipe reads survives and a late job.pipe() replays the exact bytes while seen still fits in the retained start. Text readers (output.read, tail(), previews) decode a window on demand. output.read returns at most 64 KiB per call and raises for a larger limit rather than shortening it, since a caller that asked for everything and got a silent prefix builds files from it; the rest is paged with offset. A single print longer than what could be retained is not encoded whole when it is ASCII: the buffers see both ends and the middle only counts (unless a spill file takes it, below).

job output #

priv/python/albedo_output.py owns JobOutput, shared by local jobs and SSH command-mode jobs. head(n) and tail(n) are previews: up to 64 KiB, with lines= taking precedence. Zero returns empty. An out-of-range request or a character/line request the retained window cannot satisfy raises with job.read() / job.save(path) as the recovery, never silently clamps. A line cut at a window boundary is not counted as a complete line.

After completion, job.read() returns complete UTF-8 text up to 1 MiB; larger whole-text requests raise with the pagination call to bound allocations and remote result frames. job.read(offset=..., limit=...) reads byte-offset pages up to 64 KiB, including while running, and transparently reads spill files. output.read(id, offset=..., limit=...) uses the same disk-backed paging for jobs and cells. Non-UTF-8 bytes are ignored at page boundaries, as in existing previews; whole text reads decode with replacement.

job.save(path) streams the exact bytes into a temporary file in 64 KiB chunks, atomically replaces the destination, and returns its absolute path. It never loads the whole spill file into memory. Running jobs, output lost past the disk cap, or unavailable spill files refuse rather than saving partial output; failed saves leave the destination untouched and remove the temporary file. Pages crossing lost output also refuse instead of returning a short page or false EOF. The error names the retained prefix and its pagination call. For artifacts beyond the disk cap, attach .pipe("tee", path) before awaiting; a late pipe cannot recover dropped bytes.

remote.connect() mirrors the same 64 KiB tail window, coalesced at the existing interval. Larger reads and saves use the ordinary remote method transport. Remote spill files and saves live on the remote host; degraded SSH command-mode jobs spill and save locally, where their capture lives.

spilled output #

albedo_capture.Capture owns spilling and byte reads. Cells spill past the 32 KiB result preview (CELL_PREVIEW), jobs past the 1 MiB in-memory prefix. Both retain up to SPILL_LIMIT (16 MiB) on disk. The first overflowing write opens a private spill file with all preceding bytes; subsequent writes append without buffering, so pages can read live output. Job completion closes the writer. Memory still holds only the first 1 MiB and rolling 64 KiB tail, irrespective of disk output size.

Cell results name their spill file, seen, and whether all output is retained (Capture.spill_note). The directory sits beside the run directories: $ALBEDO_HOME/output locally, ~/.albedo-remote/output remotely, and albedo-output in the temp dir for a kernel over stdio. Spill files survive output channel eviction and are pruned after SPILL_KEEP (14 days) on kernel boot. A spill creation or write failure does not interrupt the command: in-memory previews remain available, but pages beyond the retained prefix and whole-output saves refuse. Missing or shortened spill files raise visibly on reads and saves.

what the model reads #

A cell's result is stored as JSON for clients (python/extension.gleam, outcome_json). The request carries it through python/view.gleam: one JSON line holding status, duration, and only the fields that say something (value for a final expression, cell_id when a failure has no repair note, truncated, repair, image_errors), then the output as printed. Output inside JSON is escaped, which made Go source cost 28% more tokens than the same text raw. A call that could not run ({"cell_id", "error"}) is sent as stored.

The output itself is trimmed of what only the kernel needs. A traceback (cell_traceback) drops the executor's frames (albedo_cells.py, the albedo_memory open guard) and every kernel frame except the one that raised, and names the cell's file <cell>. Compiling a cell ignores SyntaxWarning, so a regex in a plain string ("\d") adds nothing. A cell has a value only when it ends in an expression; the coroutine an awaiting cell runs as is not one. A failed cell of 10 or more lines gets a one-line repair note naming cells.run; a shorter one is cheaper to resend and gets none.

memory cap #

The kernel's physical footprint (albedo_memory.footprint: the resident set from /proc/self/statm, or Darwin's ri_phys_footprint through libproc, the figure vmmap and Activity Monitor show) is capped at ALBEDO_KERNEL_MEMORY_BYTES, 256 MiB by default, which the daemon passes through to local kernels. A thread samples it every 100 ms while cells run and interrupts, with reason memory, every running cell whose threshold the footprint passed; the cell ends interrupted with a note naming what is held and the cap, and the namespace stays. A cell's threshold is the cap, or 8 MiB over the footprint it started at when the kernel was already past the cap (a large variable left behind), so del big; gc.collect() runs instead of being interrupted at once. A cell that catches KeyboardInterrupt is told once. Jobs are separate processes and are not counted.

One allocation can jump past a sample, so the kernel also sets a soft limit at start (albedo_memory.enforce): an allocation past it raises MemoryError inside the cell, and the kernel keeps answering (a cell that hit it gets 64 MiB of room while its result is built). On Linux it is RLIMIT_DATA (private writable mappings, thread stacks included) at the address space the kernel started with plus the cap plus half the cap. A shared anonymous mmap is not counted. macOS cannot set RLIMIT_DATA; there it is RLIMIT_AS relative to the virtual size task_info reports, plus the cap plus twice the cap. That counts file mappings, and the allocator keeps freed regions mapped, so after a spike later large allocations fail until the process is recycled: on the Mac it is a fuse, not a soft cap. Children inherit the soft limit, so subprocess.Popen, os.posix_spawn(p) and os.system lift it for the duration of the call.

After a cell ends with the kernel above its cap (albedo_spill.relieve), each large top-level value (8 MiB or more, the size saved state would skip) is written to the output directory and its name is rebound to a Paged handle: a str as text, bytes raw, a list of single-line strings as lines, anything else pickled. The handle pages like job output (read(offset, limit) by byte offset, head(), tail(lines=)), iterates a text or line value, and load() brings it back unless it is past the cap. It is small and pickles, so it survives a release. The cell's output gets one note naming the values moved and what the kernel holds now. A kernel still above the cap after that reports bloated in its work frames (runtime.bloated); the session then releases it as soon as it rests, without waiting out the idle window or the budget, under the same keep rules: a live cell or job, or a name the save could not keep retain the kernel (a client watching the session does not), and the note says the process stays above the cap.

open() (and io.open, so Path.read_text) returns a thin wrapper for a file larger than the cap opened for reading: read() without a bounded size and readlines() raise MemoryError naming the file and pointing at files.read, line iteration or read(n). Smaller files and write modes are untouched.

Finished results are kept by cell id (the newest 16) for execute replays. Their base64 image text is capped at 8 MiB across all of them, newest first; an older result past the budget replays without its images and says so in its output.

background cells #

The kernel returns a python tool result with status=backgrounded after 60 seconds (ALBEDO_CELL_BACKGROUND_SECONDS overrides the threshold, in seconds). The result includes output so far, the cell id, and instructions to do useful work or report status and end the turn, without polling or sleeping. This is not a final outcome: the journal stays started until cells.finish records it. cells.info/read/trace remain available; traces appear when execution finishes. The original timeout_ms is carried to the kernel and still interrupts the cell at its deadline, even after the daemon's original execute wait has ended.

New cells can run while a detached cell awaits. They share the namespace, so avoid modifying variables the detached cell uses. Python output, images, and activity traces use the task's context; native fd output goes to native when multiple cells run, since it cannot be attributed safely. A synchronous cell can return its tool result early, but other cells must wait until it yields or finishes. Background and deadline timers run outside the asyncio loop so a synchronous cell cannot prevent its deadline from firing. A delayed SIGINT does not interrupt a cell whose code has already returned (the memory guard can sample a cell in its cleanup), nor one whose task is already cancelling; its cancellation cleanup reports jobs that continue running. Nor does it land in a job's own supervising tasks (run.supervising, marked by albedo_proc.SUPERVISING), which inherit the cell's context: the interrupt is the cell's, and its queued cancellation reaches the cell's await while the job keeps running.

At most 16 cells run concurrently. Running captures are protected from eviction; completed output rolls out under the existing 16-capture limit. await cells.cancel(id) requests cancellation and suppresses that cell's wake. Jobs outlive the cell that started them, so the cancel prints the ones it leaves running, local and remote (plugins register on_cell_jobs; api.cell names the cell a job starts in), services aside. Explicit kernel reset/replacement ends its cells; a daemon restart merely reattaches. Internal cells count frames, replayed in the hello, keep idle reaping and automatic stale-kernel replacement from ending a running cell without pretending the cells are process jobs in the public API.

Completion is committed before cells.completed sends a wake. The transcript gets a one-line cell finished (status=ok, ran 90.0s): <first code line> from cell, while the model gets the full notice with the cell id separately. Unread completions share a wake and retry every two seconds while the session is busy; there is no model polling. A wake that parks while the kernel is being replaced starts its turn on the replacement, together with the upgrade notice. The final expression is appended to retained output under [result], so output.read(id) can read it after a background completion. Reading finished output, source, status, or trace retires a pending wake. Reading partial output while the cell still runs does not retire its eventual completion. A read racing with the completion journal also retires the wake.

processes #

• kernel: started in its own session and process group, stdio not tied to anyone. it listens on kernel.sock in its run directory, $ALBEDO_HOME/run/<kernel id>/ (0700; the socket is bound relative to the directory so a long temp home stays under the ~104-byte sun_path limit). kernel.log there catches what it writes before its own output capture starts.
• relay (in albedo_python.erl): how the port owner reaches a local kernel. an erlang process owns a gen_tcp connection to kernel.sock ({packet, 4}) and passes frames to the port owner as a bridge port would, so no python process stays beside a local kernel. a fresh boot starts albedo_kernel.py --run <run_dir> <modules-json> itself (token as its one stdin line; no bridge interpreter, which cost 36 ms CPU per boot on macOS). the kernel leaves the daemon's session, sends early stderr to kernel.log, and closes its stdout once its socket listens, which the relay waits for. the relay announces nothing, so the port owner hashes the bundle itself, the same way albedo_bundle.digest() does. a refused or missing socket reads as exit 3. a run directory whose socket path does not fit sun_path (104 bytes) keeps the bridge.
• bridge (priv/python/albedo_bridge.py): what the daemon's erlang port runs for a remote kernel (the same argv over ssh, see remote kernels below) or a run directory too deep for the relay, with the same {packet, 4} stdio framing the kernel used to have. start <run_dir> <modules-json> starts the kernel beside it (the token goes over the kernel's stdin, never argv) and waits for its socket; attach <run_dir> connects to a running one. it announces its own bundle hash ({"bridge": {"bundle"}}), then copies bytes both ways until either side closes. exit status 3 means no kernel is there. killing it never touches the kernel.
• port owner (albedo_python.erl): one erlang process per kernel, as before. it owns the job groups the kernel reports and the albedo_signal.py termination ladder, aimed at the pid/pgid/leader the kernel declares in its hello. the ladder runs in one shared helper (albedo_signals.erl) that the first stop starts and that exits once a minute passes without a request.

a kernel run with plain albedo_kernel.py <modules> (no --run) keeps the old raw stdio protocol and dies with its pipe: the remote plugin's targets use that.

fresh spares #

albedo_spare.py imports the kernel and packaged default plugins without calling plugin setup or starting kernel threads. The custodian's erlang port starts it directly (no helper interpreter); it leaves the daemon's session and waits on one private stdin adoption frame. Adoption installs the session's cwd, sanitized launcher environment (sent by the custodian), module list, run directory, and token, then starts the ordinary kernel. The kernel closes its stdout once its socket listens; the custodian confirms that socket before reporting adoption. Adoption exposes albedo_cells as __main__, as a cold launch does. Closing the adoption channel before sending any bytes retires an unused spare cleanly; a partial or malformed adoption frame remains an error.

On Linux (or anywhere with ALBEDO_KERNEL_ZYGOTE=1; 0 turns it off) spares are forked rather than started. albedo_zygote.py imports what a spare imports, runs gc.freeze() so children share those pages, and forks one child per connection to its unix socket; the child speaks the same ready/adoption lines over that connection and gets its own event loop. The zygote stays single-threaded, is held by a registered albedo_zygote process that serves the first runtime owner and stops with it, starts with the first spare that needs it, exits once a minute passes without a fork, and exits by itself when the bundle digest changes at a fork, so the next spare starts a fresh one. A forked child remembers the digest its code was imported from, so a bundle edit between fork and adoption is refused like any stale spare. Any zygote failure falls back to a directly started spare and logs zygote_failed (startup) or zygote_fork_failed (one fork) to the daemon log, whichever process asked (a probe's rpc included). A zygote that finds its bundle changed answers the fork with {"stale": digest} before exiting; that fallback is expected and not logged. A cold open with no spare in hand also takes a fork from a running zygote (any owner's), adopting it at once, and launches only when that fails, so a burst past the spares costs forks, not launches. A spare is consumed once; it never sees a second agent. Namespace restore still uses the existing state file.

Each detached kernel writes a private identity.json beside its socket, with its home, run directory, cwd, pid, process group and leader start time, never its token. The link atomically refreshes it under its lock when a job group starts or ends, so recovery retains the kernel's job groups too. At startup the python extension scans this home's run directories, skipping every linked, staged or orphaned kernel. The native discovery entry point lists the run directory first and avoids starting an interpreter when it is absent, empty, or contains only those recorded paths. Unknown entries still go through Python's full identity validation before any ownership record is created. A matching live process identity with no record is recorded in kernel_orphans for the ordinary idle-limit reaper. Old kernels without an identity file, stale identities and other homes are left alone.

Replacing a published kernel after an attach failure first moves its record into kernel_orphans in the same transaction, so the maintenance sweep still owns its cleanup.

The native custodian monitors the runtime owner. Stopping an unused spare checks its process group is gone; an adopted kernel becomes the ordinary link's owned process and survives daemon detachment as before. A source-bundle change rejects adoption, cleans the spare, and falls back to a cold launch. spare_current only checks custodian liveness; the adoption checks the source digest again.

python.live_work reads only the last published count of cells and local/remote jobs, without messaging a blocked owner. Unknown is an error, not zero. A session with a live job or cell must retain its kernel.

Release also requires a positive quiescent observation. The kernel compares threads, asyncio tasks and timers, and open descriptors with its post-adoption baseline; only its own link threads/sockets and sampler are excluded. A failed sample keeps the kernel. Open remote connections, owned browsers, and a live CUA driver keep it through plugin resource predicates. Closing the resource permits a later release; a plain namespace snapshot cannot revive these resources.

Environment additions, changes, or deletions also keep the kernel. Only a hash of the post-adoption environment is retained in memory; environment values are not written to namespace snapshots. Returning the environment to its original values permits release again. Eligibility is published with cell counts before results and refreshed once a second while the event loop can run.

Release keeps what the kernel retained. At save, every retained cell, job, and native output channel is described by a spill file under $ALBEDO_HOME/output (bytes held only in memory are copied there first, and the capture itself is left alone, so a save while work runs does not stop its spill), with the tail the file lacks kept in the index; the run plugin saves its finished-job book (ids, commands, exit codes, durations, start times) through on_state. Both go into the saved state, and a restore revives them before the namespace, so output.read, a job handle's read/tail/head, and jobs answer as before, from disk, holding no cell/job output in memory. The writable native channel reloads its bounded prefix and tail so later writes append. Retention rules stay the kernel's: 16 cells, 64 jobs, 16 MiB spills, 14 days. The restore result's output_lost names every retained channel that did not come back (output the save could not copy, a spill file gone since, plugin state that failed to save or restore); empty means nothing was lost, and it is absent for a state saved before this existed. A state hook that kept only part of its state raises albedo_api.PartialSave(state, lost): the kept part is saved and restored, and only the named parts count as lost (the job book does this for a job it cannot save, and restores every job whose output is still on disk).

The run plugin also writes <state>.jobs.json atomically beside the state: the finished jobs as the job book lists them, plus each one's spill path and stored tail. While no kernel is attached the run routes (/jobs/recent, /jobs/<id>/output) answer from it, so a viewer polling jobs never boots a kernel; with a kernel attached they ask the kernel as before. albedo_job_window.erl pages it exactly as read_window pages a finished job (test/harness/saved_job_window_test.py pages one file through both). The file is deleted with the state.

A lossless restore is silent, during a model wait or between turns. Automatic releases keep the kernel if variables, retained output, or plugin state cannot be saved. Live jobs keep it until they finish; the next save includes their finished handles and output. Spill filenames use a bounded readable prefix and an ID hash, so provider tool-call IDs never become filesystem component limits; old spill paths still restore unchanged.

Only actual restore losses produce a model note and a daemon log entry: missed variables with their reasons, retained output channels or plugin state, and jobs or cells ended by the restart. Old states without output-loss accounting do not by themselves imply a loss. A forced release or crash can still lose values.

handshake #

the daemon writes {"attach": {"kernel", "token", "ack", "grace"}} first. the kernel checks the token (kept in the daemon's sqlite), bumps its epoch, cuts off any older connection (newest attach wins), and answers {"hello": {protocol, bundle, epoch, ack, pid, pgid, leader, ready, jobs, external, awaited_external, external_jobs, dropped}}, or {"refused": reason}. protocol (1) and the hello reports the running cells count. bundle is albedo_bundle.digest(), the same content hash the remote plugin stages under; a hello whose bundle or protocol differs from the bridge's (or, for a local kernel, the port owner's own hash) makes the kernel stale (see version skew). jobs (live job_start frames) and external let a fresh port owner take over job ownership and live-job observations.

session layer #

after the handshake every frame is {"seq": n, "ack": m, "frame": {...}}, or a bare {"ack": m}. ack is cumulative: everything up to it arrived.

• kernel side (priv/python/albedo_link.py): an in-memory outbox of unacknowledged frames, resent after each hello. mirror (per handle), trace (per cell) and jobs coalesce latest-wins; everything else must be delivered. bounded at 4096 frames / 64 MiB: coalescing frames go first, then the oldest, counted in the next hello's dropped. a daemon frame at or below the last seq applied is a replay and is dropped; each applied frame is acknowledged at once.
• daemon side (python/link.gleam, tables from the python extension's kernel_links schema migration): kernel_links (one row per session: kernel id, token, run dir, workspace, modules, pid/pgid/leader, epoch, last seq sent, the job groups it owns, rewritten as they change, and the $ALBEDO_HOME of the daemon that booted it), kernel_outbox keyed (session, kernel, seq), bounded at 2048 frames / 64 MiB per kernel, and kernel_calls. the port owner persists each frame before writing it, and acknowledges kernel frames within 100 ms. what the kernel acknowledges is deleted in the commit that saves the next frame (or reply), not in a commit of its own: a frame kept past its ack is only resent on the next attach, and the kernel drops frames it has seen. its own last-seen seq lives in memory: after a daemon restart it accepts everything the kernel resends, and the dedupe rules below make that safe.

dedupe:

• host call id: a call already running in this daemon is ignored (its reply goes out through the outbox); one answered already is ignored the same way; one an earlier daemon started without answering is answered {ok: false, code: "unknown"}, never run again. a call's row goes once the kernel acknowledges its reply. cells.started is the exception: it gets no row and its reply is not saved, because the host is passed the call id and answers a resend of the same id the way it answered the first (the cell row keeps the id that started it) while refusing any other start, so the call costs one commit instead of three. since those replies are not saved, the port owner numbers on from the larger of its saved last seq and the kernel's acknowledged seq after a hello.
• a model's cell is saved in the commit that saves the execute frame that runs it, never before; one that never reached the kernel (an invalid timeout, a busy or lost kernel) is saved with its error when the tool call settles.
• execute, by cell id: a finished cell (the newest 16) resends its done; a queued or running one is ignored. callers must not reuse cell ids.
• interrupt, reply, release are idempotent already. an interrupt for a cell still queued behind the loop is kept and cancels the cell as it starts; one for a cell the kernel no longer holds is dropped.
• job_stop: the port owner asks the local kernel to stop a remote background job with {"type": "job_stop", "id": "<job id>"}. The kernel answers with {"type": "job_stop", "id": "<job id>", "ok": true} or {"type": "job_stop", "id": "<job id>", "ok": false, "error": "<lowercase why>"} exactly once per request. The port owner does not queue while the kernel is unreachable (port := none), and gives up with an unconfirmed error after 15 s. The request is delivered like any other frame, so one sent while the kernel was briefly out of reach can still land after a reattach and end the job later: "unconfirmed" means the outcome is unknown, not that the stop failed.
• job_list, job_read: the port owner asks the kernel for its job book or one page of a job's output (see job book and output). The kernel answers each with the same type and id; a reply after the owner's deadline is dropped.
• what the kernel resends after a restart: job wakes are host calls (jobs.completed), so the call ledger covers them; job_start, job, and jobs only update ownership and observations, which converge; a done nobody waits for rewrites the same outcome; traces save by cell id.

the runtime scheduler #

harness/runtime/boot.gleam owns every worker the runtime actor starts. Kernel stops, detach, and orphan cleanup queue as StopKernel requests through boot_next. They have two separate worker slots, so blocked stops cannot occupy boot or observation slots. A monitored worker reports both its result and an abnormal exit. The runtime reserves the session in stopping until completion; opens, resets, forgets, attaches, upgrades and reloads for that session wait there. Reset, forget and delete also wait for an existing boot rather than abandoning its identity: a replacement cannot restore saved state or attach while the old kernel is still stopping, and deleting a session whose kernel is booting stops that kernel once it is up. A stop whose kernel owner exits during the call (a cell's forced stop ends the owner) supervises the recorded processes directly, as for an owner that was already gone. Completion replies follow bookkeeping updates. A failed stop retains the held kernel and composition and fences further boots, attaches, upgrades, and reloads until an explicit reset, forget, or delete confirms cleanup. Delete returns the failure, while reset and forget (whose public replies are Nil) log it. Shutdown rejects new work while draining.

Shutdown waits for admitted boots, observations, and stops before one monitored worker detaches or ends the remaining kernels, closes compositions, and closes the ledger. The daemon's shutdown deadline still bounds this drain.

Booting, attaching, swapping a stale kernel at open, an explicit upgrade, recomposing for a reload (proposed or desired), and a composition read all queue as a BootRequest and run through boot_next, which admits them into boot_slots workers, one class never starving the other. A composition read for a session whose composition is being prepared waits until it is, then reads the prepared one instead of repeating its discovery (a new child's creation reply observes its composition while the child prepares it). A handler that queues a request ends in |> boot.boot_next in serve; nothing spawns a kernel worker outside start. Each worker computes off the actor, then delivers its message or, when the runtime stopped meanwhile, releases what it made (work). runtime/state names the generation lifecycle a request belongs to: admit opens one, current_waiters says whether a result is still for it, generation_over ends it and replays what waited.

fresh spare kernels #

The runtime keeps a bounded pool of fresh local Python processes, but only while released sessions wake: one asking for its kernel again keeps the pool warm for a minute, and once a minute passes without another the ready spares exit. A release alone warms nothing, since most released sessions stay idle. At rest there are none, so one person's single agent pays for no spares. ALBEDO_KERNEL_SPARES caps the pool at 0 to 32 for the machine's size; the default is 2. A spare imports the kernel core without an agent namespace, workspace, plugin setup, or session environment. A spare is consumed once. Released kernels exit instead of returning to the pool.

boot_next admits warmup and cleanup requests through the existing worker budget. Warmups wait behind session work and leave one admission slot free. The runtime holds one ticket per queued or running warmup. Admission removes a spare from the ready list before starting an open worker, so concurrent sessions cannot adopt the same process. An empty pool uses the ordinary cold open. Adoption consumes the spare even when an existing kernel reattaches or a remote workspace uses another launch path.

The Python owner checks the source bundle before adoption. A stale spare stops before a cold replacement starts. Explicit reloads and upgrades retire ready spares and invalidate in-flight warmups. Failed cleanup retains its resource in the bounded pool accounting. Warmup failures retry after one second and mark the pool unhealthy until the retry. Shutdown drains admitted workers, stops all remaining spares, then closes the ledger. A spare also monitors the runtime actor and stops when that owner dies; spares never survive a daemon restart.

Kernels stay resident while their session uses them. A resting session releases its kernel once two minutes pass without a turn, cell or job, whether or not a client is watching it (idle_release_ms in session.gleam; each turn end or job ending restarts the wait, and a kernel swap or boot under way when the wait ends defers it a second at a time), and a long model wait releases a waiting turn's kernel (runtime.md). The kernel memory budget (ALBEDO_KERNEL_BUDGET_MB, default a tenth of physical memory, at most 256 MiB; 0 releases every idle kernel) is a cap on top. Each maintenance sweep adds up the live kernels' resident sets; near or over nine tenths of the budget it marks the daemon pressed (albedo_daemon:kernel_pressure/0, read without a call; between sweeps live kernels times the last average count too), releases idle kernels past the 30 s detached grace (or the idle limit, if shorter) and then the least recently used ones until the sum is back under, and while pressed a completed turn releases its kernel at once. The sum includes reattached kernels no session has loaded. A release saves the namespace, rechecks that the session is resting and has no live cell or job, and keeps a kernel whose snapshot failed or whose live work is unknown. A kernel whose snapshot reports unsaved names (a generator, a value over the cap) is kept too, by the resting release, the sweep's idle release, its release of a reattached kernel no session has loaded, and an unload (session_namespace.may_release); it counts against the budget and stays until a person releases it. Only a sweep whose pool is still over the budget once every kernel that saves whole has gone releases one at a loss, reattached kernels no session has loaded included (relieve and release_unowned in maintenance.gleam): the longest unattended first, one at a time until the pool is back under. The daemon log names the session, and its next restore tells the model what was lost. Closing an agent also releases it. A kept kernel that runs nothing writes nothing: its snapshot answers with the last one (robot-docs/kernel-state.md). Release does not wait for a ready spare, so a drained pool costs the next open a cold boot (about 147 ms more on macOS). A test shortens the idle wait with ALBEDO_TEST_TIME_SCALE; one that needs a kernel to stay holds it the way a person would, with a running cell or a live job.

drops #

• the connection drops (the relay's socket closes, or the bridge dies): the port owner reattaches with backoff (50 ms doubling to 2 s, 40 tries); exit 3 (no socket, or nothing listening on it) or a refusal means the kernel is gone and the old abandon path reaps it, unless the kernel is local and its recorded pid still exists: macOS refuses a unix connect once the listen backlog of a kernel that stopped accepting is full (Linux does not refuse it), so that exit 3 counts as a failed try. only that proves a recorded kernel gone: an attach that fails any other way (a socket that refuses the connection for another reason, an invalid handshake, no hello within the startup window) logs kernel_unreachable_at_attach with the reason, hands the session the kernel as reattaching, and keeps attaching in the background. a local attach that gets no hello within 2 s looks at the recorded pid's state (/proc/<pid>/stat on linux, ps -o state= elsewhere): no such process is proof of absence and takes the gone path; a stopped (T) or zombie (Z) one counts as a failed try; anything else is busy (a call holding the GIL, say). a busy kernel is waited for on that connection without spending tries, looked at again every 2 s: ending it would lose its computation, and /kernel restart (restart) is what replaces one that never answers. while busy the kernel facts state is busy, the session stream gets one note ("the python kernel is busy and has not answered since the daemon restarted; /kernel restart replaces it"), a cell whose deadline passes meanwhile is answered as detached with that reason, and the model gets no history note, since nothing was lost. a local kernel still unreachable after the 40 tries is orphaned (kernel_unreachable in the log): its row moves from kernel_links to kernel_orphans (pid/pgid/leader, run directory, job groups, when), the cell waiting on it hears lost, and the session's next open boots a fresh kernel at once, so only the first open waits on the old one. the maintenance sweep ends an orphan once it has been one for ALBEDO_IDLE_SECONDS (scaled like the sweep's timer): the signal ladder on its process group and job groups (SIGTERM, then SIGKILL, which also ends a stopped process), then its run directory goes (kernel orphan <id>: ended in the log). past 64 orphans the oldest go at the next sweep; a failed stop keeps the row for the next one. orphans outlive a restart, and link.disown drops another home's. the fresh kernel's first turn tells the model the previous kernel could not be reached and its variables and jobs are not available in this one, naming the jobs left in it. a remote kernel keeps its expiry: once its grace has surely passed it has ended itself, and its host is out of reach anyway. a failed boot or proven-gone attach logs kernel_startup_failed before it is reaped. a port owner attaching after a restart starts from the record's pid/pgid/leader and groups, so a kernel that died hard while nobody was attached still has its process group and its jobs' groups ended. the leader token (start time, from /proc or Darwin's sysctl) keeps a reused pid from being signalled. the cell keeps running meanwhile.
• the cell deadline passes, or an interrupt goes unanswered for 2 s, while detached: the interrupt waits in the outbox and the caller gets kernel.Detached, which leaves the journaled cell started (the model is told to check cells.info). a done that arrives for a cell nobody waits for any more is journaled through the cells.finish host route.
• the kernel stops answering while attached: see wedges. the owner says why (asleep or computing in native code, stopped), tells a waiting model, and holds the cell to its deadline itself, since the kernel's own timers cannot fire; a kernel nothing bounds gets the idle limit. the replacement restores the last save.
• the daemon crashes: its connection closes (a bridge sees EOF and exits); the kernel waits.
• graceful shutdown: runtime.detach_kernels first, so closing sessions and stopping the runtime let kernels go instead of ending them. a shutdown with stop_kernels (albedo daemon --stop --kernels) skips the detach: sessions close as idle ones do, saving their variables, and their kernels end with their jobs. the drain has the request's timeout_ms (default 5 s; the cli asks for the 30 s maximum with --kernels): a step still running then, or one that fails, is named in daemon.log and the vm halts without it, as in a crash, so kernels not yet ended keep running for the next daemon. a drain that finishes logs each step's time on one shutdown: line.
• daemon start: first link.disown forgets, without touching them, the records another daemon wrote: a row naming another home, or an older row without one whose local run directory is outside this home (an older remote row can't be told apart and stays). a daemon booted on a copy of albedo.sqlite would otherwise attach to the original's kernels with the copied token, replay their outboxes into them, and stop their process groups when it swapped them.
• daemon start: runtime.resume_kernels reattaches every recorded kernel in the background, one at a time through the boot slots, so late results and job wakes arrive without waiting for the session. no actor is loaded for an idle session yet, so a wake finds no registered closure: albedo_wakes then calls the loader the registry installs (on_missing), which starts the session as a Lookup would, and delivers to it; only a session that cannot start answers unavailable, which the kernel does not retry. a session whose kernel came back running a job that is not a service is started at once, idle, so its cache warming resumes (robot-docs/cache-warming.md). a session that asks for its kernel meanwhile waits for that attach, and gets a fresh boot if there was nothing to attach to. it gets what an open would: an attached kernel that turned out stale (its protocol or bundle is only known from its hello) and that no live job keeps is swapped before the waiter has it. an attached kernel's runtime.origin is Resumed, and the session keeps its namespace without the disk restore. a recorded kernel whose module set no longer fits is kept and marked stale; one in another workspace is stopped and replaced. when jobs or background cells are still running, the session's first turn on a Resumed kernel carries a model note that the kernel reattached with its namespace, naming those jobs (id and command) and cells (id and first line), so a live handle is not mistaken for one the restart killed. a session with history is told even when nothing runs ("Nothing was running"), so it need not probe its variables. an origin is handed out once: later opens of the same kernel get Kept, so no notice repeats and no live cell is journaled as ended. the note is built from the port owner's observation and the cells journal, never from a new kernel frame, so a reattached kernel on an older bundle still gets it.

a recorded kernel found gone (killed while the daemon was down, or dead when its session asks) boots a Fresh replacement. the dead record's job groups carry id and command, and for a local kernel albedo_python:death_note reads why it died from <run dir>/kernel.log (Fatal Python error: ..., received SIGTERM / received SIGHUP, or the last non-empty log line before the run directory is removed), so link.lose_with_reason keeps both in kernel_lost ({"jobs": [...], "reason": "..."}; legacy array rows still decode) before the record is forgotten, and open reads them before the replacement's record clears them (Fresh(lost_jobs, loss_reason, ...)). this is needed because a starting daemon reattaches in the background and finds the kernel dead before its session asks. adopting a Fresh kernel in a session with history ends the journal's unfinished cells as interrupted (nothing of the old kernel runs) and the model note names them with the lost jobs: "Jobs stopped by the restart: id (command). Cells ended by the restart: id (first line).", after the restored-variables or reset sentence. a reset with nothing to name keeps the bare reset notice. deleting a session, and /cd which deletes its namespace, clear kernel_lost; the model's own reset names nothing.

explicit drops still end the kernel at once: reset, release by the idle sweep, /cd, session close, extension reloads (the replacement boots beside the live kernel, never attached to it), and the old kernel after a swap.

a tree that fails at startup #

new local kernels start from the daemon's own priv/python, which in a checkout is the working tree, so a half-saved edit there reaches every fresh boot in every session. each tree a fresh local kernel reaches ready from, on the bundle its port owner hashed, is copied (off the boot path) to $ALBEDO_HOME/kernel-bundles/<bundle digest>, and kernel-bundles/good names the newest; a copy whose digest no longer matches (an edit while copying) is dropped, and the four newest besides good are kept. when a session's fresh boot fails Unavailable (python exits, a plugin's setup raises, the tree changed while it booted, the startup timeout), open boots once more from the good copy if it differs from the tree as it reads now. that boot skips the spare and the zygote, which imported the failing tree, and is not stale: its bridge hashed the same copy. the session's origin is Fresh with fallback set, and the model's first note says the kernel runs the last code that started cleanly and why the current code failed; the next fresh boot tries the daemon's tree again, so a fixed tree goes live with no restart. remote kernels, upgrades and reload swaps have no fallback (an upgrade that fails keeps the kernel it had).

a local kernel's stderr goes to <run dir>/kernel.log until its own capture starts, and albedo_cells.main() keeps an open descriptor (DEATH_FD) on <run dir>/kernel.log for faulthandler and its SIGTERM/SIGHUP handler before redirecting fd 2 to its native capture pipe. a failed boot's reap removes the run directory, so the port owner appends the log's last 2000 bytes to the startup error first: "python exited at startup: Traceback ... TypeError: ...", and a crash or signal death after startup is read by death_note into the session's reset or restore notice.

version skew #

a kernel is stale when it runs another bundle than the bridge that reached it (for a local kernel, the bundle the port owner hashed), speaks another protocol (both read from the hello), or booted with another module set than its session now has (found at reattach). kernel.stale answers the reason and whether the swap was forced. GET /sessions/{id} returns the native kernel observation with instance_id, build, state, stage, stale, and live_job_count. Unknown identities and counts are null. Reading this resource does not prepare or replace a kernel. stage reports remote bundle staging while the owner is booting.

At the next idle moment, session_namespace.ready lets the runtime replace a stale kernel when no live jobs keep it. The runtime does the work off its actor:

1. Snapshot the old namespace to namespace.state in its run directory.
2. Boot a replacement beside it. The replacement's identities, groups and outbox belong to kernel_stages; the old kernel_links row stays authoritative.
3. Check that the replacement is attached and current, then restore its namespace.
4. Mark the staged replacement ready, stop the old kernel and its jobs, and confirm that the old durable ownership record was removed.
5. Publish the replacement in kernel_links and adopt its actual handle.

Boot, validation or restore failure stops the staged candidate and keeps the old kernel. A failed old-kernel stop reports the actual failure and attempts to stop the staged candidate. Shutdown may already have affected jobs; the report cannot promise to restore those external effects. Failed cleanup retains the relevant durable identities for a later verified stop. No failed stop is reported as a successful swap.

On restart, storage publishes a ready candidate if its old link is absent. The daemon then supervises abandoned staged candidates before admitting work. Staged owners use their immutable kernel ID throughout publication, so replies and ownership updates continue to reach the same namespace.

The adopted kernel has origin Upgraded. The next model input names the carried variables and definitions, the jobs the swap stopped, and the cells it ended, and says retained output from before the upgrade is gone while the cells journal keeps every cell's source and recorded result; the transcript records the upgrade. A swap ends the old kernel, so cells it was still running can never deliver their own result: they are journaled interrupted (a forced /kernel upgrade or a protocol-skew swap can end them; the automatic swap waits, because a kernel running a synchronous cell answers Busy to the snapshot and upgradable counts running cells). A background upgrade that cannot proceed keeps a usable old kernel and tries at a later idle moment. A lost old kernel is reported as lost.

The idle moment a stale kernel waits for is a real caller: a turn, a wake, a compaction. Nothing parks at a turn's end just because the queue is empty, and an arriving kernel nobody is waiting for is adopted when the held one is gone — its notice still reaches the model — rather than dropped silently. A pending cell-completion wake is such a caller: it parks while the kernel is unavailable, the swap runs under it, and its turn starts on the replacement carrying the upgrade notice, so the wake is never lost to the swap.

Live jobs keep an older bundle or module set until they end. POST /sessions/{id}/kernel/upgrade is an explicit idle-session action that can stop those jobs. A running or queued turn answers 409 session_busy; a kernel or composition still being prepared (a restart reattaching the kernel, say) is waited out, and the upgrade runs once it arrives. Its response waits for the session to adopt the replacement and reports actual old/new identities and builds, observed stopped job IDs, warnings, and failure. An empty unattached session remains unchanged. There is no automatic retry after a lost HTTP response; read the session kernel first. /kernel upgrade uses this same operation, and /kernel reads the observation.

a kernel booted now that is already stale says this daemon and its own python bundle disagree (a protocol bumped on one side only, a partial install): its boot fails with that reason (albedo_python:out_of_step/1), so a turn waits blocked on it instead of swapping one fresh kernel for another forever.

protocol skew does not wait for jobs: a kernel on another protocol cannot be supervised. its outbox is dropped, not replayed; the swap asks for a snapshot and a shutdown through the frozen frames, carries nothing if the snapshot fails, and the stop's signal ladder ends the recorded process group either way.

wedges #

A cell can leave the kernel unable to run any Python: native code that holds the GIL (a C call that blocks, catastrophic regex backtracking, a huge sort), or a process that was stopped. Then the kernel's own threads cannot fire, so neither the cell's deadline nor an interrupt nor the background notice gets out, and before this section the daemon learned of it only when a waiting cell reached its deadline (up to an hour), or never, for a backgrounded cell. The port owner (albedo_python.erl) now tells such a kernel from a busy one and acts on its own.

Probe. While the kernel has work (a cell someone waits for, cells it reported live, backgrounded cells kept to their deadline here, or a hello still owed since a daemon restart), the owner writes {"ping": n} every 5 s and the link's reader thread answers {"pong": n} (albedo_link.SocketLink._session). The answer needs only the GIL, which a cell blocking the loop in Python hands over every few milliseconds; native code holding it, or a stopped process, cannot answer. An idle kernel is never probed. Probes run on the test clock (ALBEDO_TEST_TIME_SCALE).

Verdict. After three unanswered pings the owner reads the process (sample: /proc/<pid>/stat, or ps -o state=,time=) and compares its CPU time with the sample taken at the first miss (classify, pure; test/harness/kernel_wedge_test.gleam):

• asleep: under a twentieth of the window on the CPU. A C call blocking with the GIL held: nothing will come out of it.
• spinning: more. Native code computing with the GIL held, which may finish.
• stopped: T or Z. gone: the relay reports the closed socket in a moment.
• unresponsive: a remote kernel (its pid is another machine's), or a process that could not be read.

A verdict informs; a deadline ends. Native code that holds the GIL for a while (a C call that blocks for eight seconds) looks exactly like one that never returns, and ending it would lose the namespace, so no verdict ends a kernel by itself: the attach rule that a busy kernel is waited for still holds. kernel_unresponsive is logged once with the verdict, the misses, the silent time and the CPU delta, and kernel_answering_again when a pong arrives later. A waiting caller whose cell has run past the background threshold (ALBEDO_CELL_BACKGROUND_SECONDS) is answered as backgrounded by the owner, with output saying the kernel has not answered, what holds it (asleep or computing), that it ends at its deadline, that /kernel restart ends it sooner, and to report to the user and end the turn, since the kernel's own background reply cannot get out. The cell then counts as backgrounded here. A waiting cell that reaches its deadline before the threshold goes through the ordinary ladder (interrupt, 2 s, kill), but a kernel that could not answer a probe answers the caller with a synthesized interrupted result naming the cause instead of kernel lost.

Ending and replacing. When a deadline or the idle limit ends a wedged kernel, kernel_wedged goes to the daemon log with the reason ({deadline, cell} or {idle, verdict}). A waiting caller gets a synthesized done with status interrupted whose output says what happened and that the next cell starts a fresh kernel restoring the variables last saved to disk; cells backgrounded here are journaled interrupted through cells.finish with the same note and the session gets the wake the kernel would have sent (cells.completed, retried every 2 s while the session is busy, as the kernel retries). No shutdown frame is sent, since it could not be read: the port is closed and the signal ladder ends the kernel's group and its jobs' groups with the escalated budgets. The owner exits; the session's next open finds the kernel gone and boots a Fresh one, which restores kernels/<id>.state as after any lost kernel; a restore that brings everything back is silent, as any is, and what the model was told is in the cell's result or the wake. Unlike /kernel restart, nothing is discarded. A kernel nothing bounds (no cell waited for, none backgrounded here: busy since a daemon restart, or running cells this owner never sent) that is asleep or stopped is ended once it has been silent for the idle limit (ALBEDO_IDLE_SECONDS, on the test clock), as an orphan is, instead of being waited for forever; a spinning one is still waited for, and /kernel restart is the way out of it.

Backstop deadline. Every cell answered as backgrounded keeps its deadline in the owner: timeout_ms of wall time from the execute, plus 10 s of grace on the test clock for the kernel's own interrupt to land, so a healthy kernel always ends the cell first. A cell the owner backgrounded itself gets no grace, since the kernel's timer is known to be held, and its deadline runs on the test clock, as a waiting cell's does. Past that, with the cell still running, the owner sends the deadline interrupt itself and, if the cell still runs 2 s later, ends the kernel as above with a note naming the cell and its deadline, which also ends a backgrounded cell that catches and ignores its interrupt. A cell is known to have ended by its done frame, by the cells.finish call a journaled cell makes through the owner, or by a cells frame reporting none live, so a healthy kernel that ended the cell at its own deadline is never touched. The owner's record of backgrounded cells does not survive a daemon restart; after one, a spinning kernel's backgrounded cells have only the kernel's own deadline, and /kernel restart is the way out while it computes. A detached kernel (port none) is not probed, and a backstop that fires while detached waits for the reattach.

Timing with the defaults: a kernel that stops answering is diagnosed in the log 15 s later; a waiting model is told at the background threshold (60 s); the kernel is ended at the cell's deadline (plus 12 s for a cell the kernel itself backgrounded), or after 10 minutes of silence when nothing bounds it. The probe costs one 12-byte frame every 5 s while a cell runs, and one ps (or /proc read) per unanswered ping.

restart #

/kernel restart (POST /sessions/{id}/kernel/restart, session.restart_kernel) is how a person gets a session out of a kernel that no longer answers: a cell that swallows KeyboardInterrupt in a loop, C code that never checks signals, or a kernel the daemon found busy after its own restart, which nothing else replaces before the idle limit (wedges). It needs no answer from the kernel. The session actor first does what esc does: queued inputs are cancelled and the running turn is cancelled, with its run aborted 2.5 s later. A restart that arrives while a boot, release or workspace move is under way waits for it, as an upgrade does. Then it discards the saved variables (kernels/<id>.state), journals the session's unfinished cells as interrupted, records a model notice ("The user restarted the python kernel with /kernel restart, so all variables are lost.", or ", which started you, restarted …" when an agent asked, followed by the jobs it stopped and cells it ended), drops its kernel, and sends the runtime a Restart before any later open. The runtime stops the kernel it holds for the session, or else the one recorded for it (stop_recorded, as a delete does: a restart right after a daemon restart can come before the recorded kernel is attached again), with the signal ladder on its process group and job groups, keeps the composition, and runs the restart's answer callback with the stop's outcome. The HTTP reply waits for that: old_kernel_id (the kernel the session held, null when it held none), stopped_jobs, ended_cells. A failed stop answers 503 kernel_restart_failed, is logged as kernel restart <id>: <reason>, and fences the session's boots as any failed stop does, until a later restart, forget or delete supervises the recorded processes. The next turn boots an empty kernel. revive leaves it empty instead of restoring from disk, because the notice says the variables are gone.

/kernel restart <session> restarts another session's kernel the same way: the session is resolved as mail resolves it (an id, or a child's, sibling's or parent's name), and the registry restarts it through the agents seam (agents.RestartKernel). It is the one form of /kernel the model may call (commands.kernel("restart", "<session>")), and only for its own children and peers (family.owns, the rule that also governs switching, cancelling and closing them), so a parent can rescue a child whose kernel stopped answering, which the child cannot do from inside it. The user may name any session.

lifetime #

a kernel with nothing attached for its grace period reaps its job groups (plugin cleanup) and exits, unless a job or cell is still live. the grace is ALBEDO_KERNEL_GRACE_SECONDS read by the daemon (default 3600) and sent in every attach; before its first attach a kernel waits 60 s. a kernel the daemon reattached at start is attached, so the grace never runs out for it: the maintenance sweep saves and stops one whose session has no loaded actor, no live job or cell, and no turn within ALBEDO_IDLE_SECONDS. a kernel also exits within a second of its run directory disappearing, which is how test homes clean up after themselves. the e2e harness sets a 30 s grace, the gleam test home 20 s.

remote kernels #

a session whose workspace is [user@]host:/path (workspaces.md) runs its kernel on that host. nothing above the bridge changes: the outbox, reattach with backoff and resume_kernels treat an ssh drop as a bridge that exited.

• one ssh layer: priv/python/albedo_ssh.py builds the argv (BatchMode, ConnectTimeout=10, StrictHostKeyChecking=accept-new, ControlMaster=auto, a ControlPath and ControlPersist), finds an agent socket, wraps commands in the remote login shell and stages the bundle. the model's remote plugin imports it, so the daemon and remote.connect() share masters and the staged bundle.
• aliases: albedo_ssh.alias(host, ssh_options, environment) names a host this kernel process reaches without ~/.ssh/config: its -o values go before albedo's defaults (ssh keeps the first value it is given) and its environment reaches ssh and the ProxyCommand it spawns, which is how a token stays out of every command line. it is per process, so the daemon's own remote kernels never see one. colab.md uses it.
• the user's masters: when the user's ssh config names a ControlPath for the host (ssh -G, which reads no network), albedo uses that path and its ControlPersist (600 when it says no), so a master the user opened in a terminal carries albedo, and the master albedo opened carries the user's own ssh, colmena and git in run jobs: one sign-in (one hardware-key touch) per host. with none configured it is /tmp/albedo-ssh-cm/%C for 600 s.
• clean stdout: every remote command is /bin/sh -c '<outer>', quoted with shlex in albedo_ssh.in_login_shell only. <outer> writes the command to a temporary file, keeps the real stdout and stderr on fds 3 and 4, sends the login shell's own output to /dev/null, and asks $SHELL -l -c "/bin/sh <path>": whatever the login profile prints (a motd, a greeting) is dropped, the command gets the real fds back, and the bridge's frames, the remote plugin's kernel and the probe's lines stay clean. the shell is only ever asked for that one quote-free command, so a login shell that is not POSIX (nu, fish, tcsh) still runs it; the redirections and the command itself stay POSIX, and stdin is never redirected (the bridge's frames travel on it). the script unlinks itself before it runs and the shell execs the login shell, so no process is left whose command line holds it (a test finds a host's bridge by its command line).
• fixed command lines: the probe (and, with no network, albedo_ssh.py commands <target> <home>) answers the complete remote commands the daemon runs: bridge (albedo_bridge.py --frame), signal (albedo_signal.py -) and remove. their inputs go on stdin, never in the command line, so nothing in erlang quotes for a shell: the bridge's first frame is {"bridge": {"argv": [start|attach, run_dir, modules?], "cwd": path}} (exit 4 when cwd is no folder), the ladder reads its request line, remove the run directory.
• probe (albedo_ssh.py probe <target>, run by albedo_ssh.erl): one command over the master prints os, arch, home, cores, the python3 version and whether ~/.albedo-remote/<digest[:16]> is staged; python older than 3.11 is unsupported, a missing bundle is staged (tar over the same stream, unpacked beside the target and moved into place, so a bundle directory that exists is complete). ssh's own failure (exit 255) is needs_auth when its words say a person could get in (permission denied, host key verification, passphrase, keyboard-interactive), otherwise unreachable. answers are cached in the daemon, ready ones for a minute and failures for five seconds; a second caller joins a probe in flight. albedo/harness/ssh.gleam is the gleam face.
• boot: kernel.place probes (or reuses the cached probe) before the bridge starts, so a warmed host boots fast. the port owner runs the bridge command over ssh. a fresh boot's exit 4 reads as "not a folder there", 255 as an ssh failure. run directories live under the remote home, ~/.albedo-remote/run/<kernel id>, recorded absolute.
• a host out of reach is not a kernel gone: only an attach the bridge ends with exit 3 (nothing at the run directory) or a refused attach proves a remote kernel gone. ssh failing (255), any other exit, or a startup timeout while attaching is a dropped connection: the session gets the kernel at once, reattaching, and the port owner keeps attaching in the background, backing off to every 30 s, until the kernel's grace (plus a minute) has surely passed since it lost the connection; then it forgets the record quietly, without reaching for the host. when the probe itself fails at boot, a recorded kernel is attached with commands rebuilt from its run directory (which names the home), so a wifi blip during a daemon restart keeps the namespace. a turn in a session with a kernel on record goes through while its host is unreachable or needs a sign-in, waiting like any reattach. only a fresh boot needs the host now.
• ownership: the pids a remote kernel declares are that host's. the albedo_signal.py ladder for its kernel and job groups runs there over ssh (the signal command, 15 s), never locally; if ssh can't reach the host the targets are left to the kernel's own grace exit, which reaps its jobs on the host, and the error says so. only the bridge (a local ssh client) is ever signalled here. the run directory goes with rm -rf over ssh. os_pid answers nothing for a remote kernel, so the reaper's ps never reads an unrelated local pid.
• turns: a session that holds a kernel (attached, or reattaching through its outbox) never probes on a turn. one that must boot probes for up to 3 s: still warming is left to the boot, which waits for the probe; needs_auth, unreachable or unsupported leave the turn admitted and waiting, blocked with the probe's words as its receipt's blockingReason, retried every 15 s like any blocked input (operations.md).
• the model's remote calls (albedo_plugins/remote.py on the owner side, serve_invoke in the remote kernel): one call per frame, answered by one reference. a result that can cross is inlined; anything else is kept live (LIVE, 64 entries, identity-stable handles) so methods call the real object there. awaiting a reference sends one await frame: the remote side awaits the object only when it is awaitable (a job); a settled non-awaitable object -- a spawned browser handle -- is the answer itself, back as the same reference, so an await can never strand a live process with no handle. arguments cross with the same markers as targets and values: {"__ref__": id} for a live reference, {"pending": id} for a call still in flight (resolved in order), {"__bytes__": b64} for byte data, decoded remotely by albedo_values, so rem.run(..., stdin=b"...") and byte-sized results work. rem.write takes text or bytes; either travels on the stdin of one ssh command, never in the command line. a connection that dies stops the jobs it was mirroring as far as this side can tell: its live jobs leave the /jobs list, and every one no reference here awaited or read wakes the session once, naming the job and "connection lost, exit status unknown" (a busy session is retried, as for a local wake). a reference on a dead connection raises RemoteLost from poll()/exit_code instead of reporting None, which reads as a job that never ends.
• the model gets a context line (python extension, place.gleam): "your python kernel and run jobs execute on chernobog (Linux aarch64)…", with the remote home.

Host probes #

GET /hosts?target={host} reads cached SSH probe state. POST /hosts/{host}/probe starts or joins a bounded probe and returns 202. The client polls the read until it reaches a terminal state. Probe state includes safe diagnostics and authentication instructions when operator sign-in is needed. The same probe supports remote folder gathering. See the HTTP contract for the wire format and workspaces for folder browsing.

signing in #

BatchMode stays on for the daemon. when a remote session's waiting turn is blocked, the chat asks GET /hosts?target={host}; if that says needs_auth and the daemon shares the tui's machine, the chat offers ctrl+l: it runs ssh -M -fN -o ControlPath=<control_path> -o ControlPersist=600 <host> through tea.ExecProcess, so ssh asks in the terminal, then warms the host, and the waiting turn starts on the daemon's next try; the daemon rides that master from then on. with a daemon elsewhere the notice says to run ssh <host> on its machine, which carries the daemon when the user's ssh config multiplexes that host. the offer goes once the turn starts or goes away, or the session moves.

not yet #

• profile output a remote login shell prints reaches the daemon's own stderr log on every remote command.