small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
albedo robot-docs kernel.md
25 kB · 416 lines
Markdown
at main

The detached kernel #

A session's Python kernel runs cells in a process separate from the daemon. The daemon opens or reattaches to it through kernel.open. The Python entry point is albedo_kernel.py, which starts albedo_cells.main. Kernel namespace snapshots are described in Python kernel state.

Ownership #

The kernel and its model-facing feature are core harness code under src/albedo/harness/kernel. Python is not an extension. Every session runs a kernel and composes the harness base: the python tool, the cells, user_shell_output, and session host routes, the /kernel command and its client bindings, and the remote-place context. No extension selection removes them, so every extension may rely on the kernel and its bindings. The kernel tables' transfer declaration is core's too.

The runtime owns the kernel's storage. Before it installs extensions, runtime.start_with_config creates the cell journal tables (cells, cell_traces) and then the link tables. It then records kernels that an earlier daemon left in this home's run directories. A failed reconciliation is reported and does not stop the daemon. runtime.cleaners puts the link cleanup before the extensions' cleanups when a session is deleted. The cell_images data upgrade is described in storage migrations.

The modules in that directory divide the work as follows:

  • kernel opens, executes, upgrades, and snapshots kernels, and reports their jobs and liveness.
  • kernel/link keeps kernel records, the outbox, the host-call ledger, and staged kernels.
  • kernel/journal is the cell journal: saved sources, outcomes, and traces.
  • kernel/cell_storage holds the SQL that daemon accounting and deletion run on the journal.
  • kernel/result_view renders a stored cell result for a model request.
  • kernel/tool names the cell tool. Core matches calls against tool.name for binding rewrites, progress, repetition, and the composed instructions. Do not spell the name as a literal in core.
  • kernel/base assembles the harness base that composition.compose puts ahead of every selection. Its owner name stays python for client command ids and saved selections.
  • kernel/python_tool is the python tool: its schema, its instructions, a call's journaled execution, and the recovery of an interrupted call.
  • kernel/cells serves the cells and session host routes.
  • kernel/kernel_command declares /kernel and its client bindings.
  • kernel/place tells the model where a remote session's kernel runs.
  • kernel/transfer declares the kernel tables for session transfer. daemon/transfer.adapters lists it after the core adapter. Its owner name python and its table versions are on the wire, so peers of any version stage its segments; do not rename them.
  • kernel/jobs and kernel/resources serve job observation and recorded process ownership to the run and agents extensions.
  • kernel/cell_codec encodes the stored form of a cell's outcome and trace. It reads every shape earlier daemons wrote.
  • kernel/job_window pages a released kernel's saved job output for the run routes.
  • kernel/driver is the process that owns one kernel: its connection, sequencing, job ownership, probes, and cleanup.
  • kernel/frame reads each frame the kernel sends into a typed kind once, in the driver's read_wire, and names the asks the driver sends. Match on Kind, not on the frame's type text. kernel/cell_status is how a cell ended, with its wire spelling. The journal stores its constructor atoms, so do not rename them.
  • kernel/supervision runs checked process stops, recorded stops, and run-directory discovery.
  • kernel/spare owns spare custodians and the zygote.
  • kernel/launch picks a spare, the zygote, or a cold start, and holds the cold launcher and the socket relay.
  • kernel/native binds the ports, local sockets, and OTP calls those owners share.
  • kernel/recovery, kernel/termination, and kernel/wedge hold policy that the driver calls.
  • kernel/wire decodes and frames the link's messages.
  • kernel/process_probe reads a local kernel's process state and CPU time.
  • kernel/python_tree finds the Python tree a kernel starts from and saves the trees that booted.
  • kernel/environment builds the environment of every program the daemon starts for a kernel or a model tool, and reads the kernel's grace.
  • kernel/signal_helper keeps one shared signal helper process running while stops keep coming.

Do not move kernel runtime code back into an extension, and do not add kernel housekeeping as extension plugins.

albedo_cells.main imports the plugin modules with the kernel's own directory first on sys.path, then replaces that entry with the workspace. A submodule loaded on first use, as the browser package loads its own, can import a top-level albedo_* module only if startup already loaded it. Put a helper that only such code needs under albedo_plugins, which resolves through the package rather than sys.path, as proxy and ws are. Do not put the kernel's directory back on sys.path to fix a missing module: a workspace file would then shadow it. kernel_lazy_imports_test.py imports every plugin module, then each plugin package's submodules after that switch.

Execution and results #

kernel.execute and execute_saved send code and a deadline to the kernel. Code is limited to 1 MiB and the timeout to 1 through 3,600,000 milliseconds. An outcome has an ID, status, text output, value, truncation flag, images, image errors, and an optional duration. Status is ok, error, interrupted, or backgrounded.

The interface also reports Unavailable, Busy, Lost, Invalid, and Detached errors. Detached means a cell exceeded its deadline while the kernel was unreachable. The cell continues there and its result is journaled when the kernel returns. A caller must not assume that retrying a detached cell has no side effects.

Memory cap #

albedo_memory reads ALBEDO_KERNEL_MEMORY_BYTES, which defaults to 256 MiB. On Linux, its preferred measurement sums RssAnon, RssShmem, and VmSwap from /proc/self/status. This counts resident anonymous and shared memory plus private anonymous swap. It excludes swapped shared memory and resident regular-file mappings. When the split counters are missing or unreadable, it falls back to the resident set in /proc/self/statm. Darwin uses ri_phys_footprint through libproc. The Linux status-field definitions describe which mappings each counter includes.

guard_memory samples while cells run, at 100 ms intervals. It interrupts cells that pass their threshold and tries to stop their worker threads. Sampling can overshoot the cap. A cell that starts above it gets 8 MiB of cleanup slack so it can release retained objects. albedo_spill.relieve can move large namespace values to disk and replace them with Paged handles when memory remains high.

albedo_memory.enforce also sets a soft allocation limit above the starting allocation size plus the cap and a platform margin, when supported. It uses RLIMIT_DATA on Linux and RLIMIT_AS on Darwin. This backstop and the sampled measurement cover different memory; neither is a sandbox.

Thread interruption is best-effort. A thread blocked in a C call receives the exception only when that call returns, and a thread can catch BaseException. Under allocation pressure, thread startup can fail before reporting that it started, leaving Thread.start() waiting. Start workers before beginning a deliberately allocation-heavy workload. The kernel itself starts no thread per cell: each cell's deadline and background timers are alarms on the two threads albedo_alarms starts with the kernel, so a cell that leaves the process at its limit cannot stop the next cell from starting. Deadlines have their own thread: a background alarm writes to the daemon and can block while an attached daemon is not reading, and no deadline may wait behind that write.

stop_threads can leave a worker to the guard after one second if the process is back under its cap. If workers remain after five seconds and keep the process over its cap, the foreground cell requests a kernel restart. Replacement restores the last saved variables, subject to the saved-state limits. Check side effects before rerunning a cell; current objects and background jobs need not survive replacement.

Process jobs #

Use run(program, *args, cwd=..., env=...) for external programs. It starts a job immediately and returns a handle. Pass arguments separately and use cwd for the working directory. For captured text, await run("git", "status").text() waits for completion and checks the exit status.

Bind a job before awaiting it when later calls need its handle. Start independent jobs before waiting for their results. Read bounded previews with job.tail or use the output-read interface for larger results. Do not treat a preview as complete output. The job and capture implementations are in run.py and albedo_capture.py.

The public kernel/jobs API owns live and saved job observations. A failed live read falls back to the saved book only when it is unavailable and the kernel is dead. Missing output or a rejected request does not trigger that fallback. kernel/resources provides recorded kernel-process attribution to the agents extension.

Cells that outlast the foreground timeout (status=backgrounded) also appear in the job observation (running_jobs, /jobs, and /jobs/recent) with kind: "cell" and their title comment or first line, expose their live output through the job-output route without withdrawing their completion notice, and can be stopped through the job-stop route. Live run jobs started inside a still-running backgrounded cell carry parent_id naming that cell and promote to top-level if they outlive it. live_job_count counts top-level items.

Unresponsive kernels #

While a kernel has work, the driver sends it a {"ping": n} every 5 seconds. Work means a cell someone waits for, cells the kernel reported live, backgrounded cells, or a hello still owed after a daemon restart. The link's reader thread answers with {"pong": n} whenever it can take the GIL. After three unanswered pings, wedge.classify judges the process from two samples that process_probe reads:

  • asleep: under a twentieth of the window on the CPU.
  • spinning: more CPU than that.
  • stopped: stopped by a signal, or a zombie.
  • gone: the process no longer exists.
  • unresponsive: a remote kernel, or a process that could not be read.

A verdict goes to the log once. It ends nothing by itself, because native code holding the GIL can still return. A cell's deadline ends the kernel. A kernel with no waited-for or backgrounded cell that is asleep or stopped is ended after the idle limit, ALBEDO_IDLE_SECONDS clamped to between 1 second and a week, 600 seconds by default. A waited-for cell past the background threshold, ALBEDO_CELL_BACKGROUND_SECONDS with a default of 60, is answered as backgrounded with the verdict in its output.

A backgrounded cell's deadline is kept by the driver too, because the kernel's own timer thread cannot run while native code holds the GIL. Ten seconds past the deadline the driver interrupts the cell. A cell the driver answered as backgrounded itself gets no such grace. If the cell still runs two seconds after the interrupt, the driver ends the kernel and its jobs. The next cell starts a fresh kernel and restores the variables last saved to disk.

Session layer #

link.gleam stores a session's kernel record, unacknowledged frames, and host-call ledger. Reattachment resends pending frames. The ledger distinguishes a fresh call, an answered call, and a call started by an earlier daemon whose effects are unknown. The latter is not evidence that the call had no effect.

The outbox retains at most 2,048 frames and 67,108,864 bytes per kernel. Past either limit it drops the oldest frames. The database record also identifies the daemon home that owns the kernel, so a daemon using a copied database does not claim another daemon's processes.

Opening and replacing kernels #

open_inner reconciles staged records and tries to reattach before starting a fresh kernel. For a local fresh boot, if the current Python tree fails to start, it can try the last tree that booted successfully, after recorded cleanup of the failed launch succeeds. A remote kernel uses a staged bundle and does not have that local fallback. Opening with a spare also requires spare cleanup to succeed.

python_tree owns the saved trees under kernel-bundles in the home. Each fresh local kernel that starts cleanly from the daemon's tree saves a copy, and saving prunes all but the newest four besides the one just saved. A kernel booting from a saved copy first claims it with a file under kernel-bundles/claims. The claim lasts until the daemon has confirmed the kernel and its jobs gone and removes the run directory (supervision.remove_run_dir). The kernel's own exit does not release it, because run jobs use the copy's shims and can outlive the kernel. A kernel whose cleanup is never confirmed keeps its copy. Pruning spares every claimed copy. Claims and pruning hold the same lock, so a boot either claims a copy before pruning reads the claims or finds it gone and fails. Do not delete or hand out saved trees anywhere else: a fallen-back kernel still imports plugin modules and runs job shims from its copy after startup.

upgrade snapshots the old namespace, starts a replacement, and restores state before retiring the old kernel and its jobs. The returned Carried information records what survived and what did not. A snapshot is not a guarantee that every Python object or running job can be carried. See Python kernel state.

runtime/kernels.gleam owns runtime operations for opening, staging, upgrading, stopping, and reconciling kernels. The Python interface exposes job inspection, interruption, stopping, and liveness operations in kernel.gleam.

Lifecycle owners #

launch.gleam selects reserved-spare adoption, current-tree zygote use, and cold fallback. It also decides whether the zygote runs, reads a spare's readiness line, and holds the cold launcher and the socket relay. spare.gleam owns spare custodians and the zygote. The driver owns protocol state, acknowledgments, job ownership, and checked cleanup. These owners do not introduce a second pool.

The driver is one Gleam process per kernel, addressed by its pid. Callers send it a typed Request on the subject driver.requests names. Its own timers, host-call workers, and the launcher send it a Tick on the subject driver.ticks names. Cell, stop, backstop, and reattach timers carry the identity of the request or connection they belong to, and the driver ignores a tick whose identity is no longer current. Probe and acknowledgement ticks are coalesced with a flag, so at most one is pending at a time. The relay speaks as a bridge port would, so the driver selects one connection's messages the same way for either. The driver keeps its last observation in a protected ETS table, which observation, job_count, live_work, os_pid, and bloated read without waiting on it.

The driver calls Gleam policy for each decision and encoding: wire.gleam for link messages, wedge.gleam and process_probe.gleam for liveness verdicts, and recovery.gleam and termination.gleam below. native.gleam holds the typed bindings for ports, local gen_tcp sockets in active mode, logging, and process labels. Put new policy in a Gleam policy module, and keep the bindings in native.gleam thin.

The test supports under test/harness build a driver state with driver.detached, run it with driver.serve, and call handle_frame, host_reply, and record_lost directly. The e2e probes under test/e2e read the driver's mailbox for its Stop and Rebind requests. Update them with the request shapes.

A launcher reaching EOF permits a socket connection attempt. Protocol readiness and runtime publication happen afterward. Request cancellation withdraws its waiter while runtime reconciliation retains responsibility for the launch. A late ready kernel is retired. The cold launcher still waits for EOF before its relay handles owner-death messages, so monitor installation does not guarantee cleanup of a launcher permanently blocked before EOF.

recovery.gleam owns attachment decisions, retry backoff, remote grace, and retry-token checks. The driver retains timers and connection identities. A broken transport alone does not prove kernel death. Local process evidence distinguishes a busy kernel from a stopped or absent process. Repeated connection failures do not extend an already established remote grace deadline. Replay and host-call uncertainty stay with the driver and session link. It also bounds the cell claims a turn's opening waits on after a crash. Every claim shares the deadline that open_claims set. A claim answered after that deadline journals the ended cell it carries, unless the cell was backgrounded.

termination.gleam interprets checked process-group results. Each requested label needs exactly one confirmation, and every reported group must be gone before cleanup succeeds. The driver keeps collecting late job ownership during shutdown. A shutdown request goes to the kernel even when the session link cannot record it. When the owner dies, its store can no longer record frames, and the kernel still needs the request to run its plugin cleanup callbacks before the driver signals what remains. Cleanup that started after the owner died still reports kernel ownership cleanup was not confirmed, because only the owner can forget the kernel's record. albedo_signal.py checks process identities and groups and performs the termination ladder. A local stop sends its request to the helper that signal_helper.gleam keeps running. That helper exits after a minute without requests, and is killed when a request outlives its wait with nothing else in flight. When no helper can start, supervision.gleam runs a one-shot helper. A remote kernel's helper runs on its host over ssh. A malformed verdict or diagnostic from the helper counts as an unconfirmed target. Incomplete confirmation or failed record forgetting retains cleanup ownership for another stop. Runtime failed-stop fencing and cleanup credits remain with the runtime owner.

A spare handle retains its checked process identity and cleanup callback after custodian death. A handle-local completion marker distinguishes acknowledged adoption or cleanup from disappearance. Failed adoption cleanup keeps the custodian available for stop, and a dead custodian's handle can retry checked cleanup. An uncertain direct zygote fork cannot trigger cold fallback. Before adoption, that fork reports its identity to the driver for startup cleanup and records it through the session link.

Fresh spare kernels #

The runtime kernel owner handles opens that use a spare. albedo_spare.py implements the Python spare path. Keep adoption and cleanup in those owners rather than adding another pool in daemon handlers. See extension lifecycle guidance.

A spare imports some plugin modules before adoption, so the adopting session only calls their setup. The runtime computes that list once at startup with selection.preload_modules: the modules that the harness base and the built-in default extensions declare in static plugins, plus hook. It passes the list to albedo_spare.py and albedo_zygote.py as a JSON argument. Do not keep a module list in Python. A module outside the list loads at adoption, before the workspace enters sys.path, like a cold kernel's modules. Users' default changes are left out, so every spare and the one zygote import the same list for the daemon's life.

The list does not cover every shipped module. A zygote exits after a minute without forks, and from then on each kernel it forked keeps the modules it imported as private memory. Every plugin module together cost about 3.1 MB of USS over the bare spare, against 1.1 MB for the default list. Most of the difference is colab, which pulls in http.client and urllib.request through proxy. Importing a module can also have effects: cua registers itself as cua in sys.modules, which would shadow a workspace module of that name in sessions that never enabled it. python_modules_test.gleam checks that every declared and preloaded name is a shipped module.