The detached kernel #
A session's Python kernel runs cells in a process separate from the daemon. The
daemon opens or reattaches to it through
kernel.open. The Python entry point is
albedo_kernel.py, which starts
albedo_cells.main. Kernel namespace
snapshots are described in Python kernel state.
Ownership #
The kernel and its model-facing feature are core harness code under
src/albedo/harness/kernel. Python is not an
extension. Every session runs a kernel and composes the
harness base: the python tool, the cells,
user_shell_output, and session host routes, the /kernel command and its
client bindings, and the remote-place context. No extension selection removes
them, so every extension may rely on the kernel and its bindings. The kernel
tables' transfer declaration is core's too.
The runtime owns the kernel's storage. Before it installs extensions,
runtime.start_with_config creates the
cell journal tables (cells, cell_traces) and then the
link tables. It then records kernels
that an earlier daemon left in this home's run directories. A failed
reconciliation is reported and does not stop the daemon. runtime.cleaners puts
the link cleanup before the extensions' cleanups when a session is deleted. The
cell_images data upgrade is described in
storage migrations.
The modules in that directory divide the work as follows:
kernelopens, executes, upgrades, and snapshots kernels, and reports their jobs and liveness.kernel/linkkeeps kernel records, the outbox, the host-call ledger, and staged kernels.kernel/journalis the cell journal: saved sources, outcomes, and traces.kernel/cell_storageholds the SQL that daemon accounting and deletion run on the journal.kernel/result_viewrenders a stored cell result for a model request.kernel/toolnames the cell tool. Core matches calls againsttool.namefor binding rewrites, progress, repetition, and the composed instructions. Do not spell the name as a literal in core.kernel/baseassembles the harness base thatcomposition.composeputs ahead of every selection. Its owner name stayspythonfor client command ids and saved selections.kernel/python_toolis thepythontool: its schema, its instructions, a call's journaled execution, and the recovery of an interrupted call.kernel/cellsserves thecellsandsessionhost routes.kernel/kernel_commanddeclares/kerneland its client bindings.kernel/placetells the model where a remote session's kernel runs.kernel/transferdeclares the kernel tables for session transfer.daemon/transfer.adapterslists it after the core adapter. Its owner namepythonand its table versions are on the wire, so peers of any version stage its segments; do not rename them.kernel/jobsandkernel/resourcesserve job observation and recorded process ownership to the run and agents extensions.kernel/cell_codecencodes the stored form of a cell's outcome and trace. It reads every shape earlier daemons wrote.kernel/job_windowpages a released kernel's saved job output for the run routes.kernel/driveris the process that owns one kernel: its connection, sequencing, job ownership, probes, and cleanup.kernel/framereads each frame the kernel sends into a typed kind once, in the driver'sread_wire, and names the asks the driver sends. Match onKind, not on the frame'stypetext.kernel/cell_statusis how a cell ended, with its wire spelling. The journal stores its constructor atoms, so do not rename them.kernel/supervisionruns checked process stops, recorded stops, and run-directory discovery.kernel/spareowns spare custodians and the zygote.kernel/launchpicks a spare, the zygote, or a cold start, and holds the cold launcher and the socket relay.kernel/nativebinds the ports, local sockets, and OTP calls those owners share.kernel/recovery,kernel/termination, andkernel/wedgehold policy that the driver calls.kernel/wiredecodes and frames the link's messages.kernel/process_probereads a local kernel's process state and CPU time.kernel/python_treefinds the Python tree a kernel starts from and saves the trees that booted.kernel/environmentbuilds the environment of every program the daemon starts for a kernel or a model tool, and reads the kernel's grace.kernel/signal_helperkeeps one shared signal helper process running while stops keep coming.
Do not move kernel runtime code back into an extension, and do not add kernel housekeeping as extension plugins.
albedo_cells.main imports the plugin modules
with the kernel's own directory first on sys.path, then replaces that entry
with the workspace. A submodule loaded on first use, as
the browser package loads
its own, can import a top-level albedo_* module only if startup already loaded
it. Put a helper that only such code needs under albedo_plugins, which
resolves through the package rather than sys.path, as
proxy and
ws are. Do not put the kernel's
directory back on sys.path to fix a missing module: a workspace file would
then shadow it.
kernel_lazy_imports_test.py
imports every plugin module, then each plugin package's submodules after that
switch.
Execution and results #
kernel.execute and execute_saved send
code and a deadline to the kernel. Code is limited to 1 MiB and the timeout to 1
through 3,600,000 milliseconds. An outcome has an ID, status, text output,
value, truncation flag, images, image errors, and an optional duration. Status
is ok, error, interrupted, or backgrounded.
The interface also reports Unavailable, Busy, Lost, Invalid, and
Detached errors. Detached means a cell exceeded its deadline while the
kernel was unreachable. The cell continues there and its result is journaled
when the kernel returns. A caller must not assume that retrying a detached cell
has no side effects.
Memory cap #
albedo_memory reads
ALBEDO_KERNEL_MEMORY_BYTES, which defaults to 256 MiB. On Linux, its preferred
measurement sums RssAnon, RssShmem, and VmSwap from /proc/self/status.
This counts resident anonymous and shared memory plus private anonymous swap. It
excludes swapped shared memory and resident regular-file mappings. When the
split counters are missing or unreadable, it falls back to the resident set in
/proc/self/statm. Darwin uses ri_phys_footprint through libproc. The
Linux status-field definitions
describe which mappings each counter includes.
guard_memory samples while cells run, at 100
ms intervals. It interrupts cells that pass their threshold and tries to stop
their worker threads. Sampling can overshoot the cap. A cell that starts above
it gets 8 MiB of cleanup slack so it can release retained objects.
albedo_spill.relieve can move large
namespace values to disk and replace them with Paged handles when memory
remains high.
albedo_memory.enforce also sets a soft allocation limit above the starting
allocation size plus the cap and a platform margin, when supported. It uses
RLIMIT_DATA on Linux and RLIMIT_AS on Darwin. This backstop and the sampled
measurement cover different memory; neither is a sandbox.
Thread interruption is best-effort. A thread blocked in a C call receives the
exception only when that call returns, and a thread can catch BaseException.
Under allocation pressure, thread startup can fail before reporting that it
started, leaving Thread.start() waiting. Start workers before beginning a
deliberately allocation-heavy workload. The kernel itself starts no thread per
cell: each cell's deadline and background timers are alarms on the two threads
albedo_alarms starts with the kernel, so a
cell that leaves the process at its limit cannot stop the next cell from
starting. Deadlines have their own thread: a background alarm writes to the
daemon and can block while an attached daemon is not reading, and no deadline
may wait behind that write.
stop_threads can leave a worker to the guard
after one second if the process is back under its cap. If workers remain after
five seconds and keep the process over its cap, the foreground cell requests a
kernel restart. Replacement restores the last saved variables, subject to the
saved-state limits. Check side effects before rerunning a
cell; current objects and background jobs need not survive replacement.
Process jobs #
Use
run(program, *args, cwd=..., env=...)
for external programs. It starts a job immediately and returns a handle. Pass
arguments separately and use cwd for the working directory. For captured text,
await run("git", "status").text() waits for completion and checks the exit
status.
Bind a job before awaiting it when later calls need its handle. Start
independent jobs before waiting for their results. Read bounded previews with
job.tail or use the output-read interface for larger results. Do not treat a
preview as complete output. The job and capture implementations are in
run.py and
albedo_capture.py.
The public kernel/jobs API owns
live and saved job observations. A failed live read falls back to the saved book
only when it is unavailable and the kernel is dead. Missing output or a rejected
request does not trigger that fallback.
kernel/resources provides
recorded kernel-process attribution to the agents extension.
Cells that outlast the foreground timeout (status=backgrounded) also appear in
the job observation (running_jobs, /jobs, and /jobs/recent) with
kind: "cell" and their title comment or first line, expose their live output
through the job-output route without withdrawing their completion notice, and
can be stopped through the job-stop route. Live run jobs started inside a
still-running backgrounded cell carry parent_id naming that cell and promote
to top-level if they outlive it. live_job_count counts top-level items.
Unresponsive kernels #
While a kernel has work, the driver
sends it a {"ping": n} every 5 seconds. Work means a cell someone waits for,
cells the kernel reported live, backgrounded cells, or a hello still owed after
a daemon restart. The link's reader thread answers with {"pong": n} whenever
it can take the GIL. After three unanswered pings,
wedge.classify judges the process
from two samples that
process_probe reads:
asleep: under a twentieth of the window on the CPU.spinning: more CPU than that.stopped: stopped by a signal, or a zombie.gone: the process no longer exists.unresponsive: a remote kernel, or a process that could not be read.
A verdict goes to the log once. It ends nothing by itself, because native code
holding the GIL can still return. A cell's deadline ends the kernel. A kernel
with no waited-for or backgrounded cell that is asleep or stopped is ended
after the idle limit, ALBEDO_IDLE_SECONDS clamped to between 1 second and a
week, 600 seconds by default. A waited-for cell past the background threshold,
ALBEDO_CELL_BACKGROUND_SECONDS with a default of 60, is answered as
backgrounded with the verdict in its output.
A backgrounded cell's deadline is kept by the driver too, because the kernel's own timer thread cannot run while native code holds the GIL. Ten seconds past the deadline the driver interrupts the cell. A cell the driver answered as backgrounded itself gets no such grace. If the cell still runs two seconds after the interrupt, the driver ends the kernel and its jobs. The next cell starts a fresh kernel and restores the variables last saved to disk.
Session layer #
link.gleam stores a session's
kernel record, unacknowledged frames, and host-call ledger. Reattachment resends
pending frames. The ledger distinguishes a fresh call, an answered call, and a
call started by an earlier daemon whose effects are unknown. The latter is not
evidence that the call had no effect.
The outbox retains at most 2,048 frames and 67,108,864 bytes per kernel. Past either limit it drops the oldest frames. The database record also identifies the daemon home that owns the kernel, so a daemon using a copied database does not claim another daemon's processes.
Opening and replacing kernels #
open_inner reconciles staged records and
tries to reattach before starting a fresh kernel. For a local fresh boot, if the
current Python tree fails to start, it can try the last tree that booted
successfully, after recorded cleanup of the failed launch succeeds. A remote
kernel uses a staged bundle and does not have that local fallback. Opening with
a spare also requires spare cleanup to succeed.
python_tree owns the saved
trees under kernel-bundles in the home. Each fresh local kernel that starts
cleanly from the daemon's tree saves a copy, and saving prunes all but the
newest four besides the one just saved. A kernel booting from a saved copy first
claims it with a file under kernel-bundles/claims. The claim lasts until the
daemon has confirmed the kernel and its jobs gone and removes the run directory
(supervision.remove_run_dir).
The kernel's own exit does not release it, because run jobs use the copy's
shims and can outlive the kernel. A kernel whose cleanup is never confirmed
keeps its copy. Pruning spares every claimed copy. Claims and pruning hold the
same lock, so a boot either claims a copy before pruning reads the claims or
finds it gone and fails. Do not delete or hand out saved trees anywhere else: a
fallen-back kernel still imports plugin modules and runs job shims from its copy
after startup.
upgrade snapshots the old namespace,
starts a replacement, and restores state before retiring the old kernel and its
jobs. The returned Carried information records what survived and what did not.
A snapshot is not a guarantee that every Python object or running job can be
carried. See Python kernel state.
runtime/kernels.gleam owns
runtime operations for opening, staging, upgrading, stopping, and reconciling
kernels. The Python interface exposes job inspection, interruption, stopping,
and liveness operations in kernel.gleam.
Lifecycle owners #
launch.gleam selects
reserved-spare adoption, current-tree zygote use, and cold fallback. It also
decides whether the zygote runs, reads a spare's readiness line, and holds the
cold launcher and the socket relay.
spare.gleam owns spare custodians
and the zygote. The driver owns
protocol state, acknowledgments, job ownership, and checked cleanup. These
owners do not introduce a second pool.
The driver is one Gleam process per kernel, addressed by its pid. Callers send
it a typed Request on the subject driver.requests names. Its own timers,
host-call workers, and the launcher send it a Tick on the subject
driver.ticks names. Cell, stop, backstop, and reattach timers carry the
identity of the request or connection they belong to, and the driver ignores a
tick whose identity is no longer current. Probe and acknowledgement ticks are
coalesced with a flag, so at most one is pending at a time. The relay speaks as
a bridge port would, so the driver selects one connection's messages the same
way for either. The driver keeps its last observation in a protected ETS table,
which observation, job_count, live_work, os_pid, and bloated read
without waiting on it.
The driver calls Gleam policy for each decision and encoding:
wire.gleam for link messages,
wedge.gleam and
process_probe.gleam for
liveness verdicts, and recovery.gleam and termination.gleam below.
native.gleam holds the typed
bindings for ports, local gen_tcp sockets in active mode, logging, and process
labels. Put new policy in a Gleam policy module, and keep the bindings in
native.gleam thin.
The test supports under test/harness build a driver state with
driver.detached, run it with driver.serve, and call handle_frame,
host_reply, and record_lost directly. The e2e probes under test/e2e read
the driver's mailbox for its Stop and Rebind requests. Update them with the
request shapes.
A launcher reaching EOF permits a socket connection attempt. Protocol readiness and runtime publication happen afterward. Request cancellation withdraws its waiter while runtime reconciliation retains responsibility for the launch. A late ready kernel is retired. The cold launcher still waits for EOF before its relay handles owner-death messages, so monitor installation does not guarantee cleanup of a launcher permanently blocked before EOF.
recovery.gleam owns attachment
decisions, retry backoff, remote grace, and retry-token checks. The driver
retains timers and connection identities. A broken transport alone does not
prove kernel death. Local process evidence distinguishes a busy kernel from a
stopped or absent process. Repeated connection failures do not extend an already
established remote grace deadline. Replay and host-call uncertainty stay with
the driver and session link. It also bounds the cell claims a turn's opening
waits on after a crash. Every claim shares the deadline that open_claims set.
A claim answered after that deadline journals the ended cell it carries, unless
the cell was backgrounded.
termination.gleam interprets
checked process-group results. Each requested label needs exactly one
confirmation, and every reported group must be gone before cleanup succeeds. The
driver keeps collecting late job ownership during shutdown. A shutdown request
goes to the kernel even when the session link cannot record it. When the owner
dies, its store can no longer record frames, and the kernel still needs the
request to run its plugin cleanup callbacks before the driver signals what
remains. Cleanup that started after the owner died still reports
kernel ownership cleanup was not confirmed, because only the owner can forget
the kernel's record. albedo_signal.py
checks process identities and groups and performs the termination ladder. A
local stop sends its request to the helper that
signal_helper.gleam keeps
running. That helper exits after a minute without requests, and is killed when a
request outlives its wait with nothing else in flight. When no helper can start,
supervision.gleam runs a
one-shot helper. A remote kernel's helper runs on its host over ssh. A malformed
verdict or diagnostic from the helper counts as an unconfirmed target.
Incomplete confirmation or failed record forgetting retains cleanup ownership
for another stop. Runtime failed-stop fencing and cleanup credits remain with
the runtime owner.
A spare handle retains its checked process identity and cleanup callback after custodian death. A handle-local completion marker distinguishes acknowledged adoption or cleanup from disappearance. Failed adoption cleanup keeps the custodian available for stop, and a dead custodian's handle can retry checked cleanup. An uncertain direct zygote fork cannot trigger cold fallback. Before adoption, that fork reports its identity to the driver for startup cleanup and records it through the session link.
Fresh spare kernels #
The runtime kernel owner handles opens that use a spare. albedo_spare.py implements the Python spare path. Keep adoption and cleanup in those owners rather than adding another pool in daemon handlers. See extension lifecycle guidance.
A spare imports some plugin modules before adoption, so the adopting session
only calls their setup. The runtime computes that list once at startup with
selection.preload_modules:
the modules that the harness base and the built-in default extensions declare in
static plugins, plus hook. It passes the list to albedo_spare.py and
albedo_zygote.py as a JSON argument. Do not keep a module list in Python. A
module outside the list loads at adoption, before the workspace enters
sys.path, like a cold kernel's modules. Users' default changes are left out,
so every spare and the one zygote import the same list for the daemon's life.
The list does not cover every shipped module. A zygote exits after a minute
without forks, and from then on each kernel it forked keeps the modules it
imported as private memory. Every plugin module together cost about 3.1 MB of
USS over the bare spare, against 1.1 MB for the default list. Most of the
difference is colab, which pulls in http.client and urllib.request through
proxy. Importing a module can also have effects: cua registers itself as
cua in sys.modules, which would shadow a workspace module of that name in
sessions that never enabled it.
python_modules_test.gleam checks
that every declared and preloaded name is a shipped module.