small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
albedo robot-docs migrations.md
16 kB · 261 lines
Markdown
at main

Storage migrations #

Storage upgrades run at daemon startup before sessions open. Core and extension migrations have separate owners. The extension runtime exposes migrate for extension data migrations.

Startup order #

server.main checks the home's breaking-migration record before anything opens storage. Then server.start calls registry.prepare_storage before serving sessions. Core migrations.run runs image storage, transcript classification, transcript row kinds, session profiles, prompts, prompt context blocks, titles, usage-hour, waiting-input, effort-name, and transcript text upgrades in that order, each declared with its compatibility (see Downgrades). Each result is checked, so an error stops the sequence. The waiting-input upgrade moves inputs from the retired pending_inputs table into the inbox. effort_names renames sessions' saved extra-low effort to minimal, the name types.Effort gives Antigravity's lowest tier; account migration renames a profile's at each startup. The transcript text upgrade creates the text index with every row waiting for it. Startup prints migration <name>: <rows> rows for each core, kernel, and extension migration that rewrote rows.

After the core upgrades, prepare_storage runs the kernel's cell_images upgrade and then the installed extensions' data migrations. session_titles, effort_names and cell_images record their completion in the migrations table so homes that already ran them skip them on later startups. Keep that name if the migration moves again. The kernel's schema is created by the runtime, not by an extension.

Paperclips extension contributes schema migrations alongside its table initializer. Extension-owned schema changes belong with their owner. See extensions and paperclips.

Compatibility and interruption #

session_prompts.run moves older session prompt columns into the prompt table. session_context_blocks.run rewrites saved prompt contexts from rendered inputs to JSON name and body blocks. transcript_classes.run handles older transcript rows. transcript_kinds.run records what each older user row is, in transcript.row_kind. It is the only place that reads the frames earlier daemons wrote for notes, notices, letters and skill activations. Readers ask the row's kind and never match the text. A writer of a user row names its kind. Inspect each migration's transaction and marker logic before changing its restart behavior. These upgrades do not all share one transaction.

Downgrades #

A daemon refuses to start on a home that a newer build upgraded with a breaking migration this build does not know. Additive changes never block a rollback.

Every migration declares a migration.Compatibility:

  • Additive: a build without the migration still reads the home correctly. A new table or index, a nullable or defaulted column, or a backfill of one of those is additive. Nothing is recorded.
  • Breaking: a build without the migration would misread or lose data. Rows moved to another table, a column renamed, dropped, or given a new meaning, and values rewritten into a new format are breaking.

Record #

migration.apply and migration.record write a breaking migration's name to the breaking_migrations table before the migration writes anything. The table is created on first use. A data migration's record commits in its own transaction first, so a run interrupted half way is still recorded. An extension schema migration's record commits in the same transaction as its schema change. Breaking migrations run at every startup and record themselves each time, so the insert is idempotent.

Transfers do not carry breaking_migrations. Each daemon records its own upgrades.

Refusal #

server.main calls registry.check_storage after it takes the home lock and before server.start opens storage. It compares the record with registry.known_migrations: every core and kernel migration name, every migration that the build's declared extensions contain, including quarantined ones, and the retired names. A missing albedo.sqlite is a new home.

The check writes none of the database's files. A read-write connection would checkpoint a crashed daemon's write-ahead log into the database and delete it on close, and plain mode=ro creates missing sidecars or rebuilds the -shm index. migration.unknown picks its options from the files present:

  • No -wal, or an empty one: mode=ro&immutable=1. Everything committed is in the main file. immutable=1 ignores a log, so it is used only when none holds commits. The home lock keeps another daemon from writing one.
  • A -wal with commits and its -shm, as a stopped or crashed daemon leaves them: mode=ro&readonly_shm=1. This reads the log's commits and changes neither file.
  • A -wal with commits but no -shm: SQLite cannot read the log without writing an index. The check fails and startup stops, rather than miss a record that only the log holds.
  • A -wal that exists but cannot be read: the check fails the same way. Only a -wal that is not there at all (enoent) counts as absent.

The home lock keeps other daemons out, but not every program. Another program's last connection can checkpoint and delete the log after the check picks readonly_shm=1 and before it opens, and the open then fails. When the files then call for other options, the check reads again with those, up to three times.

The read is bounded at 30 seconds. A read that takes longer is killed, and the check fails.

The read runs in a process of its own, and the check returns once that process has exited. sqlight.close leaves a connection open until its statements are garbage collected. A read-only connection left open makes SQLite refuse the daemon's own writes to the same file.

migration_test checks that the three files are byte-identical after the check, for a stopped home and for one whose record is only in the log.

When a name is unknown, the daemon prints this line to daemon.log and halts with status 78 through lifecycle.refuse_storage:

This home was upgraded by a newer albedo (<names>); update albedo or restore a backup.

The CLI reports a daemon that exits during startup with the log written since launch, so albedo shows the line. A failure to read the record stops startup too. A login service does not restart the refused daemon. See the daemon lifecycle.

Backups #

The backup the refusal names is taken by the build that runs a breaking migration. When the check passes, registry.check_storage lists the breaking migrations this build declares that the home has not recorded. If there are any and the home already has a record, migration.back_up copies the database whole with VACUUM INTO to backups/albedo-before-breaking-<ms>.sqlite in the home. It then records those names in the same connection. This happens before the runtime opens storage, so the copy predates every write those migrations make. Recording the names with the copy means the next startup does not copy the home again, even when a migration's owner fails to install.

The copy is written as <name>.partial and renamed into place only once it is complete. A failed copy removes its partial file, records nothing, and stops startup. The copy holds secrets such as webhook signing keys, so backups/ is made 0700 and the copy 0600.

After a successful copy, check_storage keeps the newest three finished copies and deletes older ones, along with any .partial file a crash left. Partial files never count toward the three. A home with no record yet is a new home, or one from before the record whose first startup backfills it. Neither is copied.

To roll back across a breaking migration, stop the daemon, put the copy in place of albedo.sqlite, and remove albedo.sqlite-wal and albedo.sqlite-shm. Anything the newer build wrote after the copy is lost.

Declare a migration #

  1. Decide whether the change is additive or breaking. Prefer an additive form, such as a new column the old one is copied into, when it keeps the change useful.
  2. Give the migration a name that is unique across the daemon. Prefix an extension's migration with the extension's name. migration_names_test checks uniqueness.
  3. Put a core migration in migrations.steps with its compatibility. Declare an extension migration as extension.SchemaMigration or extension.DataMigration, each with a name and compatibility. See extension migrations.
  4. Put a breaking change in a declared migration. A table initialiser's rewrite is not recorded.

Never rename a migration or reuse a name. Once a home records a name, every later build must know it. When you remove a breaking migration's declaration, move its name to retired in migrations.gleam. This covers a core step, an extension migration, and every breaking migration of an extension that is removed whole. retired is the one list for every owner.

Homes from before the record #

The first build with the record runs every migration at startup, so it records each breaking migration it knows. Homes it upgraded earlier gain those records then. Every build with the check knows those names, so the backfill refuses no build that has the check.

A build from before the record does not check it. Rolling a home back to such a build is not refused, even when a later build recorded a breaking migration. The refusal protects only rollbacks to builds that include it.

Classification #

Migration Owner Compatibility Reason
image_store core Breaking Transcript rows keep references to images moved into images.
transcript_classes core Additive Fills the new row_class column and adds indexes over it.
session_profiles core Additive A new table, filled once and kept by triggers on rows every build writes.
session_prompts core Breaking Moves the prompt columns out of sessions and drops them.
session_titles core Additive Sets auto_title_attempted, which every build with the column reads the same way.
usage_hours core Additive A new table, filled once and kept by a trigger on rows every build writes.
pending_inputs_inbox core Breaking Moves waiting inputs out of pending_inputs, and later inputs never go there.
transcript_text core Additive A new index, kept by triggers on rows every build writes.
cell_images kernel Breaking Cell outputs keep references to images moved into images.
coding_lcm_name coding-lcm Breaking Renames the lcm extension in settings and session choices.
paperclips_title, paperclips_reply, paperclips_resolution, paperclips_revision paperclips Additive Defaulted columns.
paperclips_scope paperclips Additive Drops an index that a build which wants it creates again.
work_cwd work Additive A defaulted column and an index.
rolling_observation rolling Additive Drops a table that cached each request's observation. An older build creates it again, empty.
schedule_revision schedule Additive Defaulted columns.

Two older rewrites run inside table initialisers and are not recorded. Every build with the check includes both. mail_kinds copies the mail table and rewrites event kinds. The webhooks filed upgrade in the webhook ledger posts inbox letters for deliveries that waited from before the inbox.

downgrade_test.py opens a home this checkout upgraded with the reference build. It also starts this checkout on a home that records an unknown breaking migration and checks the CLI's message and that the database is unchanged.

Foreign-key indexes #

Make each foreign key's child columns the leading columns of an index. For a composite key, the index prefix must contain all child columns. An INTEGER PRIMARY KEY needs no separate index.

Deletion looks up referencing child rows. Without an index, that lookup scans the child table for each deleted parent row and can block the store writer. test_every_foreign_key_column_is_indexed checks this invariant across the schema. Preserve it when adding or changing extension tables as well as core tables.

Migration test inventory #

test/e2e/migrations_test.py exercises startup with older data. Keep an upgrade scenario while its old format remains supported. A fresh database does not exercise that path. The normal test gate is described in test execution.