Storage migrations #
Storage upgrades run at daemon startup before sessions open. Core and extension
migrations have separate owners. The
extension runtime exposes migrate for
extension data migrations.
Startup order #
server.main checks the home's
breaking-migration record before anything opens storage. Then
server.start calls registry.prepare_storage before serving sessions.
Core migrations.run runs image storage,
transcript classification, transcript row kinds, session profiles, prompts,
prompt context blocks, titles, usage-hour, waiting-input, effort-name, and
transcript text upgrades in that order, each declared with its compatibility
(see Downgrades). Each result is checked, so an error stops the
sequence. The waiting-input upgrade moves inputs from the retired
pending_inputs table into the inbox.
effort_names renames
sessions' saved extra-low effort to minimal, the name
types.Effort gives Antigravity's
lowest tier; account migration
renames a profile's at each startup. The transcript text upgrade creates the
text index with every row waiting
for it. Startup prints migration <name>: <rows> rows for each core, kernel,
and extension migration that rewrote rows.
After the core upgrades, prepare_storage runs the kernel's
cell_images
upgrade and then the installed extensions' data migrations. session_titles,
effort_names and cell_images record their completion in the migrations
table so homes that already ran them skip them on later startups. Keep that name
if the migration moves again. The kernel's schema is created by the
runtime, not by an extension.
Paperclips extension contributes schema migrations alongside its table initializer. Extension-owned schema changes belong with their owner. See extensions and paperclips.
Compatibility and interruption #
session_prompts.run
moves older session prompt columns into the prompt table.
session_context_blocks.run
rewrites saved prompt contexts from rendered inputs to JSON name and body
blocks.
transcript_classes.run
handles older transcript rows.
transcript_kinds.run
records what each older user row is, in transcript.row_kind. It is the only
place that reads the frames earlier daemons wrote for notes, notices, letters
and skill activations. Readers ask the row's kind and never match the text. A
writer of a user row names its kind. Inspect each migration's transaction and
marker logic before changing its restart behavior. These upgrades do not all
share one transaction.
Downgrades #
A daemon refuses to start on a home that a newer build upgraded with a breaking migration this build does not know. Additive changes never block a rollback.
Every migration declares a
migration.Compatibility:
Additive: a build without the migration still reads the home correctly. A new table or index, a nullable or defaulted column, or a backfill of one of those is additive. Nothing is recorded.Breaking: a build without the migration would misread or lose data. Rows moved to another table, a column renamed, dropped, or given a new meaning, and values rewritten into a new format are breaking.
Record #
migration.apply and migration.record write a breaking migration's name to
the breaking_migrations table before the migration writes anything. The table
is created on first use. A data migration's record commits in its own
transaction first, so a run interrupted half way is still recorded. An extension
schema migration's record commits in the same transaction as its schema change.
Breaking migrations run at every startup and record themselves each time, so the
insert is idempotent.
Transfers do not carry breaking_migrations. Each daemon records its own
upgrades.
Refusal #
server.main calls registry.check_storage after it takes the home lock and
before server.start opens storage. It compares the record with
registry.known_migrations: every core and kernel migration name, every
migration that the build's declared extensions contain, including quarantined
ones, and the retired names. A missing albedo.sqlite is a new home.
The check writes none of the database's files. A read-write connection would
checkpoint a crashed daemon's write-ahead log into the database and delete it on
close, and plain mode=ro creates missing sidecars or rebuilds the -shm
index. migration.unknown picks its options from the files present:
- No
-wal, or an empty one:mode=ro&immutable=1. Everything committed is in the main file.immutable=1ignores a log, so it is used only when none holds commits. The home lock keeps another daemon from writing one. - A
-walwith commits and its-shm, as a stopped or crashed daemon leaves them:mode=ro&readonly_shm=1. This reads the log's commits and changes neither file. - A
-walwith commits but no-shm: SQLite cannot read the log without writing an index. The check fails and startup stops, rather than miss a record that only the log holds. - A
-walthat exists but cannot be read: the check fails the same way. Only a-walthat is not there at all (enoent) counts as absent.
The home lock keeps other daemons out, but not every program. Another program's
last connection can checkpoint and delete the log after the check picks
readonly_shm=1 and before it opens, and the open then fails. When the files
then call for other options, the check reads again with those, up to three
times.
The read is bounded at 30 seconds. A read that takes longer is killed, and the check fails.
The read runs in a process of its own, and the check returns once that process
has exited. sqlight.close leaves a connection open until its statements are
garbage collected. A read-only connection left open makes SQLite refuse the
daemon's own writes to the same file.
migration_test checks that the three
files are byte-identical after the check, for a stopped home and for one whose
record is only in the log.
When a name is unknown, the daemon prints this line to daemon.log and halts
with status 78 through lifecycle.refuse_storage:
This home was upgraded by a newer albedo (<names>); update albedo or restore a backup.
The CLI reports a daemon that exits during startup with the log written since
launch, so albedo shows the line. A failure to read the record stops startup
too. A login service does not restart the refused daemon. See
the daemon lifecycle.
Backups #
The backup the refusal names is taken by the build that runs a breaking
migration. When the check passes, registry.check_storage lists the breaking
migrations this build declares that the home has not recorded. If there are any
and the home already has a record, migration.back_up copies the database whole
with VACUUM INTO to backups/albedo-before-breaking-<ms>.sqlite in the home.
It then records those names in the same connection. This happens before the
runtime opens storage, so the copy predates every write those migrations make.
Recording the names with the copy means the next startup does not copy the home
again, even when a migration's owner fails to install.
The copy is written as <name>.partial and renamed into place only once it is
complete. A failed copy removes its partial file, records nothing, and stops
startup. The copy holds secrets such as webhook signing keys, so backups/ is
made 0700 and the copy 0600.
After a successful copy, check_storage keeps the newest three finished copies
and deletes older ones, along with any .partial file a crash left. Partial
files never count toward the three. A home with no record yet is a new home, or
one from before the record whose first startup backfills it. Neither is copied.
To roll back across a breaking migration, stop the daemon, put the copy in place
of albedo.sqlite, and remove albedo.sqlite-wal and albedo.sqlite-shm.
Anything the newer build wrote after the copy is lost.
Declare a migration #
- Decide whether the change is additive or breaking. Prefer an additive form, such as a new column the old one is copied into, when it keeps the change useful.
- Give the migration a name that is unique across the daemon. Prefix an
extension's migration with the extension's name.
migration_names_testchecks uniqueness. - Put a core migration in
migrations.stepswith its compatibility. Declare an extension migration asextension.SchemaMigrationorextension.DataMigration, each with a name and compatibility. See extension migrations. - Put a breaking change in a declared migration. A table initialiser's rewrite is not recorded.
Never rename a migration or reuse a name. Once a home records a name, every
later build must know it. When you remove a breaking migration's declaration,
move its name to retired in
migrations.gleam. This covers a core
step, an extension migration, and every breaking migration of an extension that
is removed whole. retired is the one list for every owner.
Homes from before the record #
The first build with the record runs every migration at startup, so it records each breaking migration it knows. Homes it upgraded earlier gain those records then. Every build with the check knows those names, so the backfill refuses no build that has the check.
A build from before the record does not check it. Rolling a home back to such a build is not refused, even when a later build recorded a breaking migration. The refusal protects only rollbacks to builds that include it.
Classification #
| Migration | Owner | Compatibility | Reason |
|---|---|---|---|
image_store |
core | Breaking | Transcript rows keep references to images moved into images. |
transcript_classes |
core | Additive | Fills the new row_class column and adds indexes over it. |
session_profiles |
core | Additive | A new table, filled once and kept by triggers on rows every build writes. |
session_prompts |
core | Breaking | Moves the prompt columns out of sessions and drops them. |
session_titles |
core | Additive | Sets auto_title_attempted, which every build with the column reads the same way. |
usage_hours |
core | Additive | A new table, filled once and kept by a trigger on rows every build writes. |
pending_inputs_inbox |
core | Breaking | Moves waiting inputs out of pending_inputs, and later inputs never go there. |
transcript_text |
core | Additive | A new index, kept by triggers on rows every build writes. |
cell_images |
kernel | Breaking | Cell outputs keep references to images moved into images. |
coding_lcm_name |
coding-lcm | Breaking | Renames the lcm extension in settings and session choices. |
paperclips_title, paperclips_reply, paperclips_resolution, paperclips_revision |
paperclips | Additive | Defaulted columns. |
paperclips_scope |
paperclips | Additive | Drops an index that a build which wants it creates again. |
work_cwd |
work | Additive | A defaulted column and an index. |
rolling_observation |
rolling | Additive | Drops a table that cached each request's observation. An older build creates it again, empty. |
schedule_revision |
schedule | Additive | Defaulted columns. |
Two older rewrites run inside table initialisers and are not recorded. Every
build with the check includes both.
mail_kinds copies the mail
table and rewrites event kinds. The webhooks filed upgrade in
the webhook ledger
posts inbox letters for deliveries that waited from before the inbox.
downgrade_test.py opens a home this checkout
upgraded with the reference build. It also
starts this checkout on a home that records an unknown breaking migration and
checks the CLI's message and that the database is unchanged.
Foreign-key indexes #
Make each foreign key's child columns the leading columns of an index. For a
composite key, the index prefix must contain all child columns. An
INTEGER PRIMARY KEY needs no separate index.
Deletion looks up referencing child rows. Without an index, that lookup scans
the child table for each deleted parent row and can block the store writer.
test_every_foreign_key_column_is_indexed
checks this invariant across the schema. Preserve it when adding or changing
extension tables as well as core tables.
Migration test inventory #
test/e2e/migrations_test.py exercises startup with older data. Keep an upgrade scenario while its old format remains supported. A fresh database does not exercise that path. The normal test gate is described in test execution.