# Storage migrations Storage upgrades run at daemon startup before sessions open. Core and extension migrations have separate owners. The [extension runtime](../src/albedo/harness/runtime.gleam) exposes `migrate` for extension data migrations. ## Startup order [server.main](../src/albedo/daemon/server.gleam) checks the home's [breaking-migration record](#downgrades) before anything opens storage. Then `server.start` calls `registry.prepare_storage` before serving sessions. [Core migrations.run](../src/albedo/daemon/migrations.gleam) runs image storage, transcript classification, transcript row kinds, session profiles, prompts, prompt context blocks, titles, usage-hour, waiting-input, effort-name, and transcript text upgrades in that order, each declared with its compatibility (see [Downgrades](#downgrades)). Each result is checked, so an error stops the sequence. The waiting-input upgrade moves inputs from the retired `pending_inputs` table into the [inbox](inbox.md). [`effort_names`](../src/albedo/daemon/migrations/effort_names.gleam) renames sessions' saved `extra-low` effort to `minimal`, the name [`types.Effort`](../src/albedo/openai_api/types.gleam) gives Antigravity's lowest tier; [account migration](../src/albedo/harness/account_migration.gleam) renames a profile's at each startup. The transcript text upgrade creates the [text index](agent-sessions.md#messages-and-resources) with every row waiting for it. Startup prints `migration : rows` for each core, kernel, and extension migration that rewrote rows. After the core upgrades, `prepare_storage` runs the kernel's [`cell_images`](../src/albedo/harness/kernel/migrations/cell_images.gleam) upgrade and then the installed extensions' data migrations. `session_titles`, `effort_names` and `cell_images` record their completion in the `migrations` table so homes that already ran them skip them on later startups. Keep that name if the migration moves again. The kernel's schema is created by the [runtime](kernel.md#ownership), not by an extension. [Paperclips extension](../src/albedo/harness/extensions/paperclips/extension.gleam) contributes schema migrations alongside its table initializer. Extension-owned schema changes belong with their owner. See [extensions](extensions.md) and [paperclips](paperclips.md). ## Compatibility and interruption [session_prompts.run](../src/albedo/daemon/migrations/session_prompts.gleam) moves older session prompt columns into the prompt table. [session_context_blocks.run](../src/albedo/daemon/migrations/session_context_blocks.gleam) rewrites saved prompt contexts from rendered inputs to JSON name and body blocks. [transcript_classes.run](../src/albedo/daemon/migrations/transcript_classes.gleam) handles older transcript rows. [transcript_kinds.run](../src/albedo/daemon/migrations/transcript_kinds.gleam) records what each older user row is, in `transcript.row_kind`. It is the only place that reads the frames earlier daemons wrote for notes, notices, letters and skill activations. Readers ask the row's kind and never match the text. A writer of a user row names its kind. Inspect each migration's transaction and marker logic before changing its restart behavior. These upgrades do not all share one transaction. ## Downgrades A daemon refuses to start on a home that a newer build upgraded with a breaking migration this build does not know. Additive changes never block a rollback. Every migration declares a [`migration.Compatibility`](../src/albedo/harness/sdk/migration.gleam): - `Additive`: a build without the migration still reads the home correctly. A new table or index, a nullable or defaulted column, or a backfill of one of those is additive. Nothing is recorded. - `Breaking`: a build without the migration would misread or lose data. Rows moved to another table, a column renamed, dropped, or given a new meaning, and values rewritten into a new format are breaking. ### Record `migration.apply` and `migration.record` write a breaking migration's name to the `breaking_migrations` table before the migration writes anything. The table is created on first use. A data migration's record commits in its own transaction first, so a run interrupted half way is still recorded. An extension schema migration's record commits in the same transaction as its schema change. Breaking migrations run at every startup and record themselves each time, so the insert is idempotent. Transfers do not carry `breaking_migrations`. Each daemon records its own upgrades. ### Refusal `server.main` calls `registry.check_storage` after it takes the home lock and before `server.start` opens storage. It compares the record with `registry.known_migrations`: every core and kernel migration name, every migration that the build's declared extensions contain, including quarantined ones, and the `retired` names. A missing `albedo.sqlite` is a new home. The check writes none of the database's files. A read-write connection would checkpoint a crashed daemon's write-ahead log into the database and delete it on close, and plain `mode=ro` creates missing sidecars or rebuilds the `-shm` index. `migration.unknown` picks its options from the files present: - No `-wal`, or an empty one: `mode=ro&immutable=1`. Everything committed is in the main file. `immutable=1` ignores a log, so it is used only when none holds commits. The home lock keeps another daemon from writing one. - A `-wal` with commits and its `-shm`, as a stopped or crashed daemon leaves them: `mode=ro&readonly_shm=1`. This reads the log's commits and changes neither file. - A `-wal` with commits but no `-shm`: SQLite cannot read the log without writing an index. The check fails and startup stops, rather than miss a record that only the log holds. - A `-wal` that exists but cannot be read: the check fails the same way. Only a `-wal` that is not there at all (`enoent`) counts as absent. The home lock keeps other daemons out, but not every program. Another program's last connection can checkpoint and delete the log after the check picks `readonly_shm=1` and before it opens, and the open then fails. When the files then call for other options, the check reads again with those, up to three times. The read is bounded at 30 seconds. A read that takes longer is killed, and the check fails. The read runs in a process of its own, and the check returns once that process has exited. `sqlight.close` leaves a connection open until its statements are garbage collected. A read-only connection left open makes SQLite refuse the daemon's own writes to the same file. [`migration_test`](../test/harness/migration_test.gleam) checks that the three files are byte-identical after the check, for a stopped home and for one whose record is only in the log. When a name is unknown, the daemon prints this line to `daemon.log` and halts with status 78 through `lifecycle.refuse_storage`: ```text This home was upgraded by a newer albedo (); update albedo or restore a backup. ``` The CLI reports a daemon that exits during startup with the log written since launch, so `albedo` shows the line. A failure to read the record stops startup too. A login service does not restart the refused daemon. See [the daemon lifecycle](../docs/daemon-lifecycle.md). ### Backups The backup the refusal names is taken by the build that runs a breaking migration. When the check passes, `registry.check_storage` lists the breaking migrations this build declares that the home has not recorded. If there are any and the home already has a record, `migration.back_up` copies the database whole with `VACUUM INTO` to `backups/albedo-before-breaking-.sqlite` in the home. It then records those names in the same connection. This happens before the runtime opens storage, so the copy predates every write those migrations make. Recording the names with the copy means the next startup does not copy the home again, even when a migration's owner fails to install. The copy is written as `.partial` and renamed into place only once it is complete. A failed copy removes its partial file, records nothing, and stops startup. The copy holds secrets such as webhook signing keys, so `backups/` is made 0700 and the copy 0600. After a successful copy, `check_storage` keeps the newest three finished copies and deletes older ones, along with any `.partial` file a crash left. Partial files never count toward the three. A home with no record yet is a new home, or one from before the record whose first startup backfills it. Neither is copied. To roll back across a breaking migration, stop the daemon, put the copy in place of `albedo.sqlite`, and remove `albedo.sqlite-wal` and `albedo.sqlite-shm`. Anything the newer build wrote after the copy is lost. ### Declare a migration 1. Decide whether the change is additive or breaking. Prefer an additive form, such as a new column the old one is copied into, when it keeps the change useful. 2. Give the migration a name that is unique across the daemon. Prefix an extension's migration with the extension's name. [`migration_names_test`](../test/daemon/migration_names_test.gleam) checks uniqueness. 3. Put a core migration in `migrations.steps` with its compatibility. Declare an extension migration as `extension.SchemaMigration` or `extension.DataMigration`, each with a name and compatibility. See [extension migrations](extensions.md#migrations). 4. Put a breaking change in a declared migration. A table initialiser's rewrite is not recorded. Never rename a migration or reuse a name. Once a home records a name, every later build must know it. When you remove a breaking migration's declaration, move its name to `retired` in [migrations.gleam](../src/albedo/daemon/migrations.gleam). This covers a core step, an extension migration, and every breaking migration of an extension that is removed whole. `retired` is the one list for every owner. ### Homes from before the record The first build with the record runs every migration at startup, so it records each breaking migration it knows. Homes it upgraded earlier gain those records then. Every build with the check knows those names, so the backfill refuses no build that has the check. A build from before the record does not check it. Rolling a home back to such a build is not refused, even when a later build recorded a breaking migration. The refusal protects only rollbacks to builds that include it. ### Classification | Migration | Owner | Compatibility | Reason | | -------------------------------------------------------------------------------------- | ---------- | ------------- | --------------------------------------------------------------------------------------------- | | `image_store` | core | Breaking | Transcript rows keep references to images moved into `images`. | | `transcript_classes` | core | Additive | Fills the new `row_class` column and adds indexes over it. | | `session_profiles` | core | Additive | A new table, filled once and kept by triggers on rows every build writes. | | `session_prompts` | core | Breaking | Moves the prompt columns out of `sessions` and drops them. | | `session_titles` | core | Additive | Sets `auto_title_attempted`, which every build with the column reads the same way. | | `usage_hours` | core | Additive | A new table, filled once and kept by a trigger on rows every build writes. | | `pending_inputs_inbox` | core | Breaking | Moves waiting inputs out of `pending_inputs`, and later inputs never go there. | | `transcript_text` | core | Additive | A new index, kept by triggers on rows every build writes. | | `cell_images` | kernel | Breaking | Cell outputs keep references to images moved into `images`. | | `coding_lcm_name` | coding-lcm | Breaking | Renames the `lcm` extension in settings and session choices. | | `paperclips_title`, `paperclips_reply`, `paperclips_resolution`, `paperclips_revision` | paperclips | Additive | Defaulted columns. | | `paperclips_scope` | paperclips | Additive | Drops an index that a build which wants it creates again. | | `work_cwd` | work | Additive | A defaulted column and an index. | | `rolling_observation` | rolling | Additive | Drops a table that cached each request's observation. An older build creates it again, empty. | | `schedule_revision` | schedule | Additive | Defaulted columns. | Two older rewrites run inside table initialisers and are not recorded. Every build with the check includes both. [`mail_kinds`](../src/albedo/daemon/migrations/mail_kinds.gleam) copies the mail table and rewrites event kinds. The webhooks `filed` upgrade in [the webhook ledger](../src/albedo/harness/extensions/webhooks/ledger.gleam) posts inbox letters for deliveries that waited from before the inbox. [`downgrade_test.py`](../test/e2e/downgrade_test.py) opens a home this checkout upgraded with the [reference build](testing.md#mixed-version-moves). It also starts this checkout on a home that records an unknown breaking migration and checks the CLI's message and that the database is unchanged. ## Foreign-key indexes Make each foreign key's child columns the leading columns of an index. For a composite key, the index prefix must contain all child columns. An `INTEGER PRIMARY KEY` needs no separate index. Deletion looks up referencing child rows. Without an index, that lookup scans the child table for each deleted parent row and can block the store writer. [`test_every_foreign_key_column_is_indexed`](../test/e2e/session_delete_test.py) checks this invariant across the schema. Preserve it when adding or changing extension tables as well as core tables. ## Migration test inventory [test/e2e/migrations_test.py](../test/e2e/migrations_test.py) exercises startup with older data. Keep an upgrade scenario while its old format remains supported. A fresh database does not exercise that path. The normal test gate is described in [test execution](testing.md). ## Related pages - [All topics](README.md) - [Change guidance](contributing.md) - [Durable operation admission](operations.md) - [Test execution](testing.md) - [The detached kernel](kernel.md) - [Extension migrations](extensions.md#migrations) - [Moves between versions](workspaces.md#moves-between-versions)