diff --git a/docs/operations.md b/docs/operations.md index 053b129..3711acf 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -113,9 +113,10 @@ sudo docker compose exec caddy wget -qO- http://index:3002/health | jq '{agents, `backfilling` is `true` until the listing is read, and a server that answers `429` is read again in rounds. The board fills as accounts are read. -After an instance is replaced, the identity log starts empty and the relay -starts with no hosts. Restore the log from the daily backup before the index -runs for long: +A replaced instance is handed the data volume, and starts from what it +holds. When the volume itself was lost, and no snapshot is newer than the +daily backup of the identity log, restore the log from that backup before the +index runs for long: ```sh . /opt/vibescrobble/static.env @@ -136,6 +137,59 @@ curl -X POST https://relay.vibescrobble.com/xrpc/com.atproto.sync.requestCrawl \ Accounts that reach the index through Bluesky's Jetstream are not on the relay's listing, so they return as each posts. +## Restore the data volume from a snapshot + +The data volume (`tofu -chdir=infra output -raw data_volume`) is snapshotted +daily, and the newest `snapshots_kept` (7 unless set) are kept. A restore is +a new volume made from a snapshot, put in place of the old one: the board +goes back to the day of the snapshot, and the relay and the index read what +was written since. + +**On the laptop**, find the snapshot: + +```sh +export AWS_PROFILE=jmm-did-bot-admin +aws ec2 describe-snapshots --owner-ids self --filters Name=tag:Snapshots,Values=vibescrobble-data \ + --query 'reverse(sort_by(Snapshots,&StartTime))[].[SnapshotId,StartTime,State]' --output table +``` + +Make a volume from it in the same zone (`availability_zone`, `us-east-1a` +unless set), encrypted, and tagged as the snapshot policy looks for: + +```sh +aws ec2 create-volume --availability-zone us-east-1a --volume-type gp3 --encrypted \ + --snapshot-id snap-... \ + --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=vibescrobble-data},{Key=Snapshots,Value=vibescrobble-data}]' +``` + +Stop the instance, so nothing is writing, and detach the old volume: + +```sh +instance="$(tofu -chdir=infra output -raw instance_id)" +aws ec2 stop-instances --instance-ids "$instance" +aws ec2 wait instance-stopped --instance-ids "$instance" +aws ec2 detach-volume --volume-id "$(tofu -chdir=infra output -raw data_volume)" +``` + +Then put the new volume in state in place of the old one, start the +instance, and deploy: + +```sh +tofu -chdir=infra state rm aws_volume_attachment.data aws_ebs_volume.data +tofu -chdir=infra import aws_ebs_volume.data vol-... +aws ec2 start-instances --instance-ids "$instance" +scripts/deploy.sh +``` + +The instance boots with no volume, so docker does not start and nothing is +brought up empty. The deploy attaches the new volume and writes its id into +the release parameter; activation starts docker, which mounts the volume, a +copy of the old one down to its filesystem's id. It is already copied to, so +nothing is copied, and the stack starts from the snapshot. The old volume is +no longer in state, so `prevent_destroy` does not cover it: delete it by hand +once the board is back. To restore in another zone, set `availability_zone` +to it as well, which replaces the subnet and the instance. + ## Check the alarms ```sh diff --git a/docs/state.md b/docs/state.md index 574c666..18d0f3e 100644 --- a/docs/state.md +++ b/docs/state.md @@ -6,7 +6,10 @@ page is what each one is, what a restart reads back from it, and what is read again from the relay and the repositories. On vibescrobble.com the files are in `/opt/vibescrobble/data` on the instance, -which the index sees as `/data`. +which the index sees as `/data`. That directory is bind-mounted from the data +volume, an EBS volume mounted at `/var/lib/vibescrobble` that outlives the +instance (`infra/disk.tf`), and snapshotted daily. A replaced instance is +handed the same volume, so it starts from these files as a restart does. | what | where | written | a restart | |---|---|---|---| @@ -111,8 +114,18 @@ again. Readers' cursors, which carry the same run, are told the same. ## Beside the index -The relay's cursors are in the `relay-data` volume, Jetstream's window is in -`/opt/vibescrobble/jetstream`, and Caddy's certificates are in `caddy-data`. -A replaced instance starts all three empty. -[operations.md](operations.md#recover-from-a-cold-start) says what to -do then. +The relay's hosts and cursors are in the `relay-data` docker volume, and +Caddy's certificates and its config in `caddy-data` and `caddy-config`. Each +is bind-mounted from the data volume too, under +`/var/lib/vibescrobble/volumes/`, so a replaced instance keeps them. +Jetstream's window, in `/opt/vibescrobble/jetstream`, is on the root volume: +it is only a window, and a replaced instance starts it empty, so the index's +saved cursor is past its start and the relay is read again. + +The data volume is mounted by the release's `data-disk.sh`, which every +activation runs: it formats the volume only when it is blank, copies the root +volume's state onto it the first time, and brings nothing up while it is not +mounted. Docker is made to require the mounts, so after a reboot without the +volume no container starts on an empty directory. +[operations.md](operations.md#restore-the-data-volume-from-a-snapshot) says +how to restore a snapshot. diff --git a/infra/README.md b/infra/README.md index 2143f33..7a2625f 100644 --- a/infra/README.md +++ b/infra/README.md @@ -96,8 +96,9 @@ later in a journal. Because the activation script ships with the release, a release that needs to come up differently changes that file and not the machine. User data is for what does not change — Docker, swap, the CloudWatch agent, the poller — and a -change to it replaces the instance, which is a certificate re-issued and the -relay's cursors gone. A deploy never touches it. +change to it replaces the instance. A deploy never touches it. What the board +keeps is on the data volume (`disk.tf`), which the new instance is handed, so a +replacement costs a few minutes, the stack's restart and Jetstream's window. ## How a deploy gets a binary onto an arm64 machine @@ -330,15 +331,17 @@ is whether it is connected to anything. ## Choices worth knowing about -**Almost nothing here is worth backing up.** Jetstream's window, the relay's -cursors and the view itself are all rebuildable or explicitly ephemeral. -[docs/state.md](../docs/state.md) says what each file holds. The -exception is `identities.jsonl`: a deleted agent's DID document stops -resolving forever, so that log is the only remaining record of who wrote an -hour-old scrobble. A systemd timer copies it to S3 daily. Token burn's -rollups, `burn.json` and its journal beside the log in `/data`, cannot be -rebuilt from the network either; they survive a restart and a reboot, and are -not backed up. +**State is on a disk of its own.** The index's files, the relay's database +and Caddy's certificates are on the data volume (`disk.tf`), which outlives +the instance and is snapshotted daily by Data Lifecycle Manager. +[docs/state.md](../docs/state.md) says what each file holds. Most of it +could be read again from the network, slowly; `identities.jsonl` and +`groups.json` could not: a deleted agent's DID document stops resolving +forever, so that log is the only remaining record of who wrote an hour-old +scrobble. A systemd timer also copies the log to S3 daily. The volume, its +zone (`availability_zone`), its size, and the snapshots' time and count are +variables. The zone is written down, because an EBS volume cannot leave its +zone. **Rollback reaches ten releases back.** ECR keeps the last ten tagged images and the bucket keeps a release tree for ninety days; the shorter window is the @@ -385,5 +388,8 @@ tofu -chdir=infra destroy -var release_sha=$(tofu -chdir=infra output -raw relea ``` Leaves the zone standing — it is permadeath-aws-org's — and the delegation with -it. The S3 bucket has `force_destroy`, so the identity backup goes too; copy it -first if it matters. +it. The data volume has `prevent_destroy`, so the destroy refuses until it is +taken out of state (`tofu state rm aws_ebs_volume.data`), and the volume is +then left behind to delete by hand; its snapshots stop with the policy and +are kept. The S3 bucket has `force_destroy`, so the identity backup goes too; +copy it first if it matters. diff --git a/infra/disk.tf b/infra/disk.tf new file mode 100644 index 0000000..7d1f5e7 --- /dev/null +++ b/infra/disk.tf @@ -0,0 +1,90 @@ +# The data volume: everything the board keeps that a new machine could not +# read back, on a disk that outlives the machine. +# +# The instance's root volume goes with the instance, and a change to user +# data replaces the instance. So the index's files, the relay's database and +# Caddy's certificates live here instead, and a replaced instance is handed +# the same disk. `files/data-disk.sh`, which every activation runs, formats +# it only when it is blank, mounts it, copies across what the root volume +# held the first time, and refuses to bring anything up without it. The +# volume reaches the box as `DATA_VOLUME` in the release parameter, so none +# of this is user data: it lands with a deploy, and no machine is replaced. + +resource "aws_ebs_volume" "data" { + availability_zone = aws_subnet.public.availability_zone + size = var.data_volume_gb + type = "gp3" + encrypted = true + + tags = { + Name = "${local.name}-data" + # What the snapshot policy below looks for. + Snapshots = "${local.name}-data" + } + + lifecycle { + # The one copy of the identity log, burn and the group ids. Deleting it + # is a deliberate act: take this out, apply, and only then destroy. + prevent_destroy = true + } +} + +resource "aws_volume_attachment" "data" { + device_name = "/dev/sdf" + volume_id = aws_ebs_volume.data.id + instance_id = aws_instance.main.id + + # Detached when the instance is replaced. Stopping it first unmounts the + # disk cleanly: `docker stop` lets the index save its state, and nothing is + # writing when the volume goes. + stop_instance_before_detaching = true +} + +# Daily snapshots, the newest `snapshots_kept` kept: for a disk lost or +# written over, or a zone gone. docs/operations.md says how to restore one. +resource "aws_iam_role" "snapshots" { + name = "${local.name}-snapshots" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Effect = "Allow" + Principal = { Service = "dlm.amazonaws.com" } + Action = "sts:AssumeRole" + }] + }) +} + +resource "aws_iam_role_policy_attachment" "snapshots" { + role = aws_iam_role.snapshots.name + policy_arn = "arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole" +} + +resource "aws_dlm_lifecycle_policy" "data" { + description = "${local.name} data volume daily snapshots" + execution_role_arn = aws_iam_role.snapshots.arn + state = "ENABLED" + + policy_details { + resource_types = ["VOLUME"] + target_tags = { Snapshots = "${local.name}-data" } + + schedule { + name = "daily" + copy_tags = true + + create_rule { + interval = 24 + interval_unit = "HOURS" + times = [var.snapshot_time] + } + + retain_rule { + count = var.snapshots_kept + } + + tags_to_add = { + Name = "${local.name}-data" + } + } + } +} diff --git a/infra/files/activate.sh b/infra/files/activate.sh index ed235d4..8c1fe34 100755 --- a/infra/files/activate.sh +++ b/infra/files/activate.sh @@ -45,21 +45,38 @@ install -D -m 0644 "$dir/Caddyfile" "$home/caddy/Caddyfile" ) mv "$home/.env.next" "$home/.env" -# The index and Jetstream run as 65532 and write into these bind mounts. -install -d -o 65532 -g 65532 -m 0750 "$home/data" "$home/jetstream" -chown -R 65532:65532 "$home/data" "$home/jetstream" +# Docker requires the data volume's mounts (data-disk.sh), so after a boot +# without the volume it is not running. Started here, it mounts them first +# and starts once the volume is attached; until then this fails, and converge +# tries again. +systemctl start docker -# Caddy and the relay run as 65532 too, and keep their certificates and -# database in named volumes, which docker makes owned by root. Each is made -# here on a first activation, under the name and labels compose would give it, -# and owned to 65532 before anything starts on it. A volume a root container -# wrote into keeps its contents and changes owner. +# Caddy and the relay keep their certificates and database in named volumes. +# Each is made here on a first activation, under the name and labels compose +# would give it, so the data volume can be mounted into it below. for volume in caddy-data caddy-config relay-data; do name="vibescrobble_$volume" docker volume inspect "$name" >/dev/null 2>&1 || docker volume create --label com.docker.compose.project=vibescrobble \ --label "com.docker.compose.volume=$volume" "$name" >/dev/null - chown -R 65532:65532 "$(docker volume inspect --format '{{.Mountpoint}}' "$name")" +done + +# The index's files and those volumes, on the disk that outlives this +# machine (infra/disk.tf). Until it is attached and mounted this fails, and +# nothing below runs: an index, relay or Caddy started without it would start +# empty, and write over nothing worth keeping only by luck. +install -d -m 0755 "$home/data" +bash "$dir/data-disk.sh" "$dir/release.env" + +# The index and Jetstream run as 65532 and write into these bind mounts. +install -d -o 65532 -g 65532 -m 0750 "$home/data" "$home/jetstream" +chown -R 65532:65532 "$home/data" "$home/jetstream" + +# Caddy and the relay run as 65532 too, and docker makes a named volume owned +# by root, so each is owned to 65532 before anything starts on it. A volume a +# root container wrote into keeps its contents and changes owner. +for volume in caddy-data caddy-config relay-data; do + chown -R 65532:65532 "$(docker volume inspect --format '{{.Mountpoint}}' "vibescrobble_$volume")" done # The operator's denylist, from the object scripts/denylist.sh edits to diff --git a/infra/files/data-disk.sh b/infra/files/data-disk.sh new file mode 100755 index 0000000..7c992ce --- /dev/null +++ b/infra/files/data-disk.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# Put the board's state on the data volume, and refuse to go on without it. +# +# activate.sh runs this before anything is brought up, with the release's +# env file, which names the volume as DATA_VOLUME (infra/disk.tf). Every step +# is safe to repeat, because every activation runs it: +# +# - the volume is formatted only when it holds no filesystem at all; one +# that holds another kind is refused rather than written over; +# - it is mounted at /var/lib/vibescrobble, by a `nofail` line in fstab, so +# a machine whose disk is missing still boots and can be reached; +# - the first time, what the root volume held is copied onto it, with the +# stack stopped so nothing is written mid-copy; a disk that has been +# copied to before, as a replaced instance's is, is never copied over; +# - each directory the stack writes is bind-mounted from it, by fstab too, +# so a reboot puts them back before docker starts; +# - docker itself is made to require those mounts, so after a reboot that +# came up without the disk, no container starts on an empty directory. +# +# Exits non-zero, with nothing started, when the volume is not attached yet +# or cannot be used: converge retries the activation every thirty seconds. +# +# VIBESCROBBLE_ROOT prefixes every path this owns, for scripts/test-data-disk.sh. +set -euo pipefail + +env_file="$1" +root="${VIBESCROBBLE_ROOT:-}" +home="$root/opt/vibescrobble" +disk="$root/var/lib/vibescrobble" +fstab="$root/etc/fstab" +dropin="$root/etc/systemd/system/docker.service.d/vibescrobble-data.conf" +# What was copied onto the disk, and when. Its presence is what says the +# disk holds the board, whatever the root volume beside it holds. +copied="$disk/.copied" + +volume="$(sed -n 's/^DATA_VOLUME=//p' "$env_file")" +if [ -z "$volume" ]; then + echo "data-disk: the release names no DATA_VOLUME; nothing is started without one" >&2 + exit 1 +fi +# The NVMe serial is the volume id without its dash, and udev names the +# device by it, whatever device name the attachment asked for. +device="$root/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_${volume//-/}" +if [ ! -e "$device" ]; then + echo "data-disk: $volume is not attached yet; nothing is started until it is" >&2 + exit 1 +fi + +# Directory the stack writes, and where on the disk it lives. The docker +# volumes are asked for their mount points: activate.sh makes them first. +targets=("$home/data") +sources=("$disk/data") +for name in caddy-data caddy-config relay-data; do + targets+=("$(docker volume inspect --format '{{.Mountpoint}}' "vibescrobble_$name")") + sources+=("$disk/volumes/$name") +done + +# One fstab line per mount point, replacing whatever line it had. +fstab_line() { + local point="$1" line="$2" + touch "$fstab" + awk -v point="$point" '$2 != point' "$fstab" > "$fstab.next" + printf '%s\n' "$line" >> "$fstab.next" + if ! cmp -s "$fstab.next" "$fstab"; then + mv "$fstab.next" "$fstab" + else + rm "$fstab.next" + fi +} + +if ! mountpoint -q "$disk"; then + if fstype="$(blkid -p -o value -s TYPE "$device")"; then + if [ "$fstype" != xfs ]; then + echo "data-disk: $volume holds $fstype, not xfs; refusing to use or format it" >&2 + exit 1 + fi + else + status=$? + if [ "$status" -ne 2 ]; then + echo "data-disk: cannot read $volume's filesystem (blkid exited $status)" >&2 + exit 1 + fi + echo "data-disk: $volume is blank; formatting it" + mkfs.xfs -q -L vibescrobble "$device" + fi + uuid="$(blkid -p -o value -s UUID "$device")" + fstab_line /var/lib/vibescrobble \ + "UUID=$uuid /var/lib/vibescrobble xfs defaults,nofail,x-systemd.device-timeout=30s 0 2" + mkdir -p "$disk" + mount "$disk" + echo "data-disk: $volume mounted" +fi + +unbound=() +for i in "${!targets[@]}"; do + mountpoint -q "${targets[$i]}" || unbound+=("$i") +done + +if [ "${#unbound[@]}" -gt 0 ]; then + # A container keeps the directory it started with, so the stack stops + # before one is mounted over, and activate.sh brings it up again. + if [ -f "$home/compose.yml" ]; then + (cd "$home" && docker compose stop) + fi + if [ ! -f "$copied" ]; then + for i in "${unbound[@]}"; do + mkdir -p "${sources[$i]}" "${targets[$i]}" + cp -a "${targets[$i]}/." "${sources[$i]}/" + done + date -u +%Y-%m-%dT%H:%M:%SZ > "$copied" + echo "data-disk: copied the root volume's state onto $volume" + fi + for i in "${unbound[@]}"; do + mkdir -p "${sources[$i]}" "${targets[$i]}" + fstab_line "${targets[$i]#"$root"}" \ + "${sources[$i]#"$root"} ${targets[$i]#"$root"} none bind,nofail,x-systemd.requires-mounts-for=/var/lib/vibescrobble 0 0" + mount "${targets[$i]}" + done +fi + +for target in "${targets[@]}"; do + if ! mountpoint -q "$target"; then + echo "data-disk: $target is not on $volume; nothing is started" >&2 + exit 1 + fi +done + +mkdir -p "$(dirname "$dropin")" +{ + echo "# Written by data-disk.sh: no container starts without the data volume." + echo "[Unit]" + for target in "${targets[@]}"; do + echo "RequiresMountsFor=${target#"$root"}" + done +} > "$dropin.next" +if ! cmp -s "$dropin.next" "$dropin"; then + mv "$dropin.next" "$dropin" + systemctl daemon-reload +else + rm "$dropin.next" +fi diff --git a/infra/network.tf b/infra/network.tf index 1ee0f07..4afbd6f 100644 --- a/infra/network.tf +++ b/infra/network.tf @@ -16,12 +16,11 @@ resource "aws_internet_gateway" "main" { tags = { Name = local.name } } -# Not every zone can run every machine. A subnet that does not name a zone is -# given one by AWS, and in us-east-1 the pick can be `us-east-1e`, which has no -# Graviton in it at all — so the subnet is fine and the instance is rejected. -# Ask which zones offer the instance type and take the lowest, rather than -# writing down a letter: zone letters are per-account, and the answer changes -# when `instance_type` does. +# Not every zone can run every machine: in us-east-1, `us-east-1e` has no +# Graviton in it at all, so a subnet there is fine and the instance is +# rejected. The zone is `var.availability_zone`, written down because the data +# volume cannot leave it; this asks which zones offer the instance type, so a +# type the zone does not sell fails the plan instead of the apply. data "aws_ec2_instance_type_offerings" "instance" { location_type = "availability-zone" @@ -32,15 +31,15 @@ data "aws_ec2_instance_type_offerings" "instance" { lifecycle { postcondition { - condition = length(self.locations) > 0 - error_message = "No us-east-1 availability zone offers ${var.instance_type}. Pick a type this region sells." + condition = contains(self.locations, var.availability_zone) + error_message = "${var.availability_zone} does not offer ${var.instance_type}. Pick a type it sells: the data volume cannot move zones." } } } resource "aws_subnet" "public" { vpc_id = aws_vpc.main.id - availability_zone = sort(data.aws_ec2_instance_type_offerings.instance.locations)[0] + availability_zone = var.availability_zone cidr_block = "10.0.1.0/24" map_public_ip_on_launch = true tags = { Name = "${local.name}-public" } diff --git a/infra/outputs.tf b/infra/outputs.tf index 47c3d2e..f742ccd 100644 --- a/infra/outputs.tf +++ b/infra/outputs.tf @@ -47,3 +47,8 @@ output "alarms_topic" { description = "Where alarms are published. The email subscription on it has to be confirmed by hand." value = aws_sns_topic.alarms.arn } + +output "data_volume" { + description = "The volume the board's state lives on, which outlives the instance. docs/operations.md restores it from a snapshot." + value = aws_ebs_volume.data.id +} diff --git a/infra/release.tf b/infra/release.tf index 32276d5..8f427c9 100644 --- a/infra/release.tf +++ b/infra/release.tf @@ -7,9 +7,10 @@ # is the record of which one is live. `scripts/deploy.sh` is the apply with # the build and the wait around it. # -# The two upstream refs, the relay's crawl limits and each container's memory -# limit ride along for the same reason: the compose file reads them, and a -# change to any is a deploy rather than an instance replacement. +# The two upstream refs, the relay's crawl limits, each container's memory +# limit and the data volume ride along for the same reason: the compose file +# and activate.sh read them, and a change to any is a deploy rather than an +# instance replacement. resource "aws_ssm_parameter" "release" { name = "/${local.name}/release" description = "The release vibescrobble.com runs. Written by an apply; read by the instance every thirty seconds." @@ -22,6 +23,7 @@ resource "aws_ssm_parameter" "release" { "RELAY_DEFAULT_ACCOUNT_LIMIT=${var.relay_default_account_limit}", "RELAY_NEW_HOSTS_PER_DAY=${var.relay_new_hosts_per_day}", "DENYLIST_OBJECT=${aws_s3_object.denylist.key}", + "DATA_VOLUME=${aws_ebs_volume.data.id}", "CADDY_MEMORY=${var.caddy_memory_mb}m", "RELAY_MEMORY=${var.relay_memory_mb}m", "JETSTREAM_MEMORY=${var.jetstream_memory_mb}m", diff --git a/infra/variables.tf b/infra/variables.tf index 2fea2f9..f52d583 100644 --- a/infra/variables.tf +++ b/infra/variables.tf @@ -42,6 +42,50 @@ variable "disk_gb" { default = 40 } +variable "availability_zone" { + description = "The zone the subnet, the instance and the data volume are in. An EBS volume can be attached only in its own zone, so this is written down rather than worked out from `instance_type`, which would move the subnet, and replace the instance, whenever the type's lowest zone changed. us-east-1a is where it has run since the start; a plan fails before anything is made if `instance_type` is not sold here. Changing it replaces the subnet and the instance, and the volume cannot follow: restore a snapshot in the new zone (docs/operations.md)." + type = string + default = "us-east-1a" + + validation { + condition = can(regex("^us-east-1[a-f]$", var.availability_zone)) + error_message = "availability_zone is a us-east-1 zone, us-east-1a to us-east-1f." + } +} + +variable "data_volume_gb" { + description = "The data volume, which outlives the instance: the index's files, the relay's database and Caddy's certificates. About 30 MB in October 2026. 8 to 200; gp3 gives the same 3000 IOPS at any size, and growing is online." + type = number + default = 20 + + validation { + condition = var.data_volume_gb >= 8 && var.data_volume_gb <= 200 + error_message = "data_volume_gb is 8 to 200." + } +} + +variable "snapshot_time" { + description = "When the data volume's daily snapshot starts, UTC, as HH:MM. Data Lifecycle Manager starts it within the hour after. Any time works; the default is the board's quietest hour." + type = string + default = "09:00" + + validation { + condition = can(regex("^([01][0-9]|2[0-3]):[0-5][0-9]$", var.snapshot_time)) + error_message = "snapshot_time is HH:MM, 00:00 to 23:59, UTC." + } +} + +variable "snapshots_kept" { + description = "How many daily snapshots of the data volume are kept, the oldest deleted as each is made. 2 to 35: one is a single bad day from nothing, and past a month, what was lost is long noticed. Snapshots are incremental, so each after the first costs only what changed." + type = number + default = 7 + + validation { + condition = var.snapshots_kept >= 2 && var.snapshots_kept <= 35 + error_message = "snapshots_kept is 2 to 35." + } +} + variable "alarm_email" { description = "Where alarms go. AWS sends a confirmation link first and delivers nothing until it is clicked." type = string diff --git a/prek.toml b/prek.toml index 6518623..066fee3 100644 --- a/prek.toml +++ b/prek.toml @@ -94,6 +94,7 @@ hooks = [ { id = "plugin-hook", name = "plugin hook tests", entry = "python3 -m unittest discover -s plugin/tests", language = "system", files = '^(plugin/|\.claude-plugin/|crates/vibescrobble-cli/src/session\.rs$)', pass_filenames = false, stages = ["pre-commit"] }, { id = "shell-parse", name = "shell parse", entry = "bash -c 'for f in \"$@\"; do bash -n \"$f\" || exit 1; done' _", language = "system", files = '^(scripts/.*|infra/files/.*)\.sh$', stages = ["pre-commit"] }, { id = "denylist-test", name = "denylist script test", entry = "scripts/test-denylist.sh", language = "system", files = '^scripts/(test-)?denylist\.sh$', pass_filenames = false, stages = ["pre-commit"] }, + { id = "data-disk-test", name = "data disk script test", entry = "scripts/test-data-disk.sh", language = "system", files = '^(scripts/test-data-disk\.sh|infra/files/(data-disk|activate)\.sh)$', pass_filenames = false, stages = ["pre-commit"] }, # The canvas's scripts, run under Node with the browser stubbed out. A page # counts as well as a script: index.html's script order is what the module # test loads. install.sh is served from web/ too, and run there under dash diff --git a/scripts/release.sh b/scripts/release.sh index 3822358..f91e2b3 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -17,8 +17,8 @@ # Upstream, Go, and built under emulation, so slow — but once per ref, # not once per release. # 3. releases// in the artifacts bucket: the canvas, the compose file, -# the Caddyfile, and activate.sh, which is what the instance runs to -# bring the release up. activate.sh goes last, so its presence means the +# the Caddyfile, data-disk.sh, and activate.sh, which is what the +# instance runs to bring the release up. activate.sh goes last, so its presence means the # tree is whole. # # A commit that is the tag v also publishes the CLI, once: the @@ -85,7 +85,7 @@ sha="$(git rev-parse --verify --quiet --short=12 "$rev^{commit}")" || { # Every file this ships has to exist at that commit, and a commit from # before any of them did cannot be released as it stands. for path in web infra/files/compose.yml infra/files/Caddyfile \ - infra/files/activate.sh infra/files/vibescrobble.Dockerfile; do + infra/files/activate.sh infra/files/data-disk.sh infra/files/vibescrobble.Dockerfile; do git cat-file -e "$sha:$path" 2>/dev/null || { echo "release: $path is not in $sha; only commits carrying it can be released" >&2 exit 1 @@ -242,11 +242,11 @@ else # web/dev/ holds the mock and its fixtures, which only the dev page loads. # The READMEs beside the assets are for people reading the repository. git archive --format=tar "$sha" web ":(exclude)web/dev" ":(exclude,glob)web/**/*.md" | gzip -n > "$work/tree/web.tar.gz" - for file in compose.yml Caddyfile activate.sh; do + for file in compose.yml Caddyfile data-disk.sh activate.sh; do git show "$sha:infra/files/$file" > "$work/tree/$file" done # activate.sh last: the instance takes its presence as the tree being whole. - for file in web.tar.gz compose.yml Caddyfile activate.sh; do + for file in web.tar.gz compose.yml Caddyfile data-disk.sh activate.sh; do aws s3 cp --region "$region" --no-progress "$work/tree/$file" "$prefix/$file" done fi diff --git a/scripts/test-data-disk.sh b/scripts/test-data-disk.sh new file mode 100755 index 0000000..916ee26 --- /dev/null +++ b/scripts/test-data-disk.sh @@ -0,0 +1,176 @@ +#!/usr/bin/env bash +# Runs infra/files/data-disk.sh under a fake root, against fakes of what it +# asks the machine: `blkid` and `mkfs.xfs` read and write a device that is a +# file, `mount` stands a directory in for a disk or a bind mount by a symlink, +# and `docker` and `systemctl` say where docker's volumes are and note what +# was asked of them. A blank disk, a disk that already holds the board, a +# disk not attached, and one holding something else. +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +script="$here/../infra/files/data-disk.sh" +T="$(mktemp -d)" +trap 'rm -rf "$T"' EXIT +mkdir "$T/bin" +volume=vol-0abc123def4567890 +device_name="nvme-Amazon_Elastic_Block_Store_${volume//-/}" + +cat >"$T/bin/blkid" <<'FAKE' +#!/usr/bin/env bash +# blkid -p -o value -s TYPE|UUID ; the device file holds "TYPE UUID", +# and is empty when blank, which blkid says by exiting 2. +set -euo pipefail +field="$5" device="$6" +read -r type uuid <"$device" || true +[ -n "${type:-}" ] || exit 2 +case "$field" in TYPE) echo "$type" ;; UUID) echo "$uuid" ;; esac +FAKE +cat >"$T/bin/mkfs.xfs" <<'FAKE' +#!/usr/bin/env bash +set -euo pipefail +for device; do :; done +echo "xfs 5a1e-uuid" >"$device" +echo "mkfs $device" >>"$FAKE_LOG" +FAKE +cat >"$T/bin/mountpoint" <<'FAKE' +#!/usr/bin/env bash +grep -qxF "$2" "$FAKE_MOUNTED" 2>/dev/null +FAKE +cat >"$T/bin/mount" <<'FAKE' +#!/usr/bin/env bash +# mount , from the fstab line naming it. The disk is the directory +# FAKE_DISK; a bind mount is a symlink to its source, with what the point held +# moved aside, as a real one hides it. +set -euo pipefail +point="$1" +line="$(awk -v p="${point#"$VIBESCROBBLE_ROOT"}" '$2 == p' "$VIBESCROBBLE_ROOT/etc/fstab")" +[ -n "$line" ] || { echo "mount: $point is not in fstab" >&2; exit 32; } +case "$line" in + UUID=*) source="$FAKE_DISK" ;; + *) source="$VIBESCROBBLE_ROOT${line%% *}" ;; +esac +mv "$point" "$point.hidden" +ln -s "$source" "$point" +echo "$point" >>"$FAKE_MOUNTED" +echo "mount $point" >>"$FAKE_LOG" +FAKE +cat >"$T/bin/docker" <<'FAKE' +#!/usr/bin/env bash +set -euo pipefail +case "$1" in + volume) for name; do :; done; echo "$VIBESCROBBLE_ROOT/var/lib/docker/volumes/$name/_data" ;; + compose) echo "compose $2" >>"$FAKE_LOG" ;; +esac +FAKE +cat >"$T/bin/systemctl" <<'FAKE' +#!/usr/bin/env bash +echo "systemctl $*" >>"$FAKE_LOG" +FAKE +chmod +x "$T/bin"/* +export PATH="$T/bin:$PATH" + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +# A machine: a root with a release env, and a disk, blank unless told. +machine() { + local name="$1" + export VIBESCROBBLE_ROOT="$T/$name/root" FAKE_DISK="$T/$name/disk" + export FAKE_LOG="$T/$name/log" FAKE_MOUNTED="$T/$name/mounted" + mkdir -p "$VIBESCROBBLE_ROOT/opt/vibescrobble/data" "$VIBESCROBBLE_ROOT/etc" \ + "$VIBESCROBBLE_ROOT/dev/disk/by-id" "$FAKE_DISK" + for v in caddy-data caddy-config relay-data; do + mkdir -p "$VIBESCROBBLE_ROOT/var/lib/docker/volumes/vibescrobble_$v/_data" + done + : >"$FAKE_LOG" + printf 'VERSION=0123456789ab\nDATA_VOLUME=%s\n' "$volume" >"$T/$name/release.env" + touch "$VIBESCROBBLE_ROOT/opt/vibescrobble/compose.yml" +} +attach() { + printf '%s' "${1:-}" >"$VIBESCROBBLE_ROOT/dev/disk/by-id/$device_name" +} +run() { + bash "$script" "$T/$1/release.env" +} +log_has() { + grep -qxF "$1" "$FAKE_LOG" +} +data() { + cat "$VIBESCROBBLE_ROOT/opt/vibescrobble/data/$1" +} + +# A blank disk beside a machine that has been running: formatted, the stack +# stopped, what the root held copied across, and every directory mounted +# from it. +machine running +attach +echo root-state >"$VIBESCROBBLE_ROOT/opt/vibescrobble/data/state.json" +echo relay-db >"$VIBESCROBBLE_ROOT/var/lib/docker/volumes/vibescrobble_relay-data/_data/relay.sqlite" +run running >/dev/null || fail "a blank disk is taken" +log_has "mkfs $VIBESCROBBLE_ROOT/dev/disk/by-id/$device_name" || fail "a blank disk is formatted" +log_has "compose stop" || fail "the stack stops before it is moved" +[ "$(data state.json)" = root-state ] || fail "the index's state is on the disk" +[ "$(cat "$FAKE_DISK/data/state.json")" = root-state ] || fail "copied onto the disk" +[ "$(cat "$FAKE_DISK/volumes/relay-data/relay.sqlite")" = relay-db ] || fail "the relay's database is on the disk" +[ -f "$FAKE_DISK/.copied" ] || fail "the copy is noted on the disk" +[ "$(grep -c . "$VIBESCROBBLE_ROOT/etc/fstab")" = 5 ] || fail "the disk and four binds in fstab: $(cat "$VIBESCROBBLE_ROOT/etc/fstab")" +grep -q '^UUID=5a1e-uuid /var/lib/vibescrobble xfs defaults,nofail' "$VIBESCROBBLE_ROOT/etc/fstab" || + fail "the disk by its UUID, nofail" +grep -qxF '/var/lib/vibescrobble/data /opt/vibescrobble/data none bind,nofail,x-systemd.requires-mounts-for=/var/lib/vibescrobble 0 0' \ + "$VIBESCROBBLE_ROOT/etc/fstab" || fail "the index's directory, bound from the disk" +dropin="$VIBESCROBBLE_ROOT/etc/systemd/system/docker.service.d/vibescrobble-data.conf" +grep -qxF 'RequiresMountsFor=/opt/vibescrobble/data' "$dropin" || fail "docker requires the mounts" +log_has "systemctl daemon-reload" || fail "systemd reads the drop-in" + +# Again, as every activation runs it: nothing is formatted, stopped, copied +# or written again. +fstab_before="$(cat "$VIBESCROBBLE_ROOT/etc/fstab")" +: >"$FAKE_LOG" +echo later-state >"$FAKE_DISK/data/state.json" +run running >/dev/null || fail "a second run is fine" +[ ! -s "$FAKE_LOG" ] || fail "a second run does nothing: $(cat "$FAKE_LOG")" +[ "$(cat "$VIBESCROBBLE_ROOT/etc/fstab")" = "$fstab_before" ] || fail "fstab is not written twice" +[ "$(data state.json)" = later-state ] || fail "what the disk holds stays" + +# The replacement machine is handed the same disk: it is not formatted, and +# the new root's empty directories are not copied over what it holds. +disk_dir="$FAKE_DISK" +machine replaced +export FAKE_DISK="$disk_dir" +attach "xfs 5a1e-uuid" +echo fresh-empty >"$VIBESCROBBLE_ROOT/opt/vibescrobble/data/state.json" +run replaced >/dev/null || fail "a populated disk is taken" +log_has "mkfs $VIBESCROBBLE_ROOT/dev/disk/by-id/$device_name" && fail "a populated disk is never formatted" +[ "$(data state.json)" = later-state ] || fail "the board comes back from the disk, not the new root" +[ "$(cat "$FAKE_DISK/volumes/relay-data/relay.sqlite")" = relay-db ] || fail "the relay's database survives" + +# Not attached yet: nothing is formatted, mounted or stopped, and the +# activation fails, so nothing is brought up. +machine missing +if run missing 2>"$T/missing.err"; then fail "a missing disk stops the activation"; fi +grep -q 'not attached yet' "$T/missing.err" || fail "and says why: $(cat "$T/missing.err")" +[ ! -s "$FAKE_LOG" ] || fail "a missing disk touches nothing: $(cat "$FAKE_LOG")" +[ ! -e "$VIBESCROBBLE_ROOT/etc/fstab" ] || fail "nor fstab" + +# A disk holding another filesystem is neither formatted nor used. +machine foreign +attach "ext4 0ther-uuid" +if run foreign 2>"$T/foreign.err"; then fail "another filesystem stops the activation"; fi +grep -q 'holds ext4' "$T/foreign.err" || fail "and says why: $(cat "$T/foreign.err")" +[ ! -s "$FAKE_LOG" ] || fail "nothing is formatted or mounted: $(cat "$FAKE_LOG")" + +# A release that names no volume brings nothing up. +machine unnamed +attach +sed -i '/^DATA_VOLUME=/d' "$T/unnamed/release.env" +if run unnamed 2>/dev/null; then fail "a release without DATA_VOLUME stops the activation"; fi + +# And activate.sh asks for the disk before it brings anything up. +activate="$here/../infra/files/activate.sh" +disk_line="$(grep -n 'data-disk.sh' "$activate" | head -1 | cut -d: -f1)" +up_line="$(grep -n 'docker compose up' "$activate" | head -1 | cut -d: -f1)" +[ -n "$disk_line" ] && [ "$disk_line" -lt "$up_line" ] || fail "activate.sh runs data-disk.sh before compose up" + +echo "data-disk: ok"