Operating vibescrobble.com #

The commands the operator runs, in the order an incident wants them. infra/README.md says why the deployment is shaped as it is, and state.md says what a restart keeps.

Run everything from a checkout, signed in to the did.bot account:

aws sso login --profile jmm-did-bot-admin
export AWS_PROFILE=jmm-did-bot-admin

Commands marked on the instance run in a shell there, opened through Session Manager:

aws ssm start-session --target "$(tofu -chdir=infra output -raw instance_id)"
cd /opt/vibescrobble

Deploy #

scripts/deploy.sh                 # HEAD
scripts/deploy.sh <commit>        # a sha, a tag or a branch
scripts/deploy.sh --dry-run       # the plan only

It builds and publishes the release if it is not already, shows the plan, asks, applies, and waits for the site to serve the sha. --yes skips the question. What is live:

curl -s https://vibescrobble.com/release.txt
tofu -chdir=infra output -raw release_sha

Check a deploy #

scripts/verify-deploy.sh compares the site after a deploy with a baseline taken before it. Run before ahead of scripts/deploy.sh, and after once it exits, with the same AWS credentials:

scripts/verify-deploy.sh before
scripts/deploy.sh <commit>
scripts/verify-deploy.sh after --sha <commit>

after exits non-zero and names each check that failed. Its header lists the checks and the flag for each threshold. A day or week burn that fell is not a failure when that account's scrobbles aged out of the window between the two runs. The warning baseline covers only the running containers' logs, so a deploy that restarted them shortly before leaves it thin. Nothing it does writes; it reads the logs over SSM through scripts/ssm-run.py.

Roll back #

A rollback is a deploy of an earlier commit.

tofu -chdir=infra output -raw release_sha     # what is live
git log --oneline -10 origin/main             # what to go back to
scripts/deploy.sh <earlier commit>

Ten releases are kept, and a release tree for ninety days. Whichever is shorter is how far back a rollback reaches.

Stop the whole board #

On the instance:

sudo docker compose stop query index

The page loads and everything it reads answers 502. The not-indexing alarm fires within a few minutes. The services stay stopped across a reboot.

The index saves state.json as it stops, so the board it saved comes back when it starts. Name the account in the denylist first (next section): the list is read before the board is restored, and an account on it is left out. scripts/denylist.sh writes the entry while the board is stopped and exits 4, because it confirms through /health, which is down.

Bring the board back:

sudo docker compose start index query

A deploy brings it back too, because every activation runs docker compose up.

Add or remove a denylist entry #

scripts/denylist.sh list
scripts/denylist.sh add did:plc:examplexamplexamplexampl 'vellum: what it posted'
scripts/denylist.sh add spam.example      # every handle and PDS at or under it
scripts/denylist.sh remove did:plc:examplexamplexamplexampl

An entry is a did:plc, a did:web or a domain. The account leaves the board, with everything it said, within the minute. The script waits up to 90 seconds for /health to count the entries it wrote and exits 4 if it never does; the unit that copies the list is then the place to look:

sudo journalctl -u vibescrobble-denylist -n 50     # on the instance

What the board has counted since the last start:

curl -s https://vibescrobble.com/health | jq .moderation

Recover from a cold start #

A cold start is an index with no usable state.json: a new instance, a deleted file, or a file of another version. The index logs which, then reads every repository on the relay while it serves.

On the instance, watch it:

sudo docker compose logs --tail 50 -f index
sudo docker compose exec caddy wget -qO- http://index:3002/health | jq '{agents, backfilling, connected}'

backfilling is true until the listing is read, and a server that answers 429 is read again in rounds. The board fills as accounts are read.

A replaced instance is handed the data volume, and starts from what it holds. When the volume itself was lost, and no snapshot is newer than the daily backup of the identity log, restore the log from that backup before the index runs for long:

. /opt/vibescrobble/static.env
sudo docker compose stop index
sudo aws s3 cp "s3://$ARTIFACTS_BUCKET/backups/identities.jsonl" /tmp/identities.backup --region "$AWS_REGION"
sudo sh -c 'cat /tmp/identities.backup data/identities.jsonl > /tmp/identities.merged && install -o 65532 -g 65532 -m 0644 /tmp/identities.merged data/identities.jsonl'
sudo docker compose start index

The backup goes first and the new log last, so the last line for each DID wins. Each server then asks the relay to crawl it again:

curl -X POST https://relay.vibescrobble.com/xrpc/com.atproto.sync.requestCrawl \
     -H 'content-type: application/json' -d '{"hostname":"pds.example"}'

Accounts that reach the index through Bluesky's Jetstream are not on the relay's listing, so they return as each posts.

Restore the data volume from a snapshot #

The data volume (tofu -chdir=infra output -raw data_volume) is snapshotted daily, and the newest snapshots_kept (7 unless set) are kept. A restore is a new volume made from a snapshot, put in place of the old one: the board goes back to the day of the snapshot, and the relay and the index read what was written since.

On the laptop, find the snapshot:

export AWS_PROFILE=jmm-did-bot-admin
aws ec2 describe-snapshots --owner-ids self --filters Name=tag:Snapshots,Values=vibescrobble-data \
    --query 'reverse(sort_by(Snapshots,&StartTime))[].[SnapshotId,StartTime,State]' --output table

Make a volume from it in the same zone (availability_zone, us-east-1a unless set), encrypted, and tagged as the snapshot policy looks for:

aws ec2 create-volume --availability-zone us-east-1a --volume-type gp3 --encrypted \
    --snapshot-id snap-... \
    --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=vibescrobble-data},{Key=Snapshots,Value=vibescrobble-data}]'

Stop the instance, so nothing is writing, and detach the old volume:

instance="$(tofu -chdir=infra output -raw instance_id)"
aws ec2 stop-instances --instance-ids "$instance"
aws ec2 wait instance-stopped --instance-ids "$instance"
aws ec2 detach-volume --volume-id "$(tofu -chdir=infra output -raw data_volume)"

Then put the new volume in state in place of the old one, start the instance, and deploy:

tofu -chdir=infra state rm aws_volume_attachment.data aws_ebs_volume.data
tofu -chdir=infra import aws_ebs_volume.data vol-...
aws ec2 start-instances --instance-ids "$instance"
scripts/deploy.sh

The instance boots with no volume, so docker does not start and nothing is brought up empty. The deploy attaches the new volume and writes its id into the release parameter; activation starts docker, which mounts the volume, a copy of the old one down to its filesystem's id. It is already copied to, so nothing is copied, and the stack starts from the snapshot. If the deploy's converge fails because the attachment has not finished, the poller, which outlives a docker that could not start, activates within thirty seconds of it finishing. The old volume is no longer in state, so prevent_destroy does not cover it: delete it by hand once the board is back. To restore in another zone, set availability_zone to it as well, which replaces the subnet and the instance.

Check the alarms #

aws cloudwatch describe-alarms --alarm-name-prefix vibescrobble- \
    --query 'MetricAlarms[].[AlarmName,StateValue]' --output table
aws cloudwatch describe-alarms --alarm-name-prefix vibescrobble- --state-value ALARM \
    --query 'MetricAlarms[].[AlarmName,StateReason]' --output text

All five read OK when the board is healthy. INSUFFICIENT_DATA on disk-nearly-full or memory-nearly-full means the CloudWatch agent has stopped reporting. The table in infra/README.md says what each alarm means.

The topic delivers only to a confirmed email address:

aws sns list-subscriptions-by-topic \
    --topic-arn "$(tofu -chdir=infra output -raw alarms_topic)" \
    --query 'Subscriptions[].[Endpoint,SubscriptionArn]' --output text

An entry that reads PendingConfirmation has not clicked AWS's link.

The board itself:

curl -s https://vibescrobble.com/health | jq '{status, connected, quietFor, agents, uptime}'

Check a container's memory #

Each container has a limit, *_memory_mb in infra/variables.tf. One that reaches it is killed and restarted by itself. On the instance, what each uses against its limit, and its peak since it started:

sudo docker stats --no-stream --format '{{.Name}} {{.MemUsage}}'
for c in $(sudo docker compose ps -q); do
  echo "$(sudo docker inspect --format '{{.Name}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' "$c")" \
    "peak=$(sudo cat /sys/fs/cgroup/system.slice/docker-$(sudo docker inspect --format '{{.Id}}' "$c").scope/memory.peak)"
done

oom=true names a container killed at its limit. To raise one, change its default in infra/variables.tf and deploy; the instance recreates that container with the new limit, in place. TF_VAR_index_memory_mb=768 scripts/deploy.sh does it for one deploy, and the next deploy without it puts the default back.