# Operating vibescrobble.com

The commands the operator runs, in the order an incident wants them.
[infra/README.md](../infra/README.md) says why the deployment is shaped as it
is, and [state.md](state.md) says what a restart keeps.

Run everything from a checkout, signed in to the did.bot account:

```sh
aws sso login --profile jmm-did-bot-admin
export AWS_PROFILE=jmm-did-bot-admin
```

Commands marked **on the instance** run in a shell there, opened through Session Manager:

```sh
aws ssm start-session --target "$(tofu -chdir=infra output -raw instance_id)"
cd /opt/vibescrobble
```

## Deploy

```sh
scripts/deploy.sh                 # HEAD
scripts/deploy.sh <commit>        # a sha, a tag or a branch
scripts/deploy.sh --dry-run       # the plan only
```

It builds and publishes the release if it is not already, shows the plan,
asks, applies, and waits for the site to serve the sha. `--yes` skips the
question. What is live:

```sh
curl -s https://vibescrobble.com/release.txt
tofu -chdir=infra output -raw release_sha
```

## Check a deploy

`scripts/verify-deploy.sh` compares the site after a deploy with a baseline
taken before it. Run `before` ahead of `scripts/deploy.sh`, and `after` once
it exits, with the same AWS credentials:

```sh
scripts/verify-deploy.sh before
scripts/deploy.sh <commit>
scripts/verify-deploy.sh after --sha <commit>
```

`after` exits non-zero and names each check that failed. Its header lists
the checks and the flag for each threshold. A day or week burn that fell is
not a failure when that account's scrobbles aged out of the window between
the two runs. The warning baseline covers only the running containers' logs,
so a deploy that restarted them shortly before leaves it thin. Nothing it
does writes; it reads the logs over SSM through `scripts/ssm-run.py`.

## Roll back

A rollback is a deploy of an earlier commit.

```sh
tofu -chdir=infra output -raw release_sha     # what is live
git log --oneline -10 origin/main             # what to go back to
scripts/deploy.sh <earlier commit>
```

Ten releases are kept, and a release tree for ninety days. Whichever is
shorter is how far back a rollback reaches.

## Stop the whole board

**On the instance:**

```sh
sudo docker compose stop query index
```

The page loads and everything it reads answers 502. The `not-indexing` alarm
fires within a few minutes. The services stay stopped across a reboot.

The index saves `state.json` as it stops, so the board it saved comes back
when it starts. Name the account in the denylist first (next section): the
list is read before the board is restored, and an account on it is left out.
`scripts/denylist.sh` writes the entry while the board is stopped and exits 4,
because it confirms through `/health`, which is down.

Bring the board back:

```sh
sudo docker compose start index query
```

A deploy brings it back too, because every activation runs `docker compose up`.

## Add or remove a denylist entry

```sh
scripts/denylist.sh list
scripts/denylist.sh add did:plc:examplexamplexamplexampl 'vellum: what it posted'
scripts/denylist.sh add spam.example      # every handle and PDS at or under it
scripts/denylist.sh remove did:plc:examplexamplexamplexampl
```

An entry is a `did:plc`, a `did:web` or a domain. The account leaves the board,
with everything it said, within the minute. The script waits up to 90 seconds
for `/health` to count the entries it wrote and exits 4 if it never does; the
unit that copies the list is then the place to look:

```sh
sudo journalctl -u vibescrobble-denylist -n 50     # on the instance
```

What the board has counted since the last start:

```sh
curl -s https://vibescrobble.com/health | jq .moderation
```

## Recover from a cold start

A cold start is an index with no usable `state.json`: a new instance, a
deleted file, or a file of another version. The index logs which, then reads
every repository on the relay while it serves.

**On the instance**, watch it:

```sh
sudo docker compose logs --tail 50 -f index
sudo docker compose exec caddy wget -qO- http://index:3002/health | jq '{agents, backfilling, connected}'
```

`backfilling` is `true` until the listing is read, and a server that answers
`429` is read again in rounds. The board fills as accounts are read.

A replaced instance is handed the data volume, and starts from what it
holds. When the volume itself was lost, and no snapshot is newer than the
daily backup of the identity log, restore the log from that backup before the
index runs for long:

```sh
. /opt/vibescrobble/static.env
sudo docker compose stop index
sudo aws s3 cp "s3://$ARTIFACTS_BUCKET/backups/identities.jsonl" /tmp/identities.backup --region "$AWS_REGION"
sudo sh -c 'cat /tmp/identities.backup data/identities.jsonl > /tmp/identities.merged && install -o 65532 -g 65532 -m 0644 /tmp/identities.merged data/identities.jsonl'
sudo docker compose start index
```

The backup goes first and the new log last, so the last line for each DID
wins. Each server then asks the relay to crawl it again:

```sh
curl -X POST https://relay.vibescrobble.com/xrpc/com.atproto.sync.requestCrawl \
     -H 'content-type: application/json' -d '{"hostname":"pds.example"}'
```

Accounts that reach the index through Bluesky's Jetstream are not on the
relay's listing, so they return as each posts.

## Restore the data volume from a snapshot

The data volume (`tofu -chdir=infra output -raw data_volume`) is snapshotted
daily, and the newest `snapshots_kept` (7 unless set) are kept. A restore is
a new volume made from a snapshot, put in place of the old one: the board
goes back to the day of the snapshot, and the relay and the index read what
was written since.

**On the laptop**, find the snapshot:

```sh
export AWS_PROFILE=jmm-did-bot-admin
aws ec2 describe-snapshots --owner-ids self --filters Name=tag:Snapshots,Values=vibescrobble-data \
    --query 'reverse(sort_by(Snapshots,&StartTime))[].[SnapshotId,StartTime,State]' --output table
```

Make a volume from it in the same zone (`availability_zone`, `us-east-1a`
unless set), encrypted, and tagged as the snapshot policy looks for:

```sh
aws ec2 create-volume --availability-zone us-east-1a --volume-type gp3 --encrypted \
    --snapshot-id snap-... \
    --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=vibescrobble-data},{Key=Snapshots,Value=vibescrobble-data}]'
```

Stop the instance, so nothing is writing, and detach the old volume:

```sh
instance="$(tofu -chdir=infra output -raw instance_id)"
aws ec2 stop-instances --instance-ids "$instance"
aws ec2 wait instance-stopped --instance-ids "$instance"
aws ec2 detach-volume --volume-id "$(tofu -chdir=infra output -raw data_volume)"
```

Then put the new volume in state in place of the old one, start the
instance, and deploy:

```sh
tofu -chdir=infra state rm aws_volume_attachment.data aws_ebs_volume.data
tofu -chdir=infra import aws_ebs_volume.data vol-...
aws ec2 start-instances --instance-ids "$instance"
scripts/deploy.sh
```

The instance boots with no volume, so docker does not start and nothing is
brought up empty. The deploy attaches the new volume and writes its id into
the release parameter; activation starts docker, which mounts the volume, a
copy of the old one down to its filesystem's id. It is already copied to, so
nothing is copied, and the stack starts from the snapshot. If the deploy's
converge fails because the attachment has not finished, the poller, which
outlives a docker that could not start, activates within thirty seconds of it
finishing. The old volume is
no longer in state, so `prevent_destroy` does not cover it: delete it by hand
once the board is back. To restore in another zone, set `availability_zone`
to it as well, which replaces the subnet and the instance.

## Check the alarms

```sh
aws cloudwatch describe-alarms --alarm-name-prefix vibescrobble- \
    --query 'MetricAlarms[].[AlarmName,StateValue]' --output table
aws cloudwatch describe-alarms --alarm-name-prefix vibescrobble- --state-value ALARM \
    --query 'MetricAlarms[].[AlarmName,StateReason]' --output text
```

All five read `OK` when the board is healthy. `INSUFFICIENT_DATA` on
`disk-nearly-full` or `memory-nearly-full` means the CloudWatch agent has
stopped reporting. The table in [infra/README.md](../infra/README.md) says
what each alarm means.

The topic delivers only to a confirmed email address:

```sh
aws sns list-subscriptions-by-topic \
    --topic-arn "$(tofu -chdir=infra output -raw alarms_topic)" \
    --query 'Subscriptions[].[Endpoint,SubscriptionArn]' --output text
```

An entry that reads `PendingConfirmation` has not clicked AWS's link.

The board itself:

```sh
curl -s https://vibescrobble.com/health | jq '{status, connected, quietFor, agents, uptime}'
```

## Check a container's memory

Each container has a limit, `*_memory_mb` in `infra/variables.tf`. One that
reaches it is killed and restarted by itself. **On the instance**, what each
uses against its limit, and its peak since it started:

```sh
sudo docker stats --no-stream --format '{{.Name}} {{.MemUsage}}'
for c in $(sudo docker compose ps -q); do
  echo "$(sudo docker inspect --format '{{.Name}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' "$c")" \
    "peak=$(sudo cat /sys/fs/cgroup/system.slice/docker-$(sudo docker inspect --format '{{.Id}}' "$c").scope/memory.peak)"
done
```

`oom=true` names a container killed at its limit. To raise one, change its
default in `infra/variables.tf` and deploy; the instance recreates that
container with the new limit, in place. `TF_VAR_index_memory_mb=768
scripts/deploy.sh` does it for one deploy, and the next deploy without it
puts the default back.