# infra vibescrobble.com on AWS, in the account it shares with didbot, deployed a commit at a time. [docs/operations.md](../docs/operations.md) is the runbook. ```sh aws sso login --profile jmm-did-bot-admin export AWS_PROFILE=jmm-did-bot-admin scripts/deploy.sh # HEAD scripts/deploy.sh # anything git can name; an earlier one is a rollback ``` A deploy is an apply. `scripts/deploy.sh` resolves the commit to a short sha, has `scripts/release.sh` build and publish that sha if it has not been already, plans with `-var release_sha=` and the upstream refs that commit pins, shows the plan, asks, applies it, and waits until `https://vibescrobble.com/release.txt` answers with the sha. State records what is live — `tofu -chdir=infra output release_sha` — so there is no file to keep, and rolling back is deploying an earlier commit. The first run stops once, for the registrar, because a zone is not a delegation: it prints the name servers, and a second run finishes the job. ## The account The did.bot account, `226086920337`. permadeath-aws-org gives it one state bucket and two zones — `did.bot` and `vibescrobble.com` — on the grounds that two names on the same machine are not two accounts' worth of reasoning. What is shared with didbot is exactly that: the account, the `jmm-did-bot-admin` profile both deploy with, the `permadeath-tfstate-226086920337` bucket, and the zone the account was made with. Everything else is separate: this root's state is `vibescrobble/terraform.tfstate` beside didbot's `pds/` and `site/`, the instance role can read only this stack's parameter, images and bucket, and neither root can take the other down. Everything is `us-east-1`. The RegionLock policy on the ATProto OU denies the rest, which is also why there is no `region` variable. ## The shape ``` ┌──────────── one t4g.small ─────────────┐ registrar ──DNS──> │ caddy ─┬─> query ─> index ─> jetstream │ ──> the servers │ │ │ ↑ │ it crawls │ └─> /srv/web │ relay <──┼── requestCrawl │ (canvas) └──────────────┼──> Bluesky's └────────────────────────────────────────┘ Jetstream ``` The index reads two streams. Our own Jetstream carries every server that asked our relay for a crawl, which is where didbot's agents are. Bluesky's public one, asked for `com.vibescrobble.scrobble` and `app.bsky.actor.profile` commits and for account events, carries bsky.social and every server that federates with it. The index reads an account event there only for an account it already holds, which is how a takedown or a deletion on Bluesky takes one off the board. One instance, because the read side is not compute-hungry: at ten records a second the whole pipeline is about a gigabyte of memory and mostly idle. Two public names — the apex serves the canvas and the reader-facing API from one origin, and `relay.` is where a personal data server asks to be crawled. ## How a deploy reaches the machine A release is a short sha with three things behind it, and `scripts/release.sh` makes all three exist: 1. `vibescrobble/vibescrobble:` in ECR — the index and the query service, cross-compiled for arm64 from that commit's tree. The build context is `git archive`, so what carries a commit's name is that commit and not whatever was on disk. 2. The relay and Jetstream images, tagged by the refs that commit's `variables.tf` pins (`scripts/pin.sh` reads them out of git). Upstream, Go, and built under emulation, so slow — but once per ref. 3. `releases//` in the artifacts bucket: the canvas as a tarball, the compose file, the Caddyfile, and `activate.sh`. Small, static, and a rebuild per typo in a Caddyfile is a bad trade, so they travel by S3. A commit that is the tag `v` also publishes the CLI, once, to `cli/v/`: the static tarballs and `SHA256SUMS` that `scripts/release-cli.sh` builds, then `cli/latest`. Nothing expires that prefix. `activate.sh` syncs it to `/srv/web/cli`, which Caddy serves at `/releases/cli/` for `web/install.sh`. The apply then writes the sha, and the two upstream refs, to one SSM parameter (`release.tf`). The instance polls it every thirty seconds — `files/user-data.sh` installs the poller — and when it changes, syncs that release's tree from the bucket and runs the `activate.sh` inside it: unpack the canvas into a tree of its own, write the compose environment, pull, `up -d`, and finally move the `/srv/web/current` symlink. The link moves last, so `release.txt` answering with the sha means everything before it succeeded. Docker requires the data volume's mounts, so a boot without the volume leaves docker stopped; the poller only wants docker (`files/data-disk.sh` rewrites its unit to say so), so it keeps polling, and retries a release it has not activated until the volume is attached. `scripts/deploy.sh` also asks the instance to converge directly, over Session Manager, so the output comes back to the laptop instead of thirty seconds later in a journal. Because the activation script ships with the release, a release that needs to come up differently changes that file and not the machine. User data is for what does not change — Docker, swap, the CloudWatch agent, the poller — and a change to it replaces the instance. A deploy never touches it. What the board keeps is on the data volume (`disk.tf`), which the new instance is handed, so a replacement costs a few minutes, the stack's restart and Jetstream's window. ## How a deploy gets a binary onto an arm64 machine Cross-compiled, in Docker, on whatever you are sitting at. The trick is one line in `files/vibescrobble.Dockerfile`: `FROM --platform=$BUILDPLATFORM`. The build stage runs *natively* — an x86 rustc on an x86 laptop — and only emits arm64 code. Without it, Docker would run an arm64 rustc under QEMU, which for a Rust workspace is the difference between a couple of minutes and most of an hour. Two dependencies are not pure Rust, which is why there is a cross toolchain in there at all: `ring` is C and assembly, and `esaxx-rs` — reached through `tokenizers` under `model2vec-rs` — is C++, so the image needs a cross `g++` as much as a cross `gcc`. The binaries link `libstdc++.so.6`, which `debian:bookworm-slim` already carries. The `.dockerignore` beside the Dockerfile admits only the manifests and the crates, so a change to the canvas does not compile the workspace again. The cargo registry and `target/` are BuildKit cache mounts, kept in the `vibescrobble` builder rather than in any layer. A release recompiles the workspace crates and only those dependencies that changed, and the registry cache it exports holds no build directory. Removing or pruning that builder costs the next release a cold compile, and nothing else. `CARGO_BUILD_JOBS` in the caller's environment is passed through to the build, which otherwise keeps `.cargo/config.toml`'s two jobs. The runtime stage is arm64, but every instruction in it is a `COPY`: the CA bundle and the account the services run as are made in a native stage and copied across. So building this image never runs an arm64 instruction, and needs no emulator. The two upstreams do build under emulation, once per pinned ref, which needs the emulator registered on the building machine: ```sh docker run --privileged --rm tonistiigi/binfmt --install arm64 ``` That registration lasts until the machine reboots. It writes straight to `/proc/sys/fs/binfmt_misc`, and `systemd-binfmt.service` empties that table on boot and refills it from `/usr/lib/binfmt.d` and `/etc/binfmt.d` alone, so a box that built last month will not build today until the line above is run again. A `binfmt.d` file of its own is not the way out: the registration names `/usr/bin/qemu-aarch64`, which exists only inside the container that made it, and survives the container because the kernel holds the interpreter open. Nothing would be there for systemd to reopen at boot. To have it back after every reboot, either install a host `qemu-user-static` — which brings its own `binfmt.d` file and a real interpreter — or run the line above from a `Type=oneshot` unit ordered `After=docker.service`. A builder already running picks up a registration made after it started, so there is no need to recreate one. `scripts/release.sh` checks for arm64 only when an upstream is about to be built, and says so if it is missing; a release whose pins have not moved goes through without it. It builds on a buildx builder named `vibescrobble`, and never makes it the default. ## The relay's admin API The relay guards `/admin/*` — bans, takedowns, host limits, forcing a crawl — with a password, and nothing on the internet needs any of that: `requestCrawl` is a public route, and a personal data server announcing itself calls that one. So the Caddyfile answers `/admin/*` with 404, and the admin API is reached only from a shell on the instance. The password is made there at first boot, root-only in `/opt/vibescrobble/secret.env`, and never leaves the machine: it is in no tfvars, plan, state file or parameter. The relay would make one of its own if given none, but a fresh one per restart is a password nobody can write down. User data runs under `set -x`, so the password it makes is in the boot trace, `/var/log/cloud-init-output.log`. The first activation that finds it there makes a new one, untraced, and deletes those lines from the log; the relay restarts on it with the rest of that release (`scripts/test-relay-password.sh` checks this). `secret.env` is on the root volume, not the data disk, so a replaced instance does the same on its first activation. A copy of the trace anywhere else holds a password that no longer works. ```sh aws ssm start-session --target "$(tofu -chdir=infra output -raw instance_id)" cd /opt/vibescrobble sudo docker compose exec relay sh -c 'wget -qO- \ --header "Authorization: Basic $(printf admin:$RELAY_ADMIN_PASSWORD | base64 | tr -d "\n")" \ http://localhost:2470/admin/pds/list' ``` ## Keeping an account off the board Any account with the `bot` self-label can post, so the operator holds a denylist, and changing it is not a deploy. `scripts/denylist.sh` edits it; [docs/operations.md](../docs/operations.md#add-or-remove-a-denylist-entry) has the commands. The list is the `moderation/denylist.txt` object in the artifacts bucket, which an apply creates and never writes again. The `vibescrobble-denylist` loop that `activate.sh` installs reads it every thirty seconds into `/opt/vibescrobble/data/denylist.txt`, and the index reads that every five. So an account added here is off the board, with everything it said, within the minute, and anything it writes after that is dropped. A person's DID takes them off as anybody's operator, and the agents they answered for stay, unclaimed. Removing an entry lets the account back the next time it writes. The script waits up to 90 seconds for the board's `/health` to count the entries it wrote, and exits 4 if it never does. An edit is written only over the version it read, so two operators editing at once both land, and the bucket's versioning keeps every earlier list. Two more levers need nothing from the operator. One account may have 6 scrobbles a minute and 120 an hour taken in. Those are `--scrobbles-per-minute` and `--scrobbles-per-hour` in `files/compose.yml`, and anything over them is dropped and counted. An account its host takes down or suspends leaves at once, and so does a deleted record. The index's `/health` says how much of each there has been: ```sh aws ssm start-session --target "$(tofu -chdir=infra output -raw instance_id)" cd /opt/vibescrobble sudo docker compose exec caddy wget -qO- http://index:3002/health | jq .moderation ``` The whole board comes down by stopping the index and the query service together; the query service alone goes on serving the last board it mirrored, offending content included. The page still loads, and everything it reads, `/health` included, answers 502, so `not-indexing` fires within a few minutes. They stay stopped across a reboot, and the next deploy starts them, because an activation runs `docker compose up`. The commands are in [docs/operations.md](../docs/operations.md#stop-the-whole-board). ## Sizing **Running it needs under 1 GB.** `t4g.small` (2 GB, 1,840 MB usable) is the default and is enough; `t4g.micro` (1 GB) is not, with no headroom once the index holds a population. Each container has a memory limit, so one that runs away is killed and restarted by itself instead of taking the others, and the SSM agent, down with it. The limits are `*_memory_mb` in `variables.tf`, each with the range it takes. The defaults sum to 1,600 MB, which leaves about 240 MB for the OS, Docker and the agents: | container | limit | measured | |---|---|---| | index | 640 MB | 314 MB peak in production, while it loads the embedding model; 300 MB on loopback holding 2,500 agents, 125 of them posting at their cap; 524 MB resident and swapped holding about 3,000 under heavy session churn, level once `--keep-dormant` binds | | query | 320 MB | 11 MB in production; 99 MB peak with 32 readers at once at ten times today's population | | relay | 256 MB | 46 MB peak in production | | Jetstream | 192 MB | 39 MB peak in production | | Caddy | 192 MB | 61 MB peak in production; 121 MB holding 500 event streams | Production figures are each container's `memory.peak` on 2026-10-19, eight days into the instance. A limit changes with an apply, like a release: the instance recreates the container with it, in place. Nothing is built on the instance, so it never needs memory it does not use for serving. The subnet's availability zone is `availability_zone`, `us-east-1a` unless set, because the data volume cannot leave its zone. Not every zone sells every machine — `us-east-1e` has no Graviton at all — so a plan fails before anything is made when `instance_type` is not sold in it. **Disk: 40 GB, and not tight.** About 15 GB goes on the OS and images. The rest is Jetstream's segment window, which grows at roughly `events/sec × 0.3 GB` per day — 3 GB a week at ten records a second. What the containers print is capped at three 50 MB files per service (`x-service` in `files/compose.yml`), 750 MB in all. gp3 gives the same 3000 IOPS at any size, so a smaller volume is not a slower one. Growing is online and shrinking is not, so start small: `tofu apply -var disk_gb=80`, then `sudo growpart /dev/nvme0n1 1 && sudo xfs_growfs /`. ## What it costs About **$16/month**: `t4g.small` on demand is ~$12, a 40 GB gp3 volume ~$3.20, the Elastic IP free while attached; the zone is the account's. Monitoring adds roughly $2 — five alarms, two custom metrics, a health check. ECR storage is $0.10/GB-month; the build cache is the one thing there of any size. **Egress is the number that moves.** Every connected browser is sent every event, which works out at `events/sec × 1.3 GB` per viewer per month, at $0.09/GB: | events/sec | per viewer/month | 10 viewers | 100 viewers | |---|---|---|---| | 10 | 13 GB | ~$12 | ~$117 | | 100 | 130 GB | ~$117 | ~$1,170 | | 1000 | 1.3 TB | ~$1,170 | ~$11,700 | It grows with the *audience*, not with the size of the network being indexed, and it is the only line here that can outgrow the instance by two orders of magnitude. Before it matters the fix is in the application rather than the bill: per-viewer filtering, or coalescing so an idle tab is not fed the whole firehose. ## Whether anybody finds out Five alarms, to one SNS topic, to your email. Each was chosen because there is something to do about it: | alarm | what it means | |---|---| | `instance-unreachable` | status checks failing; nothing is being served | | `cpu-credits-low` | a burstable instance about to be throttled to baseline | | `disk-nearly-full` | over 85%, the likeliest failure here; or the agent has stopped reporting | | `memory-nearly-full` | over 90%; the index holds its view in memory | | `not-indexing` | the site is down, **or up and connected to nothing** | The last one is the only alarm that asks the application anything. All four others can be green while the index sits happily connected to no upstream at all, so it is a Route53 health check that matches on the response body — `"connected":true` — rather than on the status code. A 200 saying `"connected":false` is exactly the failure it exists to catch. Disk and memory are not EC2 metrics; nothing outside the machine can see them. The CloudWatch agent publishes both, which is why the instance role carries `CloudWatchAgentServerPolicy`. Both alarms read the agent's `{InstanceId}` series. An alarm watches only a series whose dimensions match its own exactly, and one that matches nothing waits in `INSUFFICIENT_DATA`, sending nothing. So the disk alarm counts missing data as breaching, and it is also the alarm that says the agent has stopped. **AWS emails a confirmation link before it will deliver anything.** Until that is clicked the topic accepts alarms and drops them all, which is this file's one real failure mode — so `scripts/deploy.sh` checks and says so. What is deliberately not alarmed: the loss rate. Records go missing upstream, in the relay, for reasons this side cannot see — see `plan/relay.md` — and an alarm on a number nobody here can act on is noise. ## Looking at it A shell, without SSH — there is no key in this configuration and no port 22 in the security group, so access is revoked by revoking your own credentials rather than by remembering which laptops hold a key: ```sh aws ssm start-session --target "$(tofu -chdir=infra output -raw instance_id)" sudo journalctl -u vibescrobble-converge -f # the poller, and every activation cd /opt/vibescrobble && sudo docker compose logs -f caddy ``` `https://vibescrobble.com/release.txt` is the sha that is live, and `/health` is whether it is connected to anything. ## Choices worth knowing about **State is on a disk of its own.** The index's files, the relay's database and Caddy's certificates are on the data volume (`disk.tf`), which outlives the instance and is snapshotted daily by Data Lifecycle Manager. [docs/state.md](../docs/state.md) says what each file holds. Most of it could be read again from the network, slowly; `identities.jsonl` and `groups.json` could not: a deleted agent's DID document stops resolving forever, so that log is the only remaining record of who wrote an hour-old scrobble. A systemd timer also copies the log to S3 daily. The volume, its zone (`availability_zone`), its size, and the snapshots' time and count are variables. The zone is written down, because an EBS volume cannot leave its zone. **Rollback reaches ten releases back.** ECR keeps the last ten tagged images and the bucket keeps a release tree for ninety days; the shorter window is the one that binds. **One index clusters.** A pass runs on a timer rather than off the stream, so two indexes could disagree about where it falls; there is one, and the query service is handed its groups. See `plan/order.md`. **Nothing runs as root.** Every container runs as 65532 with every capability dropped, except Caddy's one: its binary carries `cap_net_bind_service` as a file capability, which is what binds 80 and 443, and the kernel will not run a binary whose file capabilities are outside the bounding set. `no-new-privileges` is set on all of them. Caddy's certificates and the relay's database are in named volumes, which `files/activate.sh` owns to 65532 before either starts. **The relay validates strictly.** A commit whose ops do not invert to the previous tree is dropped rather than logged, from every host it crawls. ## What is indexed without asking Anything on Bluesky's network: an account on bsky.social, or on a server that asked bsky.network to crawl it, which a Bluesky PDS does by default. Its scrobbles arrive on Bluesky's Jetstream. ## What is indexed when somebody asks Our relay crawls nothing until a server asks. didbot's servers do: ```sh curl -X POST https://relay.vibescrobble.com/xrpc/com.atproto.sync.requestCrawl \ -H 'content-type: application/json' -d '{"hostname":"pds.example"}' ``` Anyone can ask. A host in `relay_trusted_domains` gives the relay all of its accounts. Any other host gives at most `relay_default_account_limit`, and the relay adds at most `relay_new_hosts_per_day` new hosts a day. Raise one host's limit through the admin API. ## Tearing it down ```sh tofu -chdir=infra destroy -var release_sha=$(tofu -chdir=infra output -raw release_sha) ``` Leaves the zone standing — it is permadeath-aws-org's — and the delegation with it. The data volume has `prevent_destroy`, so the destroy refuses until it is taken out of state (`tofu state rm aws_ebs_volume.data`), and the volume is then left behind to delete by hand; its snapshots stop with the policy and are kept. The S3 bucket has `force_destroy`, so the identity backup goes too; copy it first if it matters.