Runbook #

How to run this repo. It is applied by hand from a laptop, by one person, with credentials that live in that person's shell and nowhere else.

Requirements #

• OpenTofu >= 1.11 (S3-native state locking; dynamodb_table is deprecated from that release). Terraform >= 1.11 behaves identically here — substitute terraform for tofu throughout.
• The AWS CLI, for the bootstrap check and for deploying the frontend.
• Administrator credentials for the one-time bootstrap. Everything else runs as the apply role. See Credentials.

Credentials #

Nothing is stored in this repo. The two phases use two different credentials.

Bootstrap, once: administrator credentials. Bootstrap creates IAM resources, so lesser credentials fail. How you get them is not recorded in this repo. Put them behind a local profile:

export AWS_PROFILE=<bootstrap profile>

Everything after: the lance-blue-apply role. Any Identity Center session in this account can assume it, if the session's own policy allows sts:AssumeRole. There is no IAM user and no access key. Bootstrap prints a ready-made ~/.aws/config block as the aws_config output; with it in place:

export AWS_PROFILE=lance-blue

Terraform and OpenTofu read AWS profiles natively, including SSO ones.

Anything sensitive that Terraform itself needs is passed at the command line:

tofu -chdir=envs/lance.blue plan -out=lance.blue.tfplan -var 'some_token=…'
TF_VAR_some_token=… tofu -chdir=envs/lance.blue plan -out=lance.blue.tfplan

First time, in order #

1. Confirm the account. Every later step is destructive in the wrong one. The account is 179302349187.

aws sts get-caller-identity

2. Bootstrap. Create the state bucket and the apply role.

cd envs/bootstrap
tofu init
tofu apply -var 'region=us-east-1'

3. Set up the profile. Append bootstrap's aws_config output to ~/.aws/config, set its source_profile to your SSO profile for this account, then check it:

export AWS_PROFILE=lance-blue
aws sts get-caller-identity     # expect assumed-role/lance-blue-apply

The administrator credentials are only needed again to change the apply role itself: the role cannot change its own permissions.

4. Initialise the environment. envs/lance.blue/backend.tf already contains the bucket name. (A backend block cannot use variables, so it is written out.)

tofu -chdir=envs/lance.blue init

5. Check the environment's values. Both variable files are committed and load on their own, so there is nothing to copy: terraform.tfvars holds the constants you edit by hand — account id, domain, task sizes, DIDs — and releases.auto.tfvars.json holds what is deployed, which a deploy rewrites. Neither holds a secret; a secret would be passed as -var or TF_VAR_*.

6. Create the hosted zone and delegate the domain. The site certificate validates through DNS, so the zone must exist and the domain must be delegated before the full apply can finish.

tofu -chdir=envs/lance.blue apply -target=module.dns
tofu -chdir=envs/lance.blue output nameservers

Set the four nameservers at the registrar, by hand. dig NS lance.blue shows when the change is live.

7. Plan, read the plan, apply.

tofu -chdir=envs/lance.blue plan -out=lance.blue.tfplan
tofu -chdir=envs/lance.blue apply lance.blue.tfplan

Day to day #

prek run --all-files          # fmt + validate, via the pre-commit hooks
tofu -chdir=envs/lance.blue plan -out=lance.blue.tfplan
tofu -chdir=envs/lance.blue apply lance.blue.tfplan

Applying the saved plan rather than re-planning means what you read is what runs. The plan file is gitignored (*.tfplan) because a plan can embed resolved variable values, including secrets passed on the command line.

scripts/deploy.sh #

The same thing in one command, and the reason the deploys are visible:

export AWS_PROFILE=lance-blue
scripts/deploy.sh                             # add --yes to skip the prompt

It plans, prints the plan, asks once, applies that plan file, and then posts to lance.blue's Bluesky account when a release id moved — nothing to announce on an infrastructure-only change, and nothing at all when the plan is empty. --dry-run stops after the plan; --no-post applies silently.

The post is built from two outputs and no others: releases and site_url. Both are public knowledge. Bucket names, the instance id and the security group ids are outputs too and deliberately stay out of it, so adding an output can never leak one by accident — posting more means editing the script.

What the post says #

The first post is the release ids that moved and a link card for the site. Bluesky unfurls nothing on its own — a post shows a card because the client that wrote it put one in the record — so the script reads the site's own og: tags, uploads the card image as a blob and writes the card itself. The address is in the card and not in the text; when the card cannot be built, the address goes back in the text as a plain link, because a post announcing a deploy has to link to it.

That address carries the deploy's timestamp as ?d=, so every deploy links to a URL of its own and can be counted separately. Nobody sees it: a card prints the domain, not the query.

Under the first post is the changelog. Every release id ends in the commit it was built from, so the pair either side of an apply is a commit range, and the script reads it out of the sibling checkouts — ../headquarters for both api and site, ../arena, ../helm. LANCE_REPOS points somewhere else if they are not siblings of this repo. A checkout that is missing, or that has not fetched the commits, is skipped without comment: a deploy is not the moment to find out somebody's clone is behind.

A post is 300 characters, which one repo's commits routinely are not, so the changelog rides in the first post when it fits and becomes a thread when it does not. A thread stops at 25 posts and says how many it left out; DEPLOY_MAX_POSTS moves that.

scripts/lib/post.sh decides all of this and talks to nothing. bash scripts/lib/post.test.sh checks the two parts worth checking — the commit ranges and the chunking — against a fixture repository, with no AWS and no PDS.

The app password #

An app password for the lance.blue account, never the account password. It lives in the login keyring, and is stored once:

sudo apt install libsecret-tools        # Debian/Ubuntu, one time
secret-tool store --label='lance.blue app password' \
  service lance.blue key atp-app-password

secret-tool is libsecret's CLI and needs a Secret Service running on the session — gnome-keyring or KWallet. Over ssh, or on a window manager without one, there is nothing to read: set ATP_APP_PASSWORD in the environment instead and the keyring is not consulted. ATP_SECRET_SERVICE and ATP_SECRET_KEY change the attributes if you stored it under other ones.

Not in AWS, on purpose. An SSM SecureString would be the obvious home, next to the api host's secrets, and the apply role can already read those. It is the wrong place: an app password is unscoped — it can read direct messages, delete posts and change the profile — so parking it there would mean whoever takes the apply role also takes the account that speaks for the project. The infrastructure can be rebuilt with an apply. The handle cannot.

Wherever it comes from, the password is never printed and never passed as an argument, so the process list does not show it. Without one the deploy still runs and says it did not post. The account itself lives on a PDS this repo does not run; ATP_PDS points elsewhere if it ever moves.

The releases output is what makes "a release id moved" answerable: the script reads it before the plan and again after the apply, and posts the difference. An apply that predates the output has nothing to compare against, so the first run after adding it says so and posts nothing.

Formatting is prek's job — the terraform_fmt hook runs tofu fmt over the files that changed on every commit (a commit that needed formatting fails; stage the result and commit again); prek run --all-files runs it by hand.

After the first apply #

Tell a repo where to push. Two outputs, because docker login wants the registry host and a push script wants the host plus namespace — each appends the basename of its own IMAGE_NAME:

cd ../arena                      # from an infra checkout
INFRA=../infra/envs/lance.blue

aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin \
      "$(tofu -chdir=$INFRA output -raw ecr_registry_host)"

./scripts/build.sh
REGISTRY=$(tofu -chdir=$INFRA output -raw ecr_push_registry) ./scripts/push.sh

The same REGISTRY works for every repository in the namespace. To see what exists: tofu -chdir=envs/lance.blue output -json ecr_repositories.

Deploy the frontend. Headquarters pushes each build to a release prefix in the site bucket (releases/<id>); serving it is a Terraform change:

# envs/lance.blue/releases.auto.tfvars.json:
#   "site": "<id>"           <- the id alone, not a path
tofu -chdir=envs/lance.blue plan -out=lance.blue.tfplan
tofu -chdir=envs/lance.blue apply lance.blue.tfplan

The apply flips the distribution to the new prefix and invalidates the cache when the path changed — flipping alone does not bust the cache, because CloudFront caches by viewer path. site_cloudfront_url is the cloudfront.net address, which works before the domain is delegated.

Deploy the API. Two SecureString parameters must exist before the api host's first boot — seed them once, by hand, so the secrets never touch Terraform state:

aws ssm put-parameter --type SecureString \
  --name /lance-blue/prod/api/session-secret \
  --value "$(openssl rand -base64 48)"

aws ssm put-parameter --type SecureString \
  --name /lance-blue/prod/api/private-key-jwk \
  --value "$(node -e 'const{generateKeyPairSync}=require("crypto");const j=generateKeyPairSync("ec",{namedCurve:"P-256"}).privateKey.export({format:"jwk"});j.kid=Date.now().toString(36);console.log(JSON.stringify(j))')"

Then a deploy is: push an arm64 image from headquarters (services/api/push.sh prints the tag), set api in releases.auto.tfvars.json, plan and apply. The apply updates an SSM parameter in place; the host polls it and swaps the container within ten seconds - seconds of downtime, no instance replacement. To check on the box:

aws ssm start-session --target "$(tofu -chdir=envs/lance.blue output -raw api_instance_id)"
curl -s https://api.lance.blue/healthz

Setting arena in the same file works the same way, on its own parameter. The api reports that tag at /api/version and the site puts it in the footer, so an arena release restarts the api container too - same ten seconds, still no instance replacement.

MegaMek's art, once per release #

The site draws units in MegaMek's own sprites, which are 39MB of CC BY-NC-SA art in the release rather than anything of ours. They live in their own bucket and are served through the site's distribution at the path they have inside an install:

https://lance.blue/assets/megamek/0.51.0/data/images/units/meks/Atlas.png

So anything that already knows which file it wants out of MegaMek builds the address rather than asking for a mapping. In the bucket the key is 0.51.0/data/images/units/meks/Atlas.png — the top level is release versions and nothing else, and the distribution takes /assets/megamek off at the edge. We upload the unit art and not the rest of the tree; a subset of the same layout costs nothing.

Run this when the pinned MegaMek moves, not on a deploy. The release tarball is 639MB against the 39MB of art it carries, so it is cached under ~/.cache/lance-blue/megamek and reused whenever its digest still matches. Take both arguments from arena's versions.env — MEGAMEK_VERSION and MEGAMEK_RELEASE_SHA256 — so the art in the bucket is provably the release the container runs:

scripts/assets-megamek.sh --dry-run 0.51.0 5d673cda…    # read the diff first
scripts/assets-megamek.sh 0.51.0 5d673cda…

A version's prefix is written once and never rewritten. A bump adds a prefix beside the old one and both stay live, because a match started under the old image has to keep drawing the art it plays with.

Which picture belongs to which design is not in the tree: MegaMek maps it through mekset.txt, and helm art --megamek <install> resolves that to the paths this sync uploads. cargo test -p helm-unitfile -- --ignored in helm is the check that every file the mapping names is one the release ships.

helm's index, when it changes #

The unit index a force-building screen filters on is generated by helm and published beside the art it describes:

helm index --megamek <install> --bridge-dir <dump> --out units.json   # in helm
scripts/assets-helm.sh --dry-run 0.51.0 units.json
scripts/assets-helm.sh 0.51.0 units.json

https://lance.blue/assets/helm/0.51.0/units.json

The version argument is checked against what the file says it describes, and a mismatch stops the upload: an index published under the wrong release is the one failure here that looks fine and is not, because a screen would fetch it and then ask for sprites under a release that never had them.

Unlike the art, this is ours and gets regenerated — a new helm, a corrected figure — so the script invalidates the path it overwrote. That is why it needs the distribution as well as the bucket.

The api reads this file too, once at startup: it is what tells a match report which of MegaMek's pictures each design is drawn with, so a report shows the machines that fought rather than one silhouette per weight class. megamek_index_version is which one it reads, and the api takes the release and the address of the art from inside the file rather than from the variable. Publish the index first and move the variable after — a version with no index published is a report drawing silhouettes, and the api will say so in its log at startup rather than failing.

Starting a match by hand #

Headquarters will do this eventually. Until it exists, this is how to run one task for testing. Everything per-match is a container override — the manifest URL is a bearer credential and does not belong in a task definition revision.

aws ecs run-task \
  --cluster       "$(tofu -chdir=envs/lance.blue output -raw match_cluster_name)" \
  --task-definition "$(tofu -chdir=envs/lance.blue output -raw match_task_definition)" \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={
      subnets=[$(tofu -chdir=envs/lance.blue output -json public_subnet_ids | jq -r 'join(",")')],
      securityGroups=[$(tofu -chdir=envs/lance.blue output -raw match_task_security_group_id)],
      assignPublicIp=ENABLED}" \
  --overrides '{"containerOverrides":[{"name":"arena","environment":[
      {"name":"ARENA_MANIFEST_URL","value":"https://…presigned…"}]}]}'

assignPublicIp=ENABLED is not optional. There is no NAT gateway, so without it the image pull hangs and then times out — which reads as the wrong problem.

Logs land in the group match_log_group names.

What a match cost #

./scripts/matches/cost.sh collect      # ask AWS, file the facts
./scripts/matches/cost.sh report       # price them

collect reads six sources — a live task from ECS, RunTask from CloudTrail, the match log group, Container Insights, the artifacts in S3, and the match's own result.json — and writes what each of them said into a ledger at ~/.local/state/lance.blue/match-costs.json. report prices that ledger against scripts/matches/prices.json and never touches AWS, so a corrected assumption re-prices every match already recorded.

The ledger is deliberately outside the repository. It holds what AWS has since forgotten, which is the one thing here that cannot be collected again, and a checkout or a worktree it lived in would take it when it went. --ledger reads or writes another.

Which cluster, bucket and log group to look at come from envs/lance.blue's outputs, read once per run. scripts/matches/cost.py can be run directly and does the same thing; the wrapper only fails faster when collect has no credentials. MATCH_CLUSTER, ARTIFACTS_BUCKET and MATCH_LOG_GROUP override the outputs one at a time.

Run collect on a schedule. Every source forgets, and they forget at different speeds: about an hour for a stopped task in ECS, a day for Container Insights, seven days for the match logs, thirty for the artifacts, ninety for CloudTrail. Run daily it keeps exact numbers; run monthly it keeps inferred ones. Nothing already in the ledger is downgraded by a later run.

Totals and billed times are coloured by how much they should worry you: amber at ten cents or an hour, red at a dollar or a day, against a match that finishes costing about a cent and a half and playing for under ten minutes. The two agree closely, because cost is very nearly linear in billed time — a row amber on one and not the other is worth a look. Both scales are the CostThreshold and TimeThreshold enums at the top of cost.py and move by editing them; --color never turns the colour off, and so does piping the output anywhere or setting NO_COLOR.

The src column says where each match's billed window came from, as start/end — ecs is the task's own record, trail/s3 is an exact RunTask and an end taken from the last artifact upload. --explain <match-id> prints one match line by line, including which sources were still readable for it.

./scripts/matches/cost.sh collect --hours 2160    # backfill CloudTrail's 90 days
./scripts/matches/cost.sh report --explain 5fe10a12
./scripts/matches/cost.sh report --format csv > costs.csv
./scripts/matches/cost.sh prices --refresh        # re-read the AWS price list

Prices come from the AWS Price List API and are committed, so a report is reproducible and a price change is a diff. The assumptions beside them — what a Spot task costs, how long provisioning takes, how often artifacts are read back — are hand-maintained and a refresh leaves them alone.

scripts/matches/cost.py documents what is counted and what is not. The short version: Fargate is around 95% of a match, and serving the report pages afterwards is the one real per-match cost that is measurable and not measured, because it lives in CloudFront's access logs and needs Athena to read.