---

id: deploy title: A named version runs somewhere that is not a laptop status: open crates: [didbot-serve, didbot-pds] dependsOn: [agent-accounts] exitCriterion: > A tagged release runs on a host nobody develops on, serving agents over real DNS and TLS, and a documented upgrade replaces it without losing a record. #

deploy #

The security shape is settled and is a constraint on this epic rather than a question for it: the server runs on a host that is not an agent host, the zone credential is scoped to exactly the one hosted zone this deployment holds and sits on the same host as the signing keys — the server writes its own DNS, so it holds its own Route53 credential — and policy is read from the operator's repository. Deployment states those; this is what it takes to satisfy them.

The marketing site is a separate deployment target from the server this epic ships: site is a static build behind S3 and CloudFront, with its own bucket and distribution to provision — see that epic's open items.

Staying up #

• Backups and restore. terraform-aws-didbot-pds/backup.tf puts the data volume — the write-ahead log and the blob store both live on it — under an AWS Backup plan, daily, kept thirty days. Deployment states what a restore recovers (accounts, records, blobs, name holds, ledger history, and the signing keys that make them the same accounts rather than new ones with the same handles), what it does not (every session, the Route53 zone, a certificate older than its own validity, and up to a day of writes), and the five-step drill that would prove it. Open until that drill succeeds against pds.did.bot: a snapshot that has never been read back is not a backup.
• Say what a restart does to sessions holding credentials and to a write in flight, and whether downtime is acceptable or has to be avoided. OAuth grants are in pds.grants and survive one; a sign-in in flight — a pushed authorization request, a consent reference, an authorization code — is in-process memory and does not, so a client mid-sign-in starts over. The restart-persistence issue (3mviwkxahvt2s) decides what else a restart keeps, and where each thing lives.

Reloading, per component #

• Write down what changes without a restart and what does not. Policy is a poll and needs nothing. Hooks need a new session. Prompt text needs a rebuild. Configuration is a file read at startup and needs a restart. The record host depends on its transport: an HTTP one restarts like any service, a stdio one is owned by the harness and comes back when the harness spawns it. The TLS certificate needs no restart either. didbot-tls builds one rustls::ServerConfig for the life of the process, backed by a certificate resolver (didbot_tls::resolver::SwappableCert) that a renewal swaps under an atomic pointer store. A handshake already in progress or a connection already established — the firehose holds these for hours to days — is unaffected; only the next handshake sees the new certificate. Proven with a real TCP/TLS loop in crates/didbot-tls/tests/rotation.rs: a connection held open across a swap keeps working, and a fresh connection made after the swap gets the new certificate. Nothing else about the listener, the router or the process is rebuilt.
• Agent hosts are their own deployment. The node agent, the hook binary and the harness settings all live there and all need updating, and node has the question of how a host knows the hook it runs is the one that was built.

Knowing it is alive #

• Alert on the things that fail quietly: certificate expiry, a policy poll that has not succeeded, clock skew against the attestation window, and disk. alerts is the channel — this server writing a record with its own key — and carries the reason that channel cannot be trusted as an alarm: the failures most worth hearing about are the ones that stop the alert being read. Certificate renewal has the escalation logic now -- didbot_tls::renew::RenewalTracker::alert_level climbs from Ok to Warning after repeated failures and to Critical inside a fixed window of actual expiry, tested without a network in crates/didbot-tls/src/renew.rs, and GET /health's certificates carries every zone's level so terraform-aws-didbot-pds/monitoring.tf's Route53 health check and CloudWatch alarm can read it. GET /health's disk carries the data volume's free and total bytes and a watermark alert, which a second check and alarm in the same file read. The other two failures above, and alerts's own channel, are what is left here.
• Debugging without a redeploy. Log level changeable at runtime. ops-dashboard's refusal log says why a request was refused.
• Capacity. pds-writes carries the load question; deployment is where an answer becomes a number.

Done #

• Deploys restart the container, not the instance. The image digest and the rendered config live in an SSM parameter (terraform-aws-didbot-pds/ release.tf), and user_data names only the parameter, so a new tag no longer replaces the instance. A converger on the instance (terraform-aws-didbot-pds/templates/converge.sh.tftpl) pulls by digest, restarts the unit, requires three healthy answers in a row and otherwise rolls back to the previous release. A stop checkpoints while still serving, then drains within [server] drain_secs, and closes the firehose with 1001; the listener binds before replay. GET /health reports revision. scripts/deploy-pds.sh is the deploy: backup, apply, converge, wait for the commit. Measured locally, a restart refuses requests for 0.2 to 0.45 seconds.
• Upgrades across a log format change, and the rollback. A data directory carries a pds.layout stamp, and didbot_pds::wal::reader is the table of layouts an image reads: the one it writes, and at most the one before it. The table a binary ships holds both. A stamp in the window is read through that layout's reader, and the first boot takes a checkpoint in today's shape, trims the log against it and rewrites the stamp — so an upgrade costs a restart rather than the accounts. One older than the window is refused with a message naming the versions the image does read, and the answer is to run one of those first. Every row in the table has a data directory under crates/didbot-pds/tests/fixtures/layout/, and crates/didbot-pds/tests/layouts.rs opens each one, answers the public surface from its expected.json, and reopens it clean. Forward-only, so the rollback is the snapshot taken before the swap: the boot that upgraded rewrote the directory, and the old image refuses it. Operations's "Upgrading" is the procedure.
• Two roots, one state bucket. infra/pds/ is this deployment and infra/site/ is the did.bot website; each has its own key in the did.bot account's state bucket, so applying one cannot lock or roll back the other. The wire between them is one value: infra/pds/'s name_servers output goes into infra/site/'s pds_zone_name_servers, which writes the NS record at pds.did.bot that makes this zone answer.
• The unit is a container image under one systemd unit. Not binaries and unit files, not an orchestrator; scripts/ is a developer's interface and no part of a deployment. Deployment's "what runs" states it and the three process properties that decide it. The deciding one is that the store is a single-writer log: terraform-aws-didbot-pds/ec2.tf is one instance rather than an autoscaling group, and the unit runs the container under a fixed --name behind an ExecStartPre that removes a previous one, so a restart racing a slow shutdown cannot produce two. That is the unit being careful, and it does not bind an operator with a shell, so the exclusion no longer rests on it: didbot_pds::lock takes an exclusive flock on pds.lock in the data directory before the layout check and holds it for the life of the Durable. A second opener — a hand-run binary against the mounted volume — is refused by name before it reads or writes a byte, and the kernel releases the lock when the holder dies, so a crash leaves no stale lock to clear. One existing test was opening two Durables on one directory and is now scoped; that it had to change is the point.
• A data directory layout, and permissions on one this server did not create. Deployment tables what is in it. Permissions do not depend on who made the directory: didbot_pds::wal::create_dir chmods an existing directory to 0700 on every open rather than trusting the umask that made it, and didbot_tls::storage::create_dir does the same for tls/; files inside are created 0600. Ownership is the one thing this cannot fix — a directory owned by another user fails to write and says so, which is right but is not a diagnosis — and it is the boot script's to get right.
• Secrets, and where each lives. Three, and this process holds all three: the account signing keys, in pds.wal; the ACME account key and the per-zone certificate keys, under tls/; and the Route53 credential, which is not on disk at all — it is the instance role, read over IMDSv2, granted ChangeResourceRecordSets on the one hosted zone and GetChange on changes. Nothing is on the unit's command line. The operator DID is an identifier rather than a credential, and this server holds nothing that can write the operator's repository, which is why policy can constrain it. One compromised host gets an attacker all three; separating the credential from the keys is not a mitigation on a single host and deployment does not offer it as one, because the credential has to be reachable from the process to write DNS-01 challenges. Scope and detection are what bound a compromise.
• The zone layout, in the server rather than in Terraform. The server at the apex, agents at *.<zone> for every zone the deployment is configured with — the recommended shape agent-accounts describes, unenforced but unchanged by this move, and no longer Terraform's to enforce. route53.tf creates the hosted zone, the apex record and the *.<zone> wildcard every agent hostname resolves through, all at the instance's Elastic IP. Which zones exist is the didbot-identity/didbot-pds zone registry's decision: didbot-pds reads [zone] zones, and --route53-zone-id <zone>=<id> for each, writes each zone's _acme-challenge and CAA records through one Route53Dns per zone, and terminates TLS for all of them from one listener — a <zone> + *.<zone> certificate each, since a wildcard matches one label. Minting still happens under the first zone only; that half is capacity's.
• Agent hostnames resolve through the zone's wildcard record. aws_route53_record.agents in route53.tf is *.<zone> at the instance's Elastic IP, beside the apex. The server writes no record per agent, so the records it holds a credential for are the _acme-challenge TXT values and the apex CAA, and the grant in iam.tf is ChangeResourceRecordSets on the one zone plus GetChange. This server never creates a hosted zone — it is handed the zones it may write.
• Certificates: issuance and renewal, in the instance. The server terminates TLS itself over a certificate it obtains and renews in-process, and terraform-aws-didbot-pds/templates/user_data.sh.tftpl's ExecStart passes --tls acme and the hosted zone id the DNS-01 challenge is published into, with acme_environment in the rendered /etc/didbot/didbot.toml beside it; no credential is on the command line or in the file, since Route53Dns reads the instance role over IMDSv2. What a failed renewal does is written down in deployment: the existing certificate keeps being served, the failure is logged with the zone and the expiry it is measured against, and the level escalates before expiry rather than at it. GET /health reports the same levels, and terraform-aws-didbot-pds/monitoring.tf alarms on them — see "alert on the things that fail quietly" above for the failures that channel still covers.
• The certificate hot-reload seam, and TLS termination itself. didbot-tls obtains a wildcard-covering certificate over ACME DNS-01 (instant-acme, on ring — its hyper-rustls feature is deliberately left off because it pulls aws-lc-rs in regardless, checked with cargo tree -i aws-lc-rs), stores it and the ACME account key 0600 inside 0700, and serves it through didbot_serve::serve_tls via a certificate resolver a renewal swaps atomically rather than a listener anything rebinds. What is genuinely exercised without a network: the renewal clock and its escalating alert level, the DNS-01 seam's handling of two identifiers sharing one challenge name, storage permissions, and — the property deployment's firehose case depends on — that an established connection survives a certificate swap while a fresh one sees the new certificate. tests/acme_conformance.rs now drives the real instant-acme client through a full order — directory, nonce, account, order, both authorizations, dns-01 challenges, finalize, certificate — against an in-process fake ACME directory, covering the apex+wildcard grouping landing on a CA-shaped server, a challenge that never validates, a renewal that keeps failing, a crash between publishing a challenge and withdrawing it (and the next order replacing the record it stranded), and two zones ordering concurrently. On 2026-10-14 pds.did.bot's zones served Let's Encrypt production certificates; see web-launch. crates/didbot-tls/src/dns.rs's seam is wired to didbot-dns's multi-value TXT support through DnsProviderAdapter.