Fork of atproto-crates for SCN forked from ngerakines.me/atproto-crates
atproto pds permissioned-data rust
atproto-crates docs deploy-entryway.md
29 kB · 251 lines
Markdown
at main

Deploying atproto-entryway #

This guide puts an entryway in front of one or more PDS instances. It sits beside deploy-railway.md, which deploys a PDS, and assumes you have read it. The crate's README is the reference for every route, command and setting. This page is the order to do things in, and why.

What you are deploying #

An entryway is the account side of a cluster of PDSes. It holds credentials, sessions, OAuth grants, the map of which PDS each account lives on, service handles, and the PLC rotation key every hosted DID lists. Each PDS behind it keeps the data side: repositories, blobs and repository signing keys.

  • One process, one volume. The entryway is a single binary. Its state is one SQLite database, entryway.sqlite, plus three key files, all under ENTRYWAY_DATA_DIRECTORY (/data in the image). Run exactly one instance. A second instance on a copy of the volume issues tokens and signs PLC operations from a diverging copy of the account table.
  • Members are PDSes in entryway mode. atproto-pds with the four PDS_ENTRYWAY_* settings, or Bluesky's reference @atproto/pds with the same four. A member hands sign-in, account changes and handle changes to the entryway, and verifies the entryway's tokens with its public JWT key.
  • Users talk to the entryway to sign up, sign in, approve OAuth clients and manage their account at /account. They talk to their PDS for everything else, with the entryway's tokens.

If the entryway is down, existing access tokens keep working at every PDS until they expire, and so do reads and writes. Sign-in, refresh, signup and handle changes fail until it is back.

1. Decide before the first boot #

Four choices are hard or impossible to change later.

  • The public URL. ENTRYWAY_PUBLIC_URL is the OAuth issuer, shown on every sign-in page and stored by every OAuth client. Every member PDS compares it to each token's iss byte for byte, so give it as an HTTPS origin with no path and no trailing slash, e.g. https://account.example.com.
  • The service DID. The default is did:web: plus the URL's host, served by the entryway itself at /.well-known/did.json. Keep the default. A reference PDS verifies the entryway's provisioning tokens by resolving this DID, so it has to resolve. A did:plc service DID works only once you have created it at PLC yourself.
  • The passkey RP ID, if passkeys are wanted. ENTRYWAY_PASSKEY_RP_ID turns passkeys on. The browser binds every passkey to the RP ID it was created under, so once anyone enrols the value cannot change without every holder re-creating theirs; serve refuses to start under a different one. It must be the issuer's host (account.example.com) or a parent domain of it (example.com). The host keeps every other name under the domain out of it, but renaming the entryway would strand every passkey. The parent domain survives a rename to another host under it, at the cost that any page under that domain can start a passkey ceremony for it, though assertions are accepted from the issuer's origin alone. A member PDS's own passkeys (PDS_PASSKEY_RP_ID) do not carry over when it moves behind an entryway (§9).
  • The handle domains. ENTRYWAY_SERVICE_HANDLE_DOMAINS must equal PDS_SERVICE_HANDLE_DOMAINS on every member. Choose, per domain, who answers /.well-known/atproto-did for its names (ENTRYWAY_HANDLE_BACKEND). With well-known, the domain's wildcard record points at the entryway and it answers by Host. With highport, the wildcard points at Highport and the entryway keeps the reservations in step (see Highport).

2. Keys #

The entryway has three private keys and refuses to start if any two are the same:

Key File If it leaks
JWT (K-256) keys/jwt.key Tokens can be minted for every account on every PDS. Rotate it (README).
PLC rotation, the hot key keys/plc-rotation.key Any hosted DID can be rewritten. The cold key below is the way back.
Service keys/service.key The entryway can be impersonated to its members' provisioning checks.
entryway keys generate --kind all

On a platform where the container has no shell, such as Railway with the production image, generate the keys on your own machine and upload them to the volume before the first boot. A volume path is relative to its root, so the container's /data/keys/jwt.key is /keys/jwt.key to Railway:

mkdir -m 700 keys
entryway keys generate --kind jwt --out keys/jwt.key
entryway keys generate --kind rotation --out keys/plc-rotation.key
entryway keys generate --kind service --out keys/service.key
railway volume files --volume <volume> upload keys /keys
# The PDS_ENTRYWAY_* settings each member needs, from the same files,
# with the values the server will have:
entryway keys show --data-directory . --public-url https://account.example.com \
  --plc-recovery-key did:key:… --service-handle-domains example.com
rm -r keys

The upload may not keep each file's owner-only mode; the entryway warns about that at startup, and in a container that runs only the entryway it does no harm. Alternatively, boot the debug image (see Running the operator commands) and run entryway keys generate --kind all in its console, so the private keys never leave the volume.

The cold recovery key. Every hosted DID lists an offline recovery key above the hot key, so that a stolen hot key can be overridden within PLC's 72-hour recovery window. The entryway only ever sees its public half. Generate it on a machine that will not hold it afterwards, keep the private half offline (a password manager entry plus paper), and decide in advance who can perform the recovery:

atpdid key generate k256       # keep the private key offline; configure the public did:key

Set the public did:key as ENTRYWAY_PLC_RECOVERY_KEY. It is required in production, and it is written into every genesis the entryway signs, so it must be in place before the first signup.

When an identity leaves. signPlcOperation will not drop the recovery key from an identity that stays in the cluster, but it will for one that is moving to a host outside it: a migration signs the new host's recommended credentials, which name only that host's key. The departure is still reversible. The entryway's hot key signs it, and the recovery key sits above the hot key in the operation it replaces, so for 72 hours after the new host submits it the recovery key can sign an operation that nullifies it. Each departure mails the holder ("Your account's identity is moving") and writes a plc.signed row to the database's audit_event table whose detail names the destination (leaving_for). If a holder says they did not ask for one, within those 72 hours:

  1. Fetch the DID's audit log from PLC and find the departing operation and the one before it, which this entryway signed and which lists the recovery key above the hot key.
  2. With the recovery key, sign an operation whose prev is that earlier operation's CID, restoring its services and keys, and submit it to PLC. PLC nullifies the departure in its favour.
  3. Change the account's password and sign it out everywhere: whoever signed the departure held a session and could read the account's mail.

3. Build and run #

Dockerfile.entryway at the workspace root builds the entryway binary with clap, hickory-dns and smtp, into a distroless image whose CMD is serve. It listens on 0.0.0.0:4900, or $PORT when the platform sets one. Add metrics to the build argument ENTRYWAY_FEATURES to get the Prometheus exporter.

docker build -f Dockerfile.entryway -t atproto-entryway .
docker build -f Dockerfile.entryway --build-arg ENTRYWAY_FEATURES=clap,hickory-dns,smtp,metrics -t atproto-entryway .

The service needs:

  • a persistent volume at /data;
  • one replica, with the old container stopped before the new one starts (on Railway: numReplicas = 1, overlapSeconds = 0, the same reasons as the PDS's railway.toml);
  • a health check on /_ready, which answers 503 when the database does not;
  • ENTRYWAY_PRODUCTION=true, under which the server refuses to start without the recovery key, the admin password file, the denylist pepper file, an existing data directory, and HTTPS URLs.

On Railway the repository's railway.toml belongs to the PDS. railway.entryway.toml is the entryway's: set the entryway service's config-as-code path to it. It builds Dockerfile.entryway, checks /_ready, runs one replica, and stops the old deploy before starting the new one. Attach a volume at /data.

Environment #

The README's configuration table lists every setting. A production deployment sets at least these:

ENTRYWAY_PRODUCTION=true
ENTRYWAY_PUBLIC_URL=https://account.example.com
ENTRYWAY_SERVICE_HANDLE_DOMAINS=example.com
ENTRYWAY_PLC_RECOVERY_KEY=did:key:zQ3sh…            # the cold key's public half
ENTRYWAY_ADMIN_PASSWORD_FILE=/data/secrets/admin
ENTRYWAY_DENYLIST_PEPPER_FILE=/data/secrets/pepper
ENTRYWAY_EMAIL_SMTP_URL=smtps://user:pass@smtp.example.com:465
ENTRYWAY_EMAIL_FROM=Example accounts <accounts@example.com>

Secrets are files, not variables, so they never show in --help or a process listing. Write each with a long random value, e.g. openssl rand -hex 32 > /data/secrets/admin.

First boot #

entryway check     # every setting and key, all problems at once
entryway serve
curl https://account.example.com/.well-known/did.json
curl https://account.example.com/xrpc/com.atproto.server.describeServer

When a domain uses Highport, check also mints a controller token, without calling Highport, and fails if it cannot. serve makes the same check but only warns, because on a first boot the controller may not be an account here yet.

Running the operator commands #

The pds, ops, reconcile, handles and accounts commands, and keys show, run one of two ways.

From your own machine, through the admin API. Give any of them --admin-url with the entryway's public URL, and the operator's admin password in ENTRYWAY_ADMIN_PASSWORD (or a local copy of the file in ENTRYWAY_ADMIN_PASSWORD_FILE). The command calls the matching com.atproto.admin.* method and prints what it would have printed beside the server:

export ENTRYWAY_ADMIN_URL=https://account.example.com
export ENTRYWAY_ADMIN_PASSWORD_FILE=./entryway-admin   # a local copy of the server's file
entryway pds list
entryway ops list --failed
entryway reconcile
entryway accounts passkeys list did:plc:…

This is the routine way on Railway. No redeploy, no shell beside the keys, and every call is audited with "via":"api". The methods take the operator's password only; a member PDS's admin password, which two passthrough methods accept, is refused. pds add and pds set read the secret files you name on your own machine and send their contents; the entryway writes them under /data/secrets/pds/ with owner-only permissions. The API never takes a path, because a path would let anyone holding the admin password register a "PDS" whose callback secret is /data/keys/plc-rotation.key and receive it on the next callback.

When a holder loses the device a passkey was on, entryway accounts passkeys list <did> shows their passkeys and entryway accounts passkeys remove <did> <credential-id> --end-sessions removes one and signs out everything it may have signed in, here and at their PDS. They are mailed either way. --all removes every passkey, and --keep-sessions leaves sessions alone without asking.

Beside the server, on the database. Without --admin-url the commands open entryway.sqlite directly, so they run where the volume is mounted. On a host with a shell, run them in the entryway's container. On Railway that is not possible with the production image: it is distroless, the service console needs a shell, and a volume attaches to one service, so no second container can open the database.

Dockerfile.entryway-debug is the way in when you need that: the API cannot be reached, or you want sqlite3 on the database, or the keys generated where they will live. It is Dockerfile.entryway with a Debian runtime stage: the same binary and features, the same /data and listener, still serving, plus bash, sqlite3, curl and jq. To use it:

  1. Point the service at it: set dockerfilePath = "Dockerfile.entryway-debug" in railway.entryway.toml, or override the Dockerfile path in the service's settings, and deploy.
  2. Open the service's console and run the commands, e.g. entryway pds list. The settings come from the service's variables, as they do for serve. Read the database with sqlite3 -readonly /data/entryway.sqlite: a writing connection from the console can hold a lock the server needs.
  3. Point the service back at Dockerfile.entryway and deploy.

Each swap is a redeploy, and with one replica and no overlap the entryway is down while the new container starts: sign-in, refresh, signup and handle changes fail for that time, as they do during any deploy. Treat the debug image as break-glass. A shell beside the server reaches /data/keys/: the PLC rotation key, which can rewrite every hosted DID, and the JWT key, which mints tokens every member PDS accepts. Deploy it for as long as the job takes, and go back.

4. Pair each PDS #

On the entryway, print what each member needs:

entryway keys show

On the PDS, set what it printed: PDS_ENTRYWAY_URL, PDS_ENTRYWAY_DID, PDS_ENTRYWAY_JWT_VERIFY_KEY_K256_PUBLIC_KEY_HEX, PDS_ENTRYWAY_PLC_ROTATION_KEY and PDS_ENTRYWAY_PLC_RECOVERY_KEY. Also create a shared peer secret and give it to the PDS as PDS_ENTRYWAY_PEER_SECRET_FILE. Keep PDS_DID_PLC_URL and PDS_SERVICE_HANDLE_DOMAINS equal to the entryway's, and leave PDS_HIGHPORT_* unset: in entryway mode Highport is the entryway's job. Restart the PDS. crates/atproto-pds/README.md says what changes on the PDS.

On the entryway, register it. The callback secret is the PDS's admin password, which the entryway uses to tell the PDS about handle, status and deletion changes:

entryway pds add --did did:web:pds.example.com --url https://pds.example.com \
  --callback-secret-file /data/secrets/pds-example-admin \
  --peer-secret-file /data/secrets/pds-example-peer --default
entryway pds check

pds check confirms the PDS's identity, that its protected-resource metadata names this entryway, and that the admin password works. Run these with --admin-url from your own machine, or beside the server (Running the operator commands). With --admin-url the two secret files are read where you run the command and their contents sent. Beside the server they are read in that container, so there they must already be on the volume: upload them first, e.g. railway volume files --volume <volume> upload pds-example-admin /secrets/pds-example-admin, and give their /data/secrets/… paths.

Without the peer secret every request a PDS forwards is placed at the PDS's own address, so all of that PDS's users share one sign-in and password-reset rate limit.

A reference @atproto/pds member works with the same four settings, with three differences. It verifies the entryway's provisioning tokens by resolving PDS_ENTRYWAY_DID, which therefore has to resolve. Its resolver refuses IP-literal URLs. And it takes a single JWT key, so a JWT key rotation is a hard switch for its users.

5. Open signups #

Invites are required by default (ENTRYWAY_INVITE_REQUIRED). Mint them with the admin password:

curl -u "admin:$(cat /data/secrets/admin)" -H 'content-type: application/json' \
  -d '{"useCount": 1}' https://account.example.com/xrpc/com.atproto.server.createInviteCode

To place a code's accounts on one member, add its DID as pds: -d '{"useCount": 1, "pds": "did:web:pds.example.com"}' (createInviteCodes takes it too, for every code it mints). The PDS must be registered; it need not be accepting signups, so this is how chosen accounts go to a closed one. admin.getInviteCodes shows each code's pds.

New accounts go to the PDS an invite names, then the default, then any PDS open to signups, by weight (entryway pds set … --accepting-signups, --weight). People sign up at https://account.example.com/account/signup, or through any app that speaks createAccount to the entryway.

To require a policy, set ENTRYWAY_POLICY_SET_ID (say 2026-10-05-terms, or a dated content hash) and ENTRYWAY_POLICY_URL together on the entryway, never PDS_POLICY_* on a member: behind an entryway the member never signs anyone up or in, and warns at boot if they are set. Turning a policy on changes three things at once:

  • Signup over XRPC stops. createAccount answers 400 PolicyAcceptanceRequired naming /account/signup, which is the only way in from then on.
  • Every existing account is asked on its next sign-in. Apps get PolicyAcceptanceRequired from createSession and access_denied from OAuth until the holder accepts at /account. Sessions already open keep refreshing.
  • The acceptance is written to the holder's repository, so a member PDS that cannot take writes leaves holders unable to accept. The page says so and nothing changes.

A revised set of documents is a new identifier, and everyone is asked again.

6. Mail #

Without ENTRYWAY_EMAIL_SMTP_URL, each message is logged by recipient and subject and dropped. Nobody can then confirm an address, reset a password or delete an account. The SPF and DKIM setup in deploy-railway.md's Mail and DNS sections applies to the entryway's sending domain unchanged.

7. Highport (optional) #

For a domain whose names Highport answers, set ENTRYWAY_HANDLE_BACKEND=example.com=highport with ENTRYWAY_HIGHPORT_SERVICE_URL and ENTRYWAY_HIGHPORT_CONTROLLER_DID. The controller is the account that owns the base at Highport. It stays an ordinary account of this entryway, hosted on a member PDS, and the entryway reaches Highport through that PDS's getServiceAuth, so the controller's key never leaves its PDS. Rehearse with ENTRYWAY_HIGHPORT_DRY_RUN=true, then run entryway reconcile and expect no drift before you switch it off, except highport-foreign: names under the base held for identities with no account at the entryway, which reconcile reports and never releases. Read that list once: each should be a name you hold for someone on purpose.

Every handle under a base is a reservation, counted against the base's maxNames at Highport, or against the deployment's BARD_CONTROL_DELEGATION_MAX_NAMES when the base named none (bard's default is 20). A rename briefly holds two. Check the bound before opening signups: getDomain shows it in the base's delegation view, and its absence means the deployment setting applies. When it runs out, signups and renames under the base fail with DelegationQuotaExceeded and EntrywayHighportDelegation fires; the comment on that rule in the alerts file is the runbook.

8. Running it #

  • The operation log. Signups, handle changes, status changes and deletions are resumable operations. One that runs out of retries is failed and stays visible: entryway ops list --failed, then entryway ops retry ID or entryway ops abandon ID. ops retry also runs a pending operation now instead of when its backoff ends, once whatever it was waiting on is fixed; ops list shows a pending one's next=. entryway ops list --did DID shows one account's, a signup's included once its genesis exists.
  • Reconciliation. serve compares the entryway, every PDS, PLC and Highport every ENTRYWAY_RECONCILE_INTERVAL (one hour), and repairs what is safe to repair. entryway reconcile shows the same report on demand. What it only reports, a person has to resolve.
  • Metrics and alerts. Build with metrics, set ENTRYWAY_METRICS_BIND to a private address, and load crates/atproto-entryway/deploy/entryway-alerts.yml into Prometheus. It alerts on a failed operation, drift that survives two reconciliations, a PDS whose calls keep failing, and a Highport delegation that has run out or lapsed.
  • Logs. RUST_LOG=info is enough. Every state change is also an audit row in the database.
  • Passkeys. With ENTRYWAY_PASSKEY_RP_ID set (§1), holders add passkeys in Settings, sign in with one (a button, and the identifier's autofill), approve applications through the consent page's passkey sign-in, confirm who they are with one before a sensitive change, and may drop the password once they hold two passkeys or one that syncs. A password reset holds passkey removal and email changes for 72 hours, and mails the holder. When a holder loses a device, entryway accounts passkeys list and remove (§3, "Running the operator commands") are the support path; the last passkey of an account with no password goes only with --send-password-reset. Leave ENTRYWAY_PASSKEY_SIGNAL_DELETIONS off: iOS 26 has been seen to delete a different account's passkey on the same RP ID when it receives those signals. Before a release that touches passkeys, run tools/passkey-browser/ci.sh, which walks the flows in headless Chrome. The crate README's Passkeys section has the rest.
  • When a session ends. A member checks the entryway's access tokens with its public key and keeps no record of them. So the entryway ends a session by refusing its refresh token, and the access token already issued keeps working at the PDS until it expires. This is the reference's own design for a PDS behind an entryway. The bound is ENTRYWAY_OAUTH_ACCESS_TTL (one hour) for an OAuth grant and ENTRYWAY_ACCESS_TTL (two hours) for a legacy session. It applies to deleteSession; to revoking an OAuth grant, which is what a client's sign-out does; to ending a grant from the portal's sessions page; and to a refresh token presented twice, whose grant ends while the copy's access token runs on. Two portal actions do better on an atproto-pds member: signing out everywhere and changing the password end the account's access tokens there at once (the end_sessions operation, through the member's admin.revokeEntrywaySessions). A reference member keeps the window for those too. The only way to narrow it everywhere is to shorten the two settings, at the cost of more refreshes at the entryway.

9. Adopting an existing PDS #

A PDS that has been running on its own already has accounts, passwords and DIDs it rotates itself. It joins the cluster in one maintenance window (plan 12.2). Tell its account holders beforehand: their sessions and OAuth grants end at step 6, and everyone signs in again at the entryway. Their passwords and app passwords keep working.

Passkeys do not come along. A passkey works only under the RP ID it was created with, and the browser refuses one that is not the page's own host or a parent of it, before any server sees the request; after the switch, sign-in happens on the entryway's host. So tell anyone who signs in to the PDS with a passkey to make sure they know their password before the window. Neither portal has a page for resetting a forgotten password: reset is the requestPasswordReset and resetPassword XRPC pair, driven by a client app, so a holder who has forgotten theirs resets it before the window opens. After the switch they sign in at the entryway with the password and re-create their passkeys there, which needs ENTRYWAY_PASSKEY_RP_ID (§1).

  1. Freeze. Close signups on the PDS. Ask holders not to change passwords, addresses or handles until it is done.
  2. Register it with the entryway, as in §4, but leave the PDS standalone for now.
  3. Open the window. Set PDS_ENTRYWAY_ADOPTION=true on the PDS and restart it. This turns on two admin routes, one of which hands out every password hash on the server.
  4. Export. atproto-pds-admin entryway export --out bundle.json writes the accounts, their password and app-password hashes, and the unused invite codes. The file is created readable by its owner only. Treat it as a secret, and delete it after step 5. It also lists any account the PDS holds no rotation key for: its holder has to add the entryway's keys to their DID themselves, with signPlcOperation once the account is at the entryway. And it lists every account that holds passkeys on the PDS, with how many: those are the holders to remind about their passwords, if the reminder has not gone out already. The bundle carries no passkeys, only that count.
  5. Import. Run entryway accounts import bundle.json --dry-run, then without --dry-run. With --admin-url the bundle is sent from where you are and never written to the entryway's disk; beside the server, copy it there first and delete it afterwards. Accounts already at the entryway are skipped, so a second run after a partial one finishes the rest. So is one whose handle or address another account there holds; settle each by hand.
  6. PLC. atproto-pds-admin entryway adopt --entryway-key <PDS_ENTRYWAY_PLC_ROTATION_KEY> --recovery-key <the cold did:key> shows each DID's new rotation keys: the recovery key and the entryway's key directly above the PDS's own. Rerun with --apply to sign and submit. PLC allows five rotation keys. A DID that would pass that is refused, with its keys listed, so you can choose one to drop first. Run it again freely: a DID that lists both keys is left alone.
  7. Switch. On the PDS, set the PDS_ENTRYWAY_* settings from entryway keys show, remove PDS_ENTRYWAY_ADOPTION and any PDS_HIGHPORT_*, and restart. If the PDS owned Highport bases, move them to the entryway (§7) and check that entryway reconcile shows no drift before --apply.
  8. Check. Every account signs in at the entryway, getSession works through the PDS, handles resolve, and entryway reconcile is clean apart from highport-foreign names you hold on purpose. Passkey holders re-create their passkeys at the entryway's portal.

Until the PDS's own keys are removed from each DID, it can still sign PLC operations. The way back is to restore its standalone settings and restart; its own sessions minted before the switch work again. Changes made at the entryway in the meantime are lost on that path, so keep the window short. Passkeys go the same way: the PDS kept its passkey rows (it refuses PDS_PASSKEY_RP_ID in entryway mode but does not delete them), so restoring that setting brings the old passkeys back, and the ones made at the entryway stay there, unusable on the PDS's host. Removing the PDS's keys later, with an update the entryway signs, frees PLC slots and closes that way back.

After the switch, signing out everywhere or changing the password at the entryway's portal still ends every session at once on this PDS, as it did standalone (§8, "When a session ends").

10. Backups #

The entryway is the only copy of the account-to-PDS map, the password hashes and the hot keys. Losing it strands every account in the cluster, though their data is safe on the PDSes, and the cold key still controls every DID.

  • The volume: entryway.sqlite and keys/. Stop the server, copy /data, start it again. A copy of the live database can capture a torn write, since SQLite's WAL is mid-flight, and the image is distroless, with no sqlite3 inside it to take an online backup.
  • The secret files (admin password, pepper, each PDS's admin password and peer secret), wherever you keep them.
  • The cold key, offline, never with the rest.

A restore is a fresh volume with the backup in place. Afterwards every account should sign in and entryway reconcile should show no drift. Rehearse it once before production (plan 10.7's drills).

11. Upgrades #

An upgrade is a redeploy of a newer image. The database migrates forward on open and there is no downgrade, so back up first. Rolling back means restoring that backup, as with the PDS.

Security checklist #

  • HTTPS on the entryway and every PDS; ENTRYWAY_PRODUCTION=true refuses plain HTTP.
  • A distinct cold recovery key, offline, in every DID.
  • Distinct admin passwords for the entryway and for each PDS, and a peer secret per PDS.
  • The metrics listener on a private address only.
  • Backups of the volume and the secrets, tested by a restore.
  • The JWT rotation procedure rehearsed before it is needed.
  • The session window understood: an ended session's access token works at a member until it expires, except after the portal's sign out everywhere or password change on an atproto-pds member (§8).
  • The debug image deployed only for as long as a job takes, and the production image back afterwards.