atmosphere-mail
Cooperative email for PDS operators
Star
Fork
Watch
atmosphere-mail docs deployment-guide.md
42 kB · 1003 lines
Markdown
at main

comail Deployment Guide #

A first-deploy guide for operators who want to stand up their own comail relay against their own outbound IP. This document is the public counterpart to operator-runbook.md: the runbook assumes you already have a running deployment and is organized around routine ops and incident response; this guide is the path from "empty VPS" to "first successful test message."

The intended audience is an operator who has not worked on the codebase before. Familiarity with Linux, SSH, DNS, and the broad shape of how SMTP delivery works is assumed. Familiarity with Nix is helpful but not required — the reference deploy path uses Nix flakes, but every step is explained literally.

If you have not yet decided whether self-hosting is the right move, read the public self-hosting overview on the docs site first. That page covers the "why" and the trade-offs. This document covers the "how."

Honest scope statement #

Before you start, three things to be straight about:

  1. What works today. A single relay node, signing with its own operator-DKIM key, delivering directly to recipient MX servers, with a self-hosted labeler issuing attestations against your own domain. That path is fully exercised — it is what the hosted comail instance runs on.

  2. What is partially built. Cross-relay federation — two relays trusting each other's attestations, sharing reputation signals, forwarding FBL complaints to the originating member's relay — is on the roadmap (internal ref) but not turnkey. If you want to peer with the hosted comail relay or another operator, expect manual coordination and bespoke configuration today.

  3. What requires comail-issued infrastructure. The hosted labeler.atmos.email issues verified-mail-operator and relay-member labels for members enrolled against the hosted relay. If you run your own relay, your members are not labeled by the hosted labeler — you run your own labeler (the binary is in this repo) and your members carry your labeler's DID in their attestation record. Clients that want to honor both labelers must subscribe to both. There is no shared "comail label vocabulary" service that third-party relays can plug into today; the lexicon (email.atmos.attestation) is open, but the issuing labeler is per operator.

If any of those three points blocks your use case, file an issue on the repo before sinking time into a deploy.

Prerequisites #

Hardware #

A small Linux VPS is enough headroom for low-thousand daily message volumes. The hosted relay runs on a Hetzner CPX21 (3 vCPU, 4 GB RAM, 80 GB disk). Anything in that class works — the relay process itself sits well under 500 MB resident; the rest goes to the SQLite state database, Prometheus metrics, and headroom for queue spikes.

Disk planning: state lives at /var/lib/atmos-relay/. The SQLite store grows roughly linearly with sent volume because of the 30-day relay_events retention window. Plan for ~50 MB per 100k messages.

Network #

  • Static IPv4. A shared or dynamic IP is workable for a private staging deployment but not for production sending. Receivers anchor reputation to the IP and they cache it; a new tenant on a previously used IP inherits whatever the previous tenant did with it.
  • Dedicated outbound IP. "Dedicated" here means "no other tenants on this IP" — both directions. If your VPS provider rotates IPs among customers, you cannot run a reputation-bearing relay on it.
  • PTR record matching forward DNS. The reverse DNS for your relay's outbound IP must resolve to a hostname that itself resolves back to the same IP. Many receivers reject helo from a host whose PTR is missing or generic (static.123.45.67.89.example-isp.net). Set the PTR via your VPS provider's control panel; verify with dig -x <relay-ip>.
  • Port 25 inbound and outbound. Inbound :25 is for bounce DSNs and FBL complaint reports; outbound :25 is for actually delivering mail. See the port-25 section below — many providers block it.
  • Port 587 inbound. Member SMTP submission lands on :587 with STARTTLS + SASL PLAIN. Open it to the public internet.
  • Port 443 inbound (optional but recommended). Serves the public web surface — enrollment wizard, account management, OAuth callbacks, List-Unsubscribe one-click endpoints. You can put this behind a reverse proxy (Caddy, Tailscale Serve) or terminate TLS in the relay.

Software #

  • NixOS or another distribution capable of running the relay binary
    • systemd. The reference deploy path is NixOS via flakes; a Docker path exists (root Dockerfile) but is less well-trodden.
  • A Nix-capable workstation. You need Nix on your dev machine to evaluate the flake and drive nixos-rebuild --target-host. Nix can be installed on macOS, Linux, or Windows (WSL).
  • SOPS + age. The flake encrypts operator secrets with sops using age recipients. The placeholder age keys in this repo are syntactically valid but have no private half — you generate your own before the first deploy.
  • ssh-to-age. Used to derive an age public key from a host's SSH host key. Available in nixpkgs.
  • dig and swaks. For verifying DNS and sending SMTP test traffic. Both are in nixpkgs and most package managers.

Domains #

You need authoritative DNS control over two distinct things:

  • An operator domain. This is the equivalent of atmos.email for the hosted instance — the domain that appears in Return-Path: on outbound mail, in the operator-DKIM d= tag, and as the anchor for FBL registrations (Microsoft JMRP, Yahoo CFL, Gmail Postmaster Tools). Pick something short and operator-branded. It does not need to be the same domain your members send from.
  • A public web hostname. Where members access the enrollment wizard and account UI. Can be on the operator domain (relay.your-op.example) or a separate marketing domain. Cert provisioning happens here.

Member sending domains are separate — each member brings their own domain at enrollment time and publishes their own DKIM records pointing at the relay-issued public halves.

Port 25 access #

Cloud providers block outbound port 25 in different ways. Verify your provider explicitly before committing to a deploy plan.

Provider Default state How to unblock
AWS EC2 Outbound 25 blocked, opaque support process to unblock Request via the AWS console quota request; expect days of latency and possible denial
Google Cloud Outbound 25 blocked, no path to unblock Use a relay provider on top (not viable for this deploy)
Azure Outbound 25 blocked on most subscriptions Enterprise agreement may have a path; standard subscriptions effectively cannot run an SMTP server
Hetzner Outbound + inbound 25 blocked by default on new accounts Open a support ticket per direction; ~1 business day per ticket. See atmosphere-mail-port25.md for the wording that worked for us
OVH Outbound 25 unblocked by default once account is verified A 24-hour grace block on brand-new accounts
DigitalOcean Outbound 25 blocked, unblockable on case-by-case basis via support Inconsistent; many operators report denials
Vultr Outbound 25 blocked by default Account-age threshold to unblock
Bare-metal colo / community ISP Usually open Confirm with the operator before assuming

To verify a deployed relay can actually send:

# From the relay host, attempt to connect to a known receiver MX.
nc -v -w 5 -z gmail-smtp-in.l.google.com 25

# Expected on success:
#   Connection to gmail-smtp-in.l.google.com (...) 25 port [tcp/smtp] succeeded!
#
# Connection timeout means outbound :25 is blocked. Do not proceed with
# deploy until this is fixed.

To verify inbound 25 works (so bounce DSNs and FBL reports can reach the relay):

# From any external host.
nc -v -w 5 -z <relay-ip> 25

# Connection refused = inbound 25 is blocked at the provider's edge
# even before the relay's firewall rules run. Open a ticket.

The hosted comail relay runs on Hetzner. The port-25 unblock took two separate tickets — one for outbound, one for inbound — and roughly one business day each. Plan accordingly.

DNS records you will own #

The relay sits at the intersection of several DNS responsibilities. This section covers the records you, the operator, publish. Member domains have their own set of records they publish themselves; that is documented on the public docs site under DNS records.

Operator DNS records #

For the operator domain (the example below uses your-operator.example):

Record Name Type Value Purpose
A relay.your-operator.example A <relay-ip> Forward DNS for the relay host
PTR (set at VPS provider) PTR relay.your-operator.example Reverse DNS — must match forward
MX your-operator.example MX 10 relay.your-operator.example Inbound bounce/FBL
MX relay.your-operator.example MX 10 relay.your-operator.example Inbound bounce/FBL (sub-domain copy)
SPF your-operator.example TXT v=spf1 ip4:<relay-ip> -all SPF for operator domain
DMARC _dmarc.your-operator.example TXT v=DMARC1; p=quarantine; rua=mailto:postmaster@your-operator.example Start with p=none, move to p=quarantine after two weeks of monitoring
Operator DKIM (RSA) atmos<YYYYMMDD>r._domainkey.your-operator.example TXT v=DKIM1; k=rsa; p=<public-key> Generated at first boot, value printed in journal
Operator DKIM (Ed25519) atmos<YYYYMMDD>e._domainkey.your-operator.example TXT v=DKIM1; k=ed25519; p=<public-key> Generated at first boot

Notes:

  • The <YYYYMMDD> selector is derived from the date the operator-DKIM file was first written. It is stable across restarts — only a key rotation changes it.
  • You publish two DKIM records (RSA + Ed25519) because the relay dual-signs every outbound message; receivers pick whichever algorithm they support.
  • The selector prefix atmos is hard-coded today; see docs/dkim-rotation.md on the docs site for the full story.

Web hostname records #

For your member-facing web surface (the example uses comail.your-op.example):

Record Name Type Value Purpose
A comail.your-op.example A <relay-ip> (or proxy) Enrollment + account UI
AAAA (optional) comail.your-op.example AAAA <relay-ipv6> If you serve IPv6
_atproto _atproto.your-op.example TXT did=did:plc:<your-operator-did> If your operator DID lives on a different domain

The comail.your-op.example host serves the atproto OAuth client metadata document at /.well-known/atproto-oauth-client-metadata.json plus the enrollment wizard, account API, and List-Unsubscribe endpoints. The relay's SiteBaseURL config field must match this hostname exactly — the atproto spec requires client_id to equal the URL the metadata is served from.

Per-member DNS #

Every member publishes a separate set of records on their own sending domain — DKIM keypair public halves, SPF including your relay's IP, DMARC, and (when their sending domain differs from their handle domain) an _atproto.<sending-domain> TXT record. The enrollment wizard generates the exact strings for them. The operator does not publish these.

SOPS + age setup #

The flake encrypts every secret with sops keyed against age recipients. The repo ships .sops.yaml with placeholder age recipients (strings starting with age1example…) so the YAML parses, but no private key exists for them. You replace them before the first deploy.

Three age keys are needed for a single-relay deployment:

  1. Admin key. Lives on your workstation. You use this to edit any sops secret locally.
  2. Relay host key. Derived automatically from the relay's SSH ed25519 host key. The relay reads its own secrets via sops-nix at activation.
  3. Optional ops host key. Only relevant if you run a separate metrics/Kafka host. Skip for a single-node deploy.
# 1. Generate the admin key on your workstation.
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt
# The printed line `# public key: age1...` is what you put in .sops.yaml.

# 2. After provisioning the relay VM (next section), derive its age
#    pubkey from the SSH host key:
ssh root@<relay-ip> 'cat /etc/ssh/ssh_host_ed25519_key.pub' \
  | nix run nixpkgs#ssh-to-age

# 3. Edit infra/.sops.yaml. Replace the placeholder `age1example…`
#    strings with the two real public keys from steps 1 and 2.

# 4. Re-encrypt the secrets file against the new recipients.
sops updatekeys infra/secrets/relay.yaml

If you skip step 4 and try to sops -d a secret with only the placeholder keys in the file's encryption metadata, decryption fails silently — the relay will fail to start on first deploy because every templated secret resolves to an empty string.

Deeper coverage of the sops setup, including the dual-key staging deployment topology, lives in the Outside operator setup section of the operator runbook.

Generating relay-side secrets #

The placeholder values in infra/secrets/relay.yaml.example document how each secret is generated. Reproduced inline here so you can run the commands without context-switching:

# Admin API bearer token. 32+ chars of base64 random.
openssl rand -base64 32

# Session HMAC secret. 32 raw bytes, base64-encoded.
head -c 32 /dev/urandom | base64

# RELAY_PII_KEY. AES-256 key (32 raw bytes, 64 hex chars).
head -c 32 /dev/urandom | xxd -p -c 64

# RELAY_OAUTH_DEK. Separate AES-256 key for OAuth token encryption.
# MUST be a different value than RELAY_PII_KEY.
head -c 32 /dev/urandom | xxd -p -c 64

# ntfy push tokens (only if you wire ntfy notifications). Per-topic.
# Generated by your ntfy server's admin tool.

Back up RELAY_PII_KEY somewhere durable the moment you generate it. Loss of this key bricks DKIM signing, contact-email decryption, and forward-to decryption. The relay-pii-key-runbook.md covers the full backup + rotation procedure and is required reading before touching the value in production.

Config file walkthrough: infra/secrets/relay.yaml #

This is the sops-encrypted secrets file. Open it with sops infra/secrets/relay.yaml; sops decrypts in place and reopens your editor on the cleartext. Save and exit to re-encrypt.

Required fields:

# Admin API bearer token. Generated above.
admin_token: "<openssl-rand-base64-32 output>"

# Labeler XRPC URL — data plane, served publicly.
# For a single-node deploy where the labeler runs on the same host:
labeler_url: "https://labeler.your-operator.example"

# Labeler admin URL — control plane, used for /admin/revoke and
# /admin/reverify. The public labeler hostname 404s /admin/* as a
# defense-in-depth backstop, so this must be an internal hostname
# the relay can reach. On a single-node deploy, loopback works:
labeler_admin_url: "http://localhost:8081"

# Session HMAC secret. Generated above.
session_secret: "<base64 32-byte output>"

# AES-256 column-level encryption key. Generated above. KEEP A BACKUP.
pii_key: "<64-hex-char output>"

# AES-256 OAuth token encryption key. MUST DIFFER from pii_key.
oauth_dek: "<64-hex-char output>"

Optional fields (skip lines you don't need):

# Restic backup destination known-hosts entry. Only relevant if you
# wire the off-site backup module in infra/nixos/restic-offsite.nix.
restic_offsite_known_hosts: "host.example ssh-ed25519 AAAA..."

# ntfy notification tokens. Only relevant if SignupNtfyTopicURL or the
# failure-alerter modules are enabled.
ntfy_token: "tk_..."
ntfy_atmos_token: "..."

Config file walkthrough: relay-config.json #

This is the cleartext config file. In the NixOS reference deploy, the content is templated by infra/nixos/default.nix and rendered to disk at /var/lib/atmos-relay/config.json with secrets interpolated from sops at activation time. You can also edit it directly for a hand-rolled deploy.

Every field maps to a struct member in cmd/relay/config.go (RelayConfig). The minimal field set for a production-style deploy:

{
  // SMTP submission listener (member outbound). Default :587.
  "smtpAddr": ":587",

  // Operator domain. Appears in Return-Path, X-Atmos headers, and as
  // the default OperatorDKIMDomain. MUST match what your DNS publishes
  // SPF + DKIM for.
  "domain": "your-operator.example",

  // Inbound SMTP listener (bounce DSN + FBL reports). Default :25.
  "inboundAddr": ":25",

  // Admin API listener. Tailscale-only is strongly recommended; bind
  // to loopback or a private interface and front via Tailscale Serve
  // or a VPN. Default :8080.
  "adminAddr": "127.0.0.1:8080",

  // CSRF allowlist for /ui/* admin dashboard POSTs. Empty list
  // fails-closed; every state-changing admin POST returns 403.
  "adminOrigins": ["https://admin.your-operator.example"],

  // Labeler data plane URL. Where the relay sends /xrpc/* label
  // queries. Public HTTPS, typically served by the labeler binary
  // behind your reverse proxy.
  "labelerURL": "https://labeler.your-operator.example",

  // Labeler control plane URL. Tailnet/loopback only; the relay calls
  // /admin/reverify and /admin/revoke here. Public labeler hosts 404
  // /admin/* as a backstop, so production deploys MUST set this.
  "labelerAdminURL": "http://localhost:8081",

  // TLS for the public + SMTP listeners. ACME-issued certs from
  // Let's Encrypt are the recommended path; the NixOS module wires
  // acme automatically. For a hand-rolled deploy point at PEM files.
  "tlsCertFile": "/var/lib/acme/relay.your-operator.example/fullchain.pem",
  "tlsKeyFile": "/var/lib/acme/relay.your-operator.example/key.pem",

  // Public HTTPS listener address. Serves enrollment, /account/*,
  // OAuth callbacks, and /u/ List-Unsubscribe.
  "publicAddr": ":443",

  // Externally-reachable URL of the public listener's INFRASTRUCTURE
  // endpoints (List-Unsubscribe links bake in this prefix). Always
  // points at the SMTP/operator hostname, not the marketing host.
  "publicBaseURL": "https://relay.your-operator.example",

  // Externally-reachable URL of the marketing + enrollment surface.
  // This is what members type into a browser. SiteBaseURL is the
  // OAuth client_id anchor; the relay serves the client metadata
  // document at SiteBaseURL + "/.well-known/atproto-oauth-client-metadata.json".
  "siteBaseURL": "https://comail.your-operator.example",

  // Admin dashboard base URL. Internal; used in operator-ping email
  // links. Typically the tailnet/private hostname for the admin
  // listener.
  "adminBaseUrl": "https://admin.your-operator.example",

  // PublicDomains lets one relay serve multiple hostnames with
  // distinct certs and routing roles. For a simple deploy with one
  // marketing host + one smtp host, populate it explicitly:
  "publicDomains": [
    {
      "role": "site",
      "host": "comail.your-operator.example",
      "redirectTo": "",
      "certFile": "/var/lib/acme/comail.your-operator.example/fullchain.pem",
      "keyFile":  "/var/lib/acme/comail.your-operator.example/key.pem"
    },
    {
      "role": "infra",
      "host": "relay.your-operator.example",
      "redirectTo": "",
      "certFile": "/var/lib/acme/relay.your-operator.example/fullchain.pem",
      "keyFile":  "/var/lib/acme/relay.your-operator.example/key.pem"
    }
  ],

  // SQLite + DKIM key file home.
  "stateDir": "/var/lib/atmos-relay",

  // Per-member rate-limit defaults. Tighter caps win when composed
  // with per-member limits stored in the DB. See warming.go.
  "hourlyLimit": 100,
  "dailyLimit": 1000,
  "globalPerMinute": 500,

  // Operator-DKIM. Defaults to {stateDir}/operator-dkim-keys.json and
  // domain=Domain. Override to point at a different domain than the
  // relay's MAIL FROM domain; the hosted instance does this so
  // operator DKIM signs with comail.at while MAIL FROM stays on
  // atmos.email.
  "operatorDKIMKeyPath": "/var/lib/atmos-relay/operator-dkim-keys.json",
  "operatorDKIMDomain": "your-operator.example",

  // Outbound IP for the SPF block shown to members at enrollment.
  // If empty, the page shows a placeholder; members will get DMARC
  // alignment failures. Always set this in production.
  "relayPublicIP": "<your-relay-ipv4>",

  // External mailbox that receives postmaster@/abuse@/fbl@ mail for
  // the operator domain. Provider FBL verification mails land here;
  // unset = the relay accepts the mail and logs it but no human ever
  // sees the contents. Wire this to a real inbox you monitor.
  "operatorForwardTo": "ops@your-operator.example",

  // Reply-To stamped on user-facing system mail (signup verification,
  // welcome, etc.). Falls back to operatorForwardTo when empty.
  "systemMailReplyTo": "ops@your-operator.example",

  // Path to the session HMAC secret file. NixOS plumbs this via
  // sops-nix into a 0600-mode file under /run/secrets.
  "sessionSecretFile": "/run/secrets/session_secret",

  // Optional: webhook for new-signup / approval / suspension events.
  // Useful for routing to Slack, Matrix, ntfy, or PagerDuty. Empty
  // disables.
  "operatorWebhookURL": "",
  "operatorWebhookSecret": "",

  // Optional: Osprey (Kafka-based reputation rules engine). Empty
  // disables; the relay's reputation work degrades to whatever the
  // labeler issues. Both fields must be set or both empty.
  "kafkaBroker": "",
  "ospreyURL": ""
}

A few fields warrant elaboration:

publicDomains and TLS roles #

Each entry's role field picks a routing profile:

  • "site" — full member-facing surface: enrollment wizard, account dashboard, OAuth callbacks, marketing pages, legal pages, List-Unsubscribe.
  • "infra" — operational endpoints only: List-Unsubscribe at /u/, /healthz, /version. Strip down everything else. Use this for the smtp.your-operator.example hostname that needs a valid cert for STARTTLS but should not expose the full web UI.
  • "redirect" — 301 to redirectTo + request path. Useful during rebrand windows.

When publicDomains is empty the listener falls back to legacy single-cert behavior using tlsCertFile/tlsKeyFile and serves the full site handler on whatever Host is requested.

Operator DKIM domain vs MAIL FROM domain #

OperatorDKIMDomain defaults to Domain. The hosted instance uses two different values intentionally — MAIL FROM lives on atmos.email (warmed, dnswl-listed) but operator DKIM signs with d=comail.at (brand identity post-rebrand). For a fresh deploy, leave them equal.

Required environment variables #

The NixOS module plumbs these from sops at activation. If you deploy by hand, set them in the systemd unit's environment file:

ADMIN_TOKEN=<from sops>
RELAY_PII_KEY=<from sops>
RELAY_OAUTH_DEK=<from sops>
OPERATOR_WEBHOOK_SECRET=<from sops>     # only if webhook URL is set
RESEND_API_KEY=<from sops>              # only if MAIL_TRANSPORT=resend
NTFY_ATMOS_TOKEN=<from sops>            # only if signupNtfyTopicURL set

The relay reads all of these via os.Getenv at startup; values present in the JSON config are overridden by the env when both are set.

Provisioning the host #

The repo ships an OpenTofu/Terraform plan in infra/main.tf that allocates a Hetzner CPX21, attaches an SSH key, and bootstraps NixOS via the standard kexec-into-installer trick. You can use it as-is by filling in infra/variables.tf, or skip it entirely if you already have a NixOS host.

Other VPS providers: any host that boots NixOS works. The shape of the flake doesn't care where the disk lives. infra/nixos/disko.nix describes the disk layout assuming a single-disk x86_64 server.

Defining your host module #

The flake's nixosConfigurations set includes atmos-relay, atmos-ops, and atmos-staging — the three hosts the hosted instance runs on. For your own deploy, you add a new attribute to the set with your hostname.

The minimal additional module:

# infra/your-relay.nix
{ config, pkgs, ... }: {
  imports = [
    ./nixos/default.nix
    ./nixos/disko.nix
  ];

  networking.hostName = "your-relay";
  networking.domain = "your-operator.example";

  # SSH key authorized to deploy + recover.
  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA... your-admin-key"
  ];

  # Your outbound IP. The disko config defaults to DHCP; override
  # if your provider requires static configuration.
  networking.interfaces.eth0.ipv4.addresses = [
    { address = "<your-relay-ipv4>"; prefixLength = 32; }
  ];

  # ACME contact for Let's Encrypt cert issuance.
  security.acme.defaults.email = "ops@your-operator.example";

  # Override the relay config to point at your domains. The default
  # module in nixos/default.nix is currently shaped for the hosted
  # instance; you'll fork the config-rendering block in your module.
  services.atmos-relay.domain = "your-operator.example";
  services.atmos-relay.publicHostnames = [
    "comail.your-operator.example"
    "relay.your-operator.example"
  ];
}

Note: The current infra/nixos/default.nix hardcodes hosted- instance values (atmos.email, smtp.atmos.email, sops template paths assuming the hosted secrets file). It is not yet a clean reusable module that takes operator overrides cleanly — porting it to a parameterized module is part of the federation work (chainlink #572). For a first deploy today, fork the file and edit in place; for a long-term setup, watch the federation roadmap.

Wire your module into the flake:

# infra/flake.nix — add inside the outputs.nixosConfigurations attrset
nixosConfigurations.your-relay = nixpkgs.lib.nixosSystem {
  inherit system;
  specialArgs = { inherit inputs; };
  modules = [
    sops-nix.nixosModules.sops
    disko.nixosModules.disko
    ./your-relay.nix
  ];
};

First deploy #

After SOPS keys, secrets, config, and host module are in place:

# From your workstation, with the flake checked out at ~/repos/comail.
cd ~/repos/comail

# Optional: dry-run the build.
nix flake check ./infra

# Deploy to the relay host. The first run is slow (full NixOS rebuild
# from scratch); subsequent runs are minutes.
nixos-rebuild switch \
  --flake ./infra#your-relay \
  --target-host root@<relay-ip> \
  --build-host root@<relay-ip>

--build-host is the same as --target-host here, meaning the build happens on the relay itself. For a faster turnaround use a build host with more cores (the hosted instance uses a Gitea Actions runner for this; see .gitea/workflows/relay-deploy.yml).

If the build fails midway, the relay continues running the previous generation. NixOS rollback is nixos-rebuild --rollback switch on the host, or nix profile history to inspect prior generations.

After a successful first deploy:

ssh root@<relay-ip> 'systemctl status atmos-relay --no-pager'
ssh root@<relay-ip> 'journalctl -u atmos-relay --since "5 min ago" --no-pager'

Look for operator_dkim.first_boot: log lines on first ever boot — these print the operator-DKIM public-key TXT records you publish to DNS. Without them in DNS, every outbound message fails alignment at receivers.

ssh root@<relay-ip> \
  "journalctl -u atmos-relay --since '5 min ago' | grep operator_dkim.first_boot"

Copy the two name=… value=… pairs from the journal into your DNS provider. They look like:

operator_dkim.first_boot: name=atmos20260519r._domainkey.your-operator.example value="v=DKIM1; k=rsa; p=MIIBIjANBgkq..."
operator_dkim.first_boot: name=atmos20260519e._domainkey.your-operator.example value="v=DKIM1; k=ed25519; p=MCowBQYDK2VwAyEA..."

Wait for DNS propagation. Verify with:

SELECTOR=atmos20260519
for v in r e; do
  echo "=== ${SELECTOR}${v}._domainkey.your-operator.example ==="
  dig +short TXT "${SELECTOR}${v}._domainkey.your-operator.example" @8.8.8.8
done

Each should return the published TXT value, possibly with embedded newlines for long records.

Subscribing to a label vocabulary #

The relay queries a labeler to learn which DIDs hold valid email.atmos.attestation records and which carry clean-sender / verified-mail-operator / relay-member labels. The labelerURL config field points at this labeler.

You have two options:

Option 1: Run your own labeler (the supported path) #

The cmd/labeler binary is in the same repo. The default NixOS host in infra/nixos/atmos-ops.nix runs it on a separate host with its own state directory, jetstream subscription, and atproto identity. For a single-node deploy you can colocate it on the same host as the relay.

# In your host module:
services.atmos-labeler = {
  enable = true;
  listenAddr = "127.0.0.1:8081";  # control plane
  publicListenAddr = ":8082";      # data plane (reverse-proxied to HTTPS)
  stateDir = "/var/lib/atmos-labeler";
  jetstreamURL = "wss://jetstream1.us-east.bsky.network/subscribe";
};

The labeler's atproto identity is a DID you create. The simplest path is to create a regular atproto account on any PDS (Bluesky's bsky.social works), then set that account's DID as the labeler's operating identity. The labeler signs label records with that DID's signing key. Plumbing the signing key into the labeler is covered in docs/operator-runbook.md under labeler operations.

Once the labeler is running, members enroll against it: they publish an email.atmos.attestation record naming your relay's domain, and your labeler queries jetstream, verifies the record, and emits the appropriate labels.

Option 2: Federate with the hosted comail labeler (the partial path) #

The hosted labeler at labeler.atmos.email is happy to issue labels for DIDs that attest to any relay — the email.atmos.attestation lexicon is open and identifies relays by DID, not by hostname.

However: the hosted labeler is not currently configured to verify attestations against third-party relays end-to-end. Today it verifies DKIM-in-DNS for the hosted relay's selectors only. Pointing your members' attestations at your relay and asking the hosted labeler to verify them is on the federation roadmap (internal ref) and is not operational today.

For now, treat option 1 as the actual path. The "federate with the hosted labeler" option is documented as a forward-looking pattern, not as something you can configure today.

DKIM key rotation: what is and isn't possible #

For full coverage see docs/dkim-rotation.md on the docs site. The short version:

  • Operator DKIM key. Generated on first boot, written to <stateDir>/operator-dkim-keys.json. Rotation today is a filesystem operation: stop the relay, delete the JSON file, restart, publish the new TXT records from the journal log. There is no live-rotation API yet — tracked as internal ref. The Routine tasks §1 block in the operator runbook is the literal step-by-step.
  • Member DKIM key. Generated at enrollment, stored encrypted at rest in SQLite under RELAY_PII_KEY. Rotation today is delete-and-re-enroll: there is no targeted "rotate this member's DKIM only" endpoint. Members can drive this from /account/domains/{domain} in the web UI, which is wired to re-enroll under the hood.

Both gaps are documented openly in the rotation page; this guide is not papering over them.

Smoke test #

Confirm the relay actually works end to end. Run these from your workstation against the deployed relay.

Inbound SMTP accepts mail #

swaks --server relay.your-operator.example \
      --port 25 \
      --ehlo your-workstation.example \
      --from probe@example.com \
      --to postmaster@your-operator.example \
      --header "Subject: inbound smoke test" \
      --body "ping"

Expected: 250 OK on . and 221 bye on QUIT. If the relay rejects with 504 5.5.2 need fully-qualified hostname, your --ehlo value is missing a dot — pass an FQDN.

Inspect the relay's inbound_messages table to confirm it landed:

ssh root@<relay-ip> "sqlite3 /var/lib/atmos-relay/relay.sqlite \
  'SELECT id, classification, from_addr, subject FROM inbound_messages ORDER BY id DESC LIMIT 5'"

Outbound SMTP signs with DKIM #

Enroll a test member (use the /enroll flow at https://comail.your-operator.example) for a domain you control, publish the DKIM TXT records, then send a test message:

swaks --server relay.your-operator.example \
      --port 587 \
      --tls \
      --auth PLAIN \
      --auth-user "<member-did>" \
      --auth-password "<member-api-key>" \
      --ehlo your-workstation.example \
      --from member@<member-domain> \
      --to <your-personal-inbox>@gmail.com \
      --header "Subject: outbound smoke test" \
      --body "ping"

Confirm in the received message's headers at Gmail:

  • Authentication-Results: shows dkim=pass for both d=<member-domain> AND d=your-operator.example
  • dmarc=pass
  • spf=pass

If any fail, the immediate suspects are: missing DNS record, stale DNS cache, or the relay binding to a different IP than the one in SPF. Use docs/dkim-rotation.md /admin/dkim-check/<domain> endpoint to triangulate.

Health + metrics #

# Health endpoint (public listener; site or infra role both serve this)
curl -s https://relay.your-operator.example/healthz

# Version endpoint
curl -s https://relay.your-operator.example/version

# Prometheus metrics (admin listener; behind ADMIN_TOKEN)
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/metrics | head -40

The metrics endpoint exposes hundreds of counters covering goroutine-crash totals, SMTP per-error-code counts, DKIM signing outcomes, Osprey emit success/failure, SQLite-busy events, labeler-RPC call counts, and the inbound-rate-limiter token-bucket state. Wire it into your Prometheus/VictoriaMetrics scraper.

Publish a domain attestation to a PDS #

End-to-end: a member completes the OAuth flow, the relay publishes an email.atmos.attestation record on their PDS, your labeler picks it up via jetstream, and your relay's /admin/members/<did> page shows the member with a green attestation row.

This is the slowest of the three smoke tests because jetstream propagation can take a minute or two. Watch the labeler logs:

ssh root@<relay-ip> "journalctl -u atmos-labeler --since '5 min ago' --no-pager"

Expected: a label.issued: did=<member-did> label=relay-member line within ~60 s of the attestation publish.

Wiring FBL registrations #

Once your relay is sending real mail, register with the three provider feedback-loop programs anchored on your operator domain:

  • Gmail Postmaster Tools. TXT-token domain verification. Dashboard shows reputation + spam-rate signals once you have ~48 h of sending volume.
  • Microsoft SNDS + JMRP. Form-based registration. Verification mail lands at postmaster@your-operator.example — make sure operatorForwardTo is set so a human sees the verification token.
  • Yahoo CFL. Form-based registration; manual review takes 1–5 days.

Step-by-step coverage of each provider's enrollment quirks lives in the FBL integrations section of the operator runbook. The provider URLs and exact field values have shifted over time; consult the runbook (and confirm via the provider's current help docs) rather than copying anything hard-coded here.

Allowlist requests #

Two early reputation wins worth doing before going live:

  • dnswl.org. Free DNS allowlist. Apply via https://dnswl.org/apply once your relay's PTR + forward DNS + SPF are clean and you have a couple weeks of send history. The hosted comail relay sits on dnswl.org with permitted-sender status.
  • Proofpoint. If you discover your IP is on Proofpoint's blocklist (a real risk for newly-allocated VPS IPs), request delisting at https://ipcheck.proofpoint.com. Manual review, 1–3 business days.

Both are tracked in atmosphere-mail-reputation.md in the hosted instance's notes; the same procedure applies to a fresh deploy.

What this guide deliberately doesn't cover #

  • Member migration between relays. If you want to import members from the hosted comail relay (or any other relay) into yours, the cooperative reputation export format (Reputation portability axis of the federation roadmap) is not yet shipped. Members can re-enroll manually; their DID-anchored reputation does not yet travel with them automatically.
  • Cross-relay complaint routing. ARF reports arrive at whichever relay's Feedback-ID is in the offending message. There is no protocol for one relay to forward complaints to a peer.
  • Shared suppression lists. Each relay maintains its own suppression list. A bounce at relay A does not suppress that recipient at relay B.
  • HA / multi-node primary. The reference deploy is a single relay node. Adding a hot standby is operator-specific (storage replication, IP failover, DNS health checks) and isn't templated in this flake. You can build it on top, but you'll do the design work.
  • Osprey worker / Kafka. Optional reputation pipeline. The hosted instance runs it on a separate ops host. For a first deploy, leave kafkaBroker and ospreyURL empty — the relay degrades gracefully without them (no Osprey events, but everything else works).
  • The atproto labeler's deep operational details. Covered in the labeler section of operator-runbook.md. The labeler is mostly self-healing via jetstream retries; the gotchas in the runbook are the ones that have actually bit us.

Where to get help #

  • Repo issues. The Tangled mirror at tangled.org/scottlanoue.com/atmosphere-mail accepts bug reports and questions. The Gitea origin is private to the hosted instance.
  • The operator runbook. docs/operator-runbook.md has the gotchas — Druid retirement, Porkbun EHLO requirements, Auth LOGIN vs PLAIN, Tangled stale branch cache. Read it once end-to-end the first time you hit something weird.
  • The PII key runbook. docs/relay-pii-key-runbook.md before touching RELAY_PII_KEY in production.
  • Federation status. internal ref tracks the federation MVP. Cross-relay attestation verification, complaint forwarding, and shared suppression are on that roadmap. Comment there if you'd like to participate as a peer relay.

Appendix: the binary set #

For reference, the four binaries the flake builds and what each does:

Binary Purpose Default port(s)
cmd/relay Main SMTP relay: submission, delivery, admin API, public web 25, 587, 443, 8080
cmd/labeler Atproto labeler: subscribes to jetstream, issues labels 8081 (admin), 8082 (public)
cmd/label-api XRPC label query frontend (optional split from labeler) configurable
cmd/web Split-out web frontend process (member-facing routes) — proxied by cmd/relay's public listener via WebProxyURL 8090

The cmd/web split is recent (internal ref / binary-split spec) — the relay binary still serves the same routes inline when WebProxyURL is empty, so for a first deploy you can ignore the split entirely. See docs/binary-split-spec.md for the split's rationale.

Appendix: file layout cheat sheet #

atmosphere-mail/
├── cmd/
│   ├── relay/            # main relay binary; config struct lives here
│   ├── labeler/          # atproto labeler
│   ├── label-api/        # XRPC label query split
│   └── web/              # split-out web frontend
├── docs/
│   ├── deployment-guide.md       # this file
│   ├── operator-runbook.md       # ops + incidents
│   ├── relay-pii-key-runbook.md  # RELAY_PII_KEY procedures
│   ├── binary-split-spec.md      # cmd/web split rationale
│   └── ...
├── infra/
│   ├── flake.nix         # entry point: nixosConfigurations live here
│   ├── nixos/
│   │   ├── default.nix   # main host module — fork for your deploy
│   │   ├── atmos-ops.nix # labeler + Kafka host module
│   │   ├── atmos-staging.nix
│   │   ├── disko.nix     # disk layout
│   │   └── ...
│   ├── secrets/
│   │   ├── relay.yaml         # sops-encrypted
│   │   ├── relay.yaml.example # field documentation
│   │   ├── ops.yaml
│   │   └── ...
│   ├── .sops.yaml        # age recipients (REPLACE PLACEHOLDERS)
│   └── main.tf           # Hetzner provisioning (optional)
├── internal/
│   ├── config/           # labeler config struct
│   ├── relay/            # core relay logic
│   ├── relaystore/       # SQLite + PII encryption
│   └── ...
├── web-frontend/         # Astro/Starlight docs site + member UI
│   └── src/content/docs/docs/   # public docs (rendered at /docs/*)
├── relay-config.json.example   # cleartext config field documentation
└── flake.nix             # top-level dev shell flake (Go toolchain)

The two flakes are intentional: flake.nix at the repo root is for the Go dev shell; infra/flake.nix builds the production binaries and defines nixosConfigurations. Deploy commands point at the infra/ flake.