tidepool
Read-only ActivityPub → atproto bridge for the threadiverse using Coves lexicons
Star
Fork
Watch
tidepool DEPLOY.md
60 kB · 1164 lines
Markdown
at main

Deploying Tidepool v2 #

The production runbook for the v2 surface: native Coves users federating outward (tasks 13–17), on top of the v1 inbound bridge that is already running.

This document describes what is in the code, not what was planned. Where a lever does not exist, it says so under Not implemented rather than describing a plausible one. A runbook that names a knob nobody built is worse than one admitting the knob is missing, because it sends an operator hunting for it during an incident. Every claim below cites the file it came from; if you change the code, change the citation.

Related: SELF_HOSTED_RELAY.md (the relay + Jetstream ingest path), README.md (the full config table and admin route table).

1. Read this before deploying HEAD #

A HEAD deploy onto a pre-v2 /opt/tidepool/.env does not start. It exits during config.Load with:

config: AP_USER_ORIGIN is required in production

AP_USER_ORIGIN is read through stringVar, which falls back to a dev default in development and returns an error in production (internal/config/config.go:797-806, read at :414). The pre-v2 env file has no such variable.

This is correct behaviour and must not be "fixed" by defaulting it away. The origin is baked into every actor_id this deployment mints; serving then derives every URL from the stored actor_id and never from config (internal/personas/serving.go:20-23). A wrong value is therefore not a restart away from repaired — it is a set of federated identities pointing at the wrong place, permanently. Two further boot-time guards exist for the same reason: it must be https in production (config.go:782-784), and its host must not be BRIDGE_HOSTNAME or a subdomain of it, which would shadow the bridged-handle namespace (config.go:786-790).

It must also be a bare origin — no trailing slash. personas.CanonicalizeOrigin refuses any path, query, fragment, or userinfo (internal/personas/origin.go:42-46), and https://coves.social/ has a path of /. That is a boot failure, not a normalization.

docker-compose.prod.yml enforces this at the edge too: it interpolates ${AP_USER_ORIGIN:?…}, so with the variable unset or empty the deploy fails at docker compose time — before any container starts — with a message pointing here. There is deliberately no compose default: a defaulted origin on a fork of this compose file would silently mint permanent identities under someone else's domain. Add the line to .env (below) as part of the deploy.

Minimum .env delta #

Nothing else is required — every other v2 variable has a real default in every environment. See .env.prod.example for the full annotated set.

# required; the rest of the v2 knobs default safely
AP_USER_ORIGIN=https://coves.social

Deploy command #

Unchanged, and still targeted — never a bare up -d:

cd /opt/tidepool && git pull
docker compose -f docker-compose.prod.yml up -d --build tidepool

The migrate one-shot and the server share one locally-built image and the server only starts after service_completed_successfully on the migration, so a failed migration holds the old container up rather than starting a new one against an unmigrated schema.

Confirm the boot actually got past config:

docker logs --tail 50 tidepool-prod | grep -E 'listening|config:'
curl -sf localhost:8091/xrpc/_health

2. The v2 flag topology #

Everything v2 is off by default and turns on in one order. Nothing here is a runtime toggle: config is read exactly once, at cmd/tidepool/main.go:109, and there is no reload signal — changing any value means editing .env and recreating the service.

CONSUMER_ENABLED=false                    ← nothing v2 runs. Today's state.
        │
        │  ON: the Jetstream consumer dials JETSTREAM_URL. The REAL enqueuer
        │      persists outbound intent; the acceptance engine admits postv2
        │      and writes community-signed acceptances; opt-out deleteRemote
        │      and confirmed account deletions purge at peers.
        │      /admin/admissions* comes into existence.
        ▼
OUTBOUND_WORKERS=0                        ← intent accumulates, nothing is POSTed.
        │
        │  >0: delivery workers drain the queue onto the wire.
        ▼
OUTBOUND_DISABLED / _HOSTS / _COMMUNITIES / _ACTORS / OUTBOUND_DRY_RUN
                                          ← per-scope parking, applied per delivery.

Three properties of that diagram are load-bearing and are the ones operators get wrong:

There is no OUTBOUND_ENABLED. Delivery starts only when OUTBOUND_WORKERS > 0 AND CONSUMER_ENABLED (cmd/tidepool/main.go:716 and :843, inside startConsumer, which is only called under the consumer flag at :537). OUTBOUND_WORKERS also defaults to 0, via intVarNonNegative rather than intVar, precisely so that 0 is a legal value and not a config error (config.go:501, :688-702).

Consumer-on / workers-0 is a designed staging step, not a broken state. With the consumer running, the real persisting enqueuer is always wired — it writes outbound_activities/outbound_deliveries inside the consumer's gate transaction, so intent past the gate is never dropped (cmd/tidepool/main.go:695-713). Raising OUTBOUND_WORKERS later drains whatever accumulated, immediately. Budget for that.

OUTBOUND_DISABLED parks, it does not fail — and a park costs writes, not retries. A blocked delivery stays pending and resumes when the switch clears; park itself never poisons and never cancels (internal/outbound/worker.go:242-244, internal/outbound/switches.go:5-12). OUTBOUND_DRY_RUN parks the same way, before any signing or POST (worker.go:245-249).

A park costs the delivery no retries. ClaimNext does attempts = attempts + 1 on every claim (internal/store/outbound_deliveries.go:143), but park and parkCausal settle through ReleaseParked, not Release (worker.go:572, :592), whose one difference is attempts = GREATEST(attempts - 1, 0) (outbound_deliveries.go:243-244) applied under the same claim fence. A full claim/park cycle therefore leaves the ledger where it found it: an hour-long global kill switch spends nothing, and once it clears the first retryable failure is a retry, not a poison. DefaultMaxDeliveryAttempts = 8 (worker.go:78) counts attempts a delivery actually made.

⚠️ It is a continuous write load, though. A parked head is re-claimed and re-parked every parkDelay = 5 * time.Second (worker.go:560), so a held switch costs one claim plus one UPDATE per parked ordering-key head per ~5s, for as long as it is engaged — indefinitely, since nothing ends the cycle on its own. That churn is the remaining argument for preferring OUTBOUND_WORKERS=0 to park everything — see Rollback — and it is a cost argument, not a safety one.

POST /admin/outbound/redrive is the repair for poisoned deliveries, not for parked ones: RedrivePoisoned sets attempts = 0 with state = 'pending' (internal/store/outbound_deliveries.go:665-666) and matches state = 'poisoned' only (:667). A parked delivery is still pending, so it is neither reached by a redrive nor in need of one — it resumes on its own when the switch clears.

Scope matching, from config.go:519-525:

Switch Matching
OUTBOUND_DISABLED_HOSTS inbox host, lowercased on load and compared case-insensitively — a kill switch must fail closed on case
OUTBOUND_DISABLED_COMMUNITIES community AP id, e.g. https://lemmy.world/c/comicstrips, exact and case-sensitive, compared against the delivery's ordering key (worker.go:237-240)
OUTBOUND_DISABLED_ACTORS actor DID, exact and case-sensitive

There is no allowlist form of any of these. See Staged rollout for what that means for a canary.

Every switch in that table — and OUTBOUND_DISABLED and OUTBOUND_DRY_RUN with them — is INERT while OUTBOUND_WORKERS=0. ConfigSwitches is only constructed inside if cfg.OutboundWorkers > 0 (cmd/tidepool/main.go:716-736), so with no worker there is nothing holding a switch and nothing consulting one. Setting OUTBOUND_DISABLED=true while workers are 0 changes nothing and confirms nothing; it is not a second belt. Conversely, that is also why OUTBOUND_WORKERS=0 costs nothing: no worker exists to claim, park, or write anything.

Confirming a switch actually engaged #

internal/outbound/metrics.go:10-13 publishes four counters, all under /admin/metrics:

Metric Bumped when
tidepool_outbound_delivered a delivery reached the peer (incl. Lemmy's duplicate-ack)
tidepool_outbound_poisoned a delivery hit a terminal failure
tidepool_outbound_cancelled the worker cancelled a claimed delivery on consent state — its only call site (worker.go:292)
tidepool_outbound_parked a kill-switch, dry-run, or causal-wait deferral

tidepool_outbound_parked is the only positive confirmation that a kill switch engaged: by_state still reads pending for a parked delivery, exactly as it does for one merely waiting its turn, so the queue view cannot tell you. Read it as a rate, not a level — a parked head is re-parked every ~5s, so the counter climbs continuously while a switch is held. That climb is the write load of the hold, not retries being spent.

Two caveats on it. It is shared with parkCausal, so a nonzero parked with no switch engaged means causal waits, not an operator action. And tidepool_outbound_cancelled rises on its own during rollout: the worker cancels a claimed delivery whenever the actor is disabled, delivery-paused, or opted out (internal/outbound/worker.go:281-296). That is the counter's only increment site, so a climbing cancelled is consent doing its job and is never evidence that somebody ran POST /admin/outbound/cancel — the admin cancel does not touch this metric, and shows up only in by_state.

3. The admin surface at incident time #

Full route table with request bodies: README. All routes are bearer-authenticated (Authorization: Bearer $ADMIN_TOKEN) behind one middleware group (internal/ingest/follow.go:148-162, internal/accept/admin.go:65-71). Production publishes 127.0.0.1:8091 on the box so admin calls skip Caddy entirely.

Four behaviours worth knowing before you need them, because each one reads like a fault and is not:

POST /admin/communities/reconcile → 501 means FOLLOW_LIST_PATH is unset. The follow reconciler is only constructed when the path is configured (cmd/tidepool/main.go:468-483), and the handler nil-checks it (internal/ingest/follow.go:575-579). Production does set FOLLOW_LIST_PATH=/repo/communities.yaml, so a 501 there means the env changed, not that reconciliation broke.

/admin/admissions and /admin/admissions/readmit → 404 means the consumer is off. Those routes are registered inside the if cfg.ConsumerEnabled block (cmd/tidepool/main.go:537-556) because a force re-admit needs the acceptance engine, which only exists there. A 404 is "the consumer is not running", never "the endpoint is broken". Every other /admin route answers regardless.

POST /admin/outbound/redrive refuses an unscoped redrive. With no activity, no community, and no explicit {"all":true} it returns 400 (internal/ingest/follow.go:210-213) — an unscoped redrive would re-attempt every poisoned delivery at once, and a malformed body must not become a silent fleet-wide replay. POST /admin/outbound/cancel likewise demands exactly one of actor or community (follow.go:236-239).

GET /admin/outbound answers even with the consumer off. The deliveries store is wired unconditionally (cmd/tidepool/main.go:454), so an empty by_state map is the truth about the queue, not a symptom of misconfiguration.

It is also the whole route. by_state counts are all it returns (internal/ingest/follow.go:172-183) — no rows, no ids, no reasons. There is no admin endpoint that exposes last_error_class or response_excerpt, so "inspect, fix, then redrive" means psql on the box; the query is in Step 3. Plan for that before the incident, not during it.

T="Authorization: Bearer $ADMIN_TOKEN"
curl -s -H "$T" localhost:8091/admin/outbound            # queue depth by state
curl -s -H "$T" localhost:8091/admin/divergence          # one sweep, synchronously
curl -s -H "$T" localhost:8091/admin/metrics             # tidepool* expvars
curl -s -H "$T" 'localhost:8091/admin/admissions?status=rejected'

Purging a native actor (spammer) #

POST /admin/actors/purge sends Delete{Person, removeData:true} for a Coves user's persona to every inbox that received its content, undoes its standing votes, and tombstones it: the actor document answers 410 forever and the actor can never be re-enabled. Irreversible. Peers that honour the Delete drop the content and cannot restore it. A disabled or paused actor still gets the Delete delivered. Only person actors are purgeable (a group answers 422); the route answers 501 when the consumer is off. Removing the content on the Coves side is separate (social.coves.moderation.removeContent).

Run scripts/purge-actor.sh on the box: it is in /opt/tidepool/scripts once the deploy that added it has been pulled (otherwise scp it to the host and ssh in). It reads ADMIN_TOKEN from /opt/tidepool/.env without echoing it. Dry run first, check inboxes and standingVotes, then execute:

/opt/tidepool/scripts/purge-actor.sh did:plc:...            # dry run: changes nothing
/opt/tidepool/scripts/purge-actor.sh did:plc:... --execute  # the purge

A real run pauses and disables the actor first, so nothing admitted while the purge prepares is delivered after its targets are read; retractions still go out. The inboxes and standingVotes in a real run's answer are that pre-purge snapshot. The log line WARN purge committed with no federation preference to mark is expected on an operator purge (the user never wrote a deleteRemote preference).

A failed or interrupted run is safe to repeat: the purge is idempotent. Run one purge per actor at a time: two simultaneous runs can each send an Undo for the same standing vote. A sequential rerun is safe.

4. coves.social Caddy — a CROSS-REPO change #

There is no Caddyfile in this repository. TLS for both tdpl.io and coves.social terminates in the Coves Caddy (coves-prod-caddy), which reaches this stack over the shared external coves-prod-network. The work below is an edit to the Coves repo's Caddyfile — a Coves-repo change, deployed on the Coves side.

Why Caddy has to do this at all #

AP_USER_ORIGIN=https://coves.social puts the native users' ActivityPub surface on the Coves hostname, served by the Tidepool process. Tidepool's Host router sends requests whose Host is coves.social to the persona surface, and everything under tdpl.io to the bridge (internal/personas/hostrouter.go:89-115). None of those AP paths reach Tidepool today — but not because Caddy sends the whole hostname to the AppView. The existing coves.social site block (the Coves repo's Caddyfile, block opens at :70; routing runs :72-117) already splits the hostname four ways:

Path Today's handler
/.well-known/* static file_server over /srv (Caddyfile:72)
/client-metadata.json static file_server over /srv (Caddyfile:79)
/img/* 301 to img.coves.social (Caddyfile:97)
everything else catch-all reverse_proxy appview:8080 (Caddyfile:102)

Only the catch-all reaches the AppView. That matters for the before/after diff: GET /.well-known/webfinger today returns a static-file-server 404 from /srv, not an AppView response, and GET /nodeinfo/2.0 and /ap/* fall to the AppView. So the signature to look for before the change is a bare 404 on webfinger — and after it, a Tidepool JRD.

The apex needs a content-negotiated split, and Tidepool deliberately cannot do it itself: internal/personas/instance.go:32-34 states in its own comment that content negotiation is absent because Caddy owns it in production, and that a peer sending ld+json, activity+json, or no Accept at all must still get the actor. Peers fetch GET / to find the instance actor; browsers fetch GET / to find the web app. Only the edge can tell them apart.

⚠️ The Caddyfile inode trap — read this first #

Edit the Caddyfile in place. Never replace the file. The production Caddy mounts the Caddyfile as a single-file bind mount, and Docker pins a single-file mount to the inode present when the container started. Anything that replaces the file rather than writing through it — git checkout, mv, sed -i, most editors' atomic-save, scp of a new file — leaves the container serving the old contents forever, while cat on the host shows the new ones. This has bitten this project before. It is the same trap that made Tidepool's own communities.yaml a directory mount (docker-compose.prod.yml:18-24).

Safe edits: nano/vim with backupcopy=yes, cat > file, tee. After any edit, confirm the container sees it:

docker exec coves-prod-caddy cat /etc/caddy/Caddyfile | grep -c '/ap/'
docker exec coves-prod-caddy caddy validate --config /etc/caddy/Caddyfile \
  --adapter caddyfile
docker exec coves-prod-caddy caddy reload --config /etc/caddy/Caddyfile \
  --adapter caddyfile

If the container's copy does not show your change, the inode moved: recreate the Caddy container (docker compose -f docker-compose.prod.yml up -d --force-recreate caddy in /opt/coves).

The change #

Inside the existing coves.social { … } site block — do not add a second block for the same hostname — add the AP routes. handle blocks in one site are mutually exclusive and sorted by path specificity, so /.well-known/webfinger wins over the existing /.well-known/* static block, and /ap/* is disjoint from everything already there.

coves.social {
    # ── Tidepool's native-user AP surface (AP_USER_ORIGIN) ──────────────
    # These paths belong to the bridge, not the AppView. More specific than
    # the /.well-known/* static block below, so they win the handle sort.
    #
    # NOTE: no `header_up Host` on these proxies. Caddy v2 forwards the
    # original Host by default, and Tidepool's Host router keys on it to
    # choose the persona surface over the bridge surface
    # (internal/personas/hostrouter.go). Rewriting Host to the upstream
    # address would 421 every one of these requests.
    handle /.well-known/webfinger {
        reverse_proxy tidepool:80 {
            header_up X-Real-IP {remote_host}
        }
    }
    handle /.well-known/nodeinfo {
        reverse_proxy tidepool:80 {
            header_up X-Real-IP {remote_host}
        }
    }
    handle /nodeinfo/2.0 {
        reverse_proxy tidepool:80 {
            header_up X-Real-IP {remote_host}
        }
    }
    # /ap/actor/{did}, /ap/actor/{did}/outbox, /ap/object/*, /ap/activity/*,
    # and POST /ap/inbox — the shared inbox for this origin.
    handle /ap/* {
        reverse_proxy tidepool:80 {
            header_up X-Real-IP {remote_host}
        }
    }

    # ── Apex: split by Accept ───────────────────────────────────────────
    # `handle /` matches the apex EXACTLY (a Caddy path matcher is exact
    # unless it ends in *), so this replaces only the bare "/" case that
    # the catch-all used to serve. Same-name directives run in Caddyfile
    # order, so the matched reverse_proxy is tried before the fallback.
    #
    # The split is inverted on purpose: EXPLICIT text/html goes to the web
    # app; everything else — activity+json, ld+json, AND a request with no
    # Accept header at all — falls through to Tidepool. instance.go:32-34
    # requires that a peer sending no Accept still gets the instance
    # actor, and a positive @ap matcher can never satisfy that (an absent
    # header matches nothing). Browsers always send text/html at the
    # apex, so they are unaffected; the one visible consequence is that a
    # bare `curl /` (Accept: */*) now gets the actor JSON, which is the
    # AS2-conventional answer for a non-browser client.
    handle / {
        @html header Accept *text/html*
        # The web app, with the SAME upstream options as the catch-all
        # below — copy them verbatim, header_up Host included (DPoP htu
        # matching depends on it).
        reverse_proxy @html appview:8080 {
            health_uri /xrpc/_health
            health_interval 30s
            health_timeout 5s
            header_up Host {host}
            header_up X-Real-IP {remote_host}
            header_up X-Forwarded-For {remote_host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-Host {host}
        }
        # Fallback: the instance actor (peers, absent-Accept fetchers).
        reverse_proxy tidepool:80 {
            header_up X-Real-IP {remote_host}
        }
    }

    # … existing handle /.well-known/*, /client-metadata.json, /img/*,
    #   catch-all handle, headers, CSP, encode — all unchanged …
}

Verify from outside #

# instance actor (Application), not the web app
curl -s -H 'Accept: application/activity+json' https://coves.social/ | jq '.type,.id'
# expect: "Application"  "https://coves.social/"

# the AS2 spelling must reach the same document
curl -s -H 'Accept: application/ld+json; profile="https://www.w3.org/ns/activitystreams"' \
  https://coves.social/ | jq '.type'

# a browser must still get the web app
curl -sI -H 'Accept: text/html' https://coves.social/ | head -1

# a peer sending NO Accept header must still get the instance actor
# (instance.go's stated requirement; this is why the split routes
# explicit-HTML to the AppView and falls through to Tidepool)
curl -s -H 'Accept:' https://coves.social/ | jq '.type'
# expect: "Application"

curl -s 'https://coves.social/.well-known/webfinger?resource=acct:alice@coves.social' | jq .
curl -s https://coves.social/.well-known/nodeinfo | jq .
curl -s https://coves.social/nodeinfo/2.0 | jq '.software.name'   # "coves"

A 421 from any of these means the Host header was rewritten on the way through. A 404 on webfinger for a user who has never federated is correct — actors are minted lazily on first federating interaction.

Not changing #

tdpl.io, the per-instance wildcard blocks, and the on-demand catch-all are untouched by v2. The on_demand_tls ask gate still points at http://tidepool:80/.well-known/tidepool-tls-ask — but note that URL names the CONTAINER, so the request arrives carrying Host: tidepool, a name the v2 host router recognizes as neither surface. The gate only keeps working because the router serves identity.TLSAskPath host-agnostically, before any Host judgment (HostAgnosticPaths, internal/personas/hostrouter.go). An earlier revision of this section claimed the gate was "unaffected" by the host router; it was not — the 421s it produced denied issuance for every new handle cert and renewal until the exemption landed (2026-08-22).

5. Staged rollout #

Decision 19's canary. Each step is a separate .env edit plus docker compose -f docker-compose.prod.yml up -d tidepool, because config is read once at boot.

Step 0 — Caddy first #

Do section 4 before enabling the consumer. A native actor that federates outward triggers Lemmy to fetch its webfinger, its actor document, and the origin apex, all at coves.social. If Caddy is not routing those to Tidepool yet, Lemmy gets the web app or a 404 and caches the failure.

Step 1 — consumer on, delivery off #

CONSUMER_ENABLED=true
OUTBOUND_WORKERS=0

Nothing reaches any peer. Watch for a day:

curl -s -H "$T" localhost:8091/admin/metrics | jq '{
  cursor_age: .tidepool_consumer_cursor_age_seconds,
  last_event_age: .tidepool_consumer_last_event_age_seconds,
  connected: .tidepool_consumer_connected,
  dead_letters: .tidepool_consumer_dead_letters,
  reconnects: .tidepool_consumer_reconnects
}'
curl -s -H "$T" localhost:8091/admin/outbound          # intent accumulating
curl -s -H "$T" 'localhost:8091/admin/admissions?status=rejected' | jq

tidepool_consumer_cursor_age_seconds and tidepool_consumer_last_event_age_seconds are what make a stalled consumer visible: the process stays up and the healthcheck stays green while events quietly stop arriving (cmd/tidepool/main.go:827-830). A climbing tidepool_consumer_dead_letters means events are failing and being parked for the redriver, not lost.

Two reading rules for those keys:

  • The whole tidepool_consumer_* family only exists while the consumer runs — PublishMetrics is called inside startConsumer (cmd/tidepool/main.go:830). Absent keys mean the consumer is off, not that it is broken and silent. The tidepool_divergence_* gauges are the opposite: registered at package init, so they are always present.
  • tidepool_consumer_dead_letters: -1 is not a count. It is the sentinel for "storage could not be read" (internal/consume/metrics.go:26-30), chosen because a 0 would claim the backlog is empty at exactly the moment nobody can tell.
  • The tidepool_divergence_* gauges use the same -1 convention, for the same reason (internal/ingest/divergence.go:166-175, divergenceUnswept). Because they are published at package init, they are present from process start — reading -1 until the startup sweep completes, and again for any class a failed sweep never wrote. -1 is "never measured", 0 is "measured, nothing diverging"; do not alert on them as if both meant healthy.
  • The four tidepool_outbound_* counters are absent until the first delivery worker touches them — they are expvar.NewInt at package init in internal/outbound/metrics.go:10-13, so the keys exist once the package is linked, but they sit at 0 while OUTBOUND_WORKERS=0. A flat 0 across all four at this step is exactly right.

Check the rejections before letting anything out. A misconfigured ADMISSION_MAX_PER_AUTHOR_PER_COMMUNITY, a stale community mapping, or a consent state you did not expect all surface here as decisionCodes while the blast radius is still zero.

Step 2 — one community #

There is no allowlist, so a one-community canary is spelled as a denylist over the rest of communities.yaml. With today's four entries, canarying !comicstrips@lemmy.world means:

OUTBOUND_WORKERS=1
OUTBOUND_DISABLED_COMMUNITIES=https://lemmy.world/c/selfhosted,https://lemmy.world/c/fediverse,https://lemmy.ml/c/linux

AP ids, exact case — not the !name@host spelling communities.yaml uses. Confirm the ids you are about to paste rather than constructing them; the community field of the list response is the AP group id (internal/ingest/follow.go:283-300):

curl -s -H "$T" localhost:8091/admin/communities | jq -r '.communities[].community'

Adding a community to communities.yaml later does not add it to this denylist. The reconciler will subscribe it and it will start federating on the next sweep. Whenever the follow list grows during a canary, extend OUTBOUND_DISABLED_COMMUNITIES in the same change.

OUTBOUND_DRY_RUN=true is a weaker instrument than it sounds, and it is not free. What it does: the worker claims a delivery, logs it, and parks before signing or POSTing (internal/outbound/worker.go:245-249). What it does not do is validate the translator. Translation happens at ENQUEUE time, inside the consumer's gate transaction (cmd/tidepool/main.go:703-710), and the worker POSTs the stored payload verbatim (worker.go:309, :319). By the time a delivery is claimable its payload has already been translated and persisted — so the moment CONSUMER_ENABLED=true, the translator has already run on everything, dry run or not. Step 1 is where you check its output (read outbound_activities.payload in psql), not here.

And because dry-run parks, it carries the same cost as any other park — which is write load, not retries: each head delivery is re-claimed and re-parked every ~5s for as long as it is engaged, and the attempt it spends doing so is handed back (see §2). Nothing needs redriving afterwards.

On announcement throttling — what actually exists #

Decision 19 asks for "deliberate throttling of initial actor announcements". Read this carefully, because the obvious knob is the wrong one.

MINT_RATE_PER_MINUTE / MINT_BURST (production overrides them to 10/20, docker-compose.prod.yml, because the public plc.directory 429s mint bursts during community backfill) gate inbound DID minting only. They are wired into ingest.NewMintGate, whose sole consumer is the materializer's minter (cmd/tidepool/main.go:271-276, :315) — the path where an unseen Lemmy author gets an atproto DID.

There is no rate limiter on the outbound side. internal/outbound contains no rate.Limiter, no sleep, and no throttle of any kind; the persona actors the consumer mints do not pass through mintGate (main.go:539 hands personasService directly to startConsumer). The only levers on outbound volume are:

  • OUTBOUND_WORKERS — worker concurrency, not a rate. Workers poll with a 1s idle interval (main.go:53); with a full queue they run flat out.
  • the scoped kill switches — which communities/actors/hosts may deliver at all.

So the canary is the throttle: OUTBOUND_WORKERS=1 plus a one-community scope. Do not go looking for an announcement rate knob; nobody built one, and this is filed under Not implemented.

Step 3 — widen #

Remove entries from OUTBOUND_DISABLED_COMMUNITIES one at a time, raising OUTBOUND_WORKERS as the queue justifies. Before each widening:

# poison depth — the queue's own verdict
curl -s -H "$T" localhost:8091/admin/outbound | jq .by_state

# echo suppression: bridge-origin content must never re-materialize
curl -s -H "$T" localhost:8091/admin/metrics | jq '{
  mapped_object: .tidepool_echo_drops_mapped_object,
  local_activity: .tidepool_echo_drops_local_activity,
  local_actor: .tidepool_echo_drops_local_actor,
  ancestor: .tidepool_echo_drops_ancestor_short_circuit
}'

# the reconciliation report
curl -s -H "$T" localhost:8091/admin/divergence | jq '{counts, truncated}'

What each one means:

  • by_state.poisoned climbing — deliveries exhausting their retries. Inspect, fix, then redrive scoped to the affected community.

    There is no admin inspect surface for the "why". GET /admin/outbound returns by_state and nothing else (internal/ingest/follow.go:172-183) — a count, no rows, no reasons. The columns that carry the diagnosis, last_error_class and response_excerpt, are written by the worker but exposed on no route; reading them means psql on the box:

    docker exec -i tidepool-prod-postgres psql -U tidepool -d tidepool -c "
  SELECT ordering_key, last_error_class, last_status_code, attempts,
         left(response_excerpt, 200) AS excerpt, updated_at
    FROM outbound_deliveries
   WHERE state = 'poisoned'
   ORDER BY updated_at DESC LIMIT 50;"

    attempts in that output counts claims that were TRIED — a park hands its own claim's increment back (see §2) — so a poisoned row should read about 8. A materially higher number is not extra peer rejections: it is claims that never settled, since a lapsed lease or a crashed worker leaves its increment behind with nobody to return it.

    last_error_class = 'kek_misconfigured' with attempts = 1 is a configuration incident, not a peer one. The actor's sealed signing key did not open under the configured BRIDGE_KEK, so the worker poisons on the first attempt instead of spending eight (internal/outbound/worker.go): retrying cannot change the answer, and the fast poison is what puts the cause in front of an operator in minutes rather than hours. Expect it in bulk and expect it to spare newly minted actors — anything sealed since the wrong key was configured opens fine, which is why half the bridge keeps working. Fix BRIDGE_KEK (§6 runbook), restart, then redrive.

  • echo drop counters rising steadily — expected and healthy: our own content arriving back from Lemmy and being correctly refused. A counter at zero while native content is flowing is the alarming case; it means classification is not firing and re-materialization is possible.

  • tidepool_divergence_* gauges — the read-only reconciliation sweep (GET /admin/divergence runs one synchronously). Watch acceptance_undelivered_stale (a delivery pending past DIVERGENCE_ACCEPTANCE_STALE_AFTER, default 12h), acceptance_undelivered_poisoned, and vote_recast_undelivered. Also watch tidepool_divergence_sweep_failures and tidepool_divergence_sweep_age_seconds: a failed sweep publishes nothing and the gauges keep their previous values (internal/ingest/divergence.go:584-591), so a stale age with quiet gauges is a silent failure mode. The sweep never repairs anything — it reports.

Rollback #

Rollback is a kill switch, not an un-deploy. In escalation order, each step being one .env edit plus up -d tidepool:

  1. OUTBOUND_DISABLED_COMMUNITIES=<the bad one> — park one community. Everything else keeps flowing; the parked deliveries resume with their retry budget intact when you clear it, and want no redrive. First because it is the narrowest. Its only cost is the same ~5s re-claim/re-park write cycle as (3), on that community's head delivery alone.
  2. OUTBOUND_WORKERS=0 — stop the workers entirely. This is the big red button for delivery. The consumer keeps running and keeps recording intent; nothing reaches any peer. NewWorker is only called inside if cfg.OutboundWorkers > 0 (cmd/tidepool/main.go:716), so at 0 there is no worker to claim anything: nothing is re-claimed, no attempts are spent, and no rows are written. It costs nothing and it is genuinely lossless.
  3. OUTBOUND_DISABLED=true — park everything outbound. Same observable effect as (2) — nothing reaches any peer — and the deliveries keep their retries: a park hands its claim's increment back (see §2), so this is safe to hold for as long as you need. It is ranked below (2) on cost, not on risk: the workers keep running, so every ordering key's head delivery is re-claimed and re-parked every ~5s — a claim and an UPDATE per key per 5s, indefinitely, against a database you may be trying to leave alone during an incident. Reach for it when you need the scope it gives you and a whole-worker stop is too blunt. What (3) buys over (2) is that each parked row carries a recorded last_error_class / response_excerpt saying why it is held; a stopped worker records nothing.
  4. CONSUMER_ENABLED=false — stop consuming. Intent stops being recorded. The consumer resumes from its stored cursor when re-enabled, so this is recoverable, but it is the only step that stops observing, and /admin/admissions disappears with it — taking your triage view down at the moment you most want it. Prefer 1–3.

Do not roll back by cancelling deliveries. POST /admin/outbound/cancel is for consent withdrawal and community removal; a cancelled delivery is not resumable the way a parked one is.

Only redeploy the previous image if the failure is a code defect rather than a federation outcome. The kill switches address the latter faster and without a schema-version question.

6. Not implemented #

Everything in this section is a real operational gap, with two exceptions kept here on purpose: BRIDGE_KEK rotation and backup and restore were both headline gaps and are both now built, so their entries have become runbooks rather than being deleted — an operator who comes here looking for either gap finds the procedure instead of a stale warning. Both are marked BUILT in their headings. Everything else below has no mechanism in the code today. It is written down because an operator who assumes one of these exists will look for it during the exact incident where looking costs the most.

Key rotation — BRIDGE_KEK (BUILT) and per-actor RSA keys (not) #

BRIDGE_KEK rotation exists: a dual-read custodian plus a rotate-kek re-seal walk, runbook below. Per-actor RSA key rotation still does not exist, and neither does a rotation for the plaintext service-actor key; both are at the end of this entry.

What the KEK seals — three places, not two, and the walk covers all three:

  1. bridged_actors.signing_key — the escrowed secp256k1 atproto repo signing key of every bridged (Lemmy-origin) identity (internal/db/migrations/002_create_bridged_actors.sql:14; sealed and opened by Custodian.EncryptActorKey/DecryptActorKey in internal/identity/keys.go). Approximately 950 of them.
  2. service_keys.key_material, row plc-rotation — the PLC escrow rotation key (RotationKeyName / LoadOrCreateRotationKey in internal/identity/keys.go), the one thing that could recover the DIDs, and itself sealed under the KEK — which is why the boot canary in step 2 of the runbook is a real test of the rotation. Note the column is key_material, not private_key_pem; migration 013 renamed it precisely because only this row is ciphertext — the sibling service-actor row is plaintext PKCS#8 PEM and is not KEK-sealed (internal/db/migrations/013_rename_service_key_column.sql).
  3. ap_actors.rsa_key_sealed — every NATIVE Coves user's ActivityPub RSA signing key, sealed under the same KEK under its own AAD prefix (internal/db/migrations/017_ap_actors.sql:46; actorRSAKeyAADPrefix in internal/identity/keys.go, used by EncryptActorRSAKey in internal/identity/actor_rsa.go). Written at mint time by the persona create path, through the same identity.Custodian handed to personas.New in run() (cmd/tidepool/main.go).

(3) is the v2 one, and it is the one a rotation plan will forget, because it did not exist when this section was first written. A rotation that handled only bridged_actors and service_keys would leave every native user unable to sign a single outbound activity — the exact population v2 exists to serve. The walk covers it; anything hand-rolled must too.

Naming trap: LoadOrCreateRotationKey is not KEK rotation. It loads or generates the did:plc escrow/recovery key — an atproto identity concept — which is itself sealed under the KEK. Do not read that symbol as evidence that KEK rotation lives there; it lives in internal/identity/reseal.go.

Why there is no key-version column #

This section used to say a rotation would need "a key-version column or KEK-id alongside each sealed blob", and a migration for the two tables that lack one. That claim is retired. The shipped design needs no schema change and no stored selector, and anyone designing an extension should know why before re-proposing one.

GCM authentication is a definitive discriminator. Opening a blob under the wrong key does not return plausible garbage — it fails the tag check, and the odds of a wrong key authenticating are about 2⁻¹²⁸. So the KEK a blob is sealed under can simply be tried, and the answer is as trustworthy as any column would have been — while a column can be wrong (written by a crashed half-rotation, restored from a mismatched backup) in a way the ciphertext cannot. NewCustodianWithPrevious (internal/identity/keys.go) is that trial on the read path; identity.Reseal (internal/identity/reseal.go) is the same trial used to classify: opens under current → already moved, leave the bytes alone; opens under previous → re-seal and write back; opens under neither → report it and never write.

ap_actors.rsa_key_version was never the missing selector, despite looking like one. It versions the actor's RSA key — a different, still-unbuilt rotation (below) — not the KEK the key is wrapped in. Reading it as a KEK selector is the trap this paragraph exists to close. Likewise the one-byte prefix on every ciphertext (internal/identity/keys.go) is the envelope format version, checked for equality and rejected otherwise; it is not a key id either.

Runbook: rotating BRIDGE_KEK #

The old key stays load-bearing for the whole of this procedure. Do not delete it from anywhere until step 6.

  1. Put both keys in /opt/tidepool/.env — the new one current, the old one previous. Generate the new one with openssl rand -hex 32.

    BRIDGE_KEK=<new key>
BRIDGE_KEK_PREVIOUS=<the key that is in there right now>

    Both are forwarded to the tidepool service in docker-compose.prod.yml; BRIDGE_KEK_PREVIOUS is empty in steady state and the bridge treats empty as unset.

  2. Restart the bridge.

    docker compose -f docker-compose.prod.yml up -d tidepool

    It must come up clean. The boot canary opens the plc-rotation key under the KEK before serving traffic, and at this point it only opens under the previous one — so a clean boot is proof the dual read is working, and a failed boot means one of the two values is wrong. Fix it here, where nothing has been rewritten yet.

  3. Run the walk.

    docker compose -f docker-compose.prod.yml run --rm tidepool rotate-kek

    Not the tidepool-migrate one-shot: that service is deliberately wired with DATABASE_URL alone, so it has no KEKs at all and rotate-kek there would fail naming a variable that container is never meant to carry.

  4. Read the inventory. One line per table, plus one line per row that could not be moved:

    level=INFO msg="kek re-seal table" table=bridged_actors resealed=948 already_current=0 skipped=3 failed=0
level=INFO msg="kek re-seal table" table=ap_actors resealed=112 already_current=0 skipped=0 failed=0
level=INFO msg="kek re-seal table" table=service_keys resealed=1 already_current=0 skipped=0 failed=0

    A row that could not be moved gets its own line, at ERROR, naming the table, the row, and the class:

    level=ERROR msg="kek re-seal failure" table=ap_actors id=did:plc:7iza… reason=malformed

    skipped is most often a bridged actor with no escrowed key (signing_key IS NULL) — normal, not a problem. It is not only that, and not only bridged_actors: on any table a row that vanishes between the walk's read and its write-back is skipped too, because the guarded update matches nothing. That is also normal (a deleted actor), but it means a nonzero skipped on ap_actors or service_keys is a row that disappeared mid-walk rather than a NULL key. The plaintext service-actor row is not counted anywhere, by design: the walk selects plc-rotation by name and never reads its plaintext sibling.

  5. Re-run until the zero-run gate opens, i.e. every table reports resealed=0 failed=0 and the command exits 0. That re-run is the only evidence that nothing is still sealed under the old key — the first run's own counts are not, because a row written between its read and its finish would not be in them.

  6. Only then remove BRIDGE_KEK_PREVIOUS from /opt/tidepool/.env and restart:

    docker compose -f docker-compose.prod.yml up -d tidepool

    The boot canary now runs on the new key alone. If it fails, put the old key back immediately and go to step 3 — the material is still intact at that point, because the walk never overwrites what it cannot open.

Warnings.

  • Never start a second rotation before the first has shown its zero run. This tool moves blobs from one previous key to one current key. Introduce a third key while blobs are still under the first and those blobs open under neither: rotate-kek reports them wrong-key and exits non-zero (the walk completes its inventory first), and nothing in this repository can recover them without the missing key itself.

  • failed=... reason=wrong-key is a key-history question first. The blob is well formed and simply not sealed under either key you supplied. The class means the blob failed authentication under both keys, which is almost always a key-history answer — but not only: rare ciphertext corruption that preserves the envelope (version byte and length intact, bytes altered), or a blob copied into a row under a different AAD, lands in this class too, because GCM cannot distinguish "wrong key" from "right key, wrong bytes". Check key history first (an older .env, a restored volume, another deployment), integrity second. Do not rotate again, which only adds a key to the search.

  • reason=malformed is data corruption, not a key problem. The blob is truncated or carries an unknown version byte; it was never decrypted under either key, so no KEK is implicated. Restore that single row from backup. Rotating again cannot help and buries the evidence.

  • reason=missing on service_keys/plc-rotation is the restore-went-wrong hazard, and it outranks everything else here. The row is not unreadable — it is gone, on a database that still holds bridged identities, so the key every one of those DIDs names as its rotation authority is absent. Almost always a restore that missed service_keys. Stop. Do not restart the bridge: LoadOrCreateRotationKey is create-on-absence and the next boot mints a replacement, after which those DIDs are permanently beyond recovery. Restore service_keys from a backup that has the row (the restore drill's plc-rotation gate exists to catch exactly this before it is deployed) and only then continue.

  • reason=contended means only that another writer kept touching the row. Nothing is wrong. Re-run rotate-kek.

  • A failed walk exits non-zero and finishes the inventory anyway. Both are deliberate: the full inventory means one pass tells the operator how much is affected rather than stopping at the first bad row, and the exit code is the gate on retiring the old key. Read it in one direction only:

    • Non-zero forbids unsetting BRIDGE_KEK_PREVIOUS. Something is still unmoved or unreadable; retiring the old key now is what makes it unrecoverable.
    • Zero permits it only on the ZERO run — the run where every table reported resealed=0 failed=0. A first run that exits zero with resealed=948 is progress, not clearance: it says the walk moved everything it saw, not that nothing was written under the old key while it was looking. That is the whole reason step 5 asks for a second run.
    • An aborted walk still logs an inventory, and that inventory is PARTIAL. Counts from a run that ended early describe the rows it reached, not the table; never read a resealed=0 out of one as a zero run. The error line is the discriminator, not the counts.

    So a script may key on the exit code to refuse, but nothing may key on the exit code alone to permit — permission needs the zero-run counts as well.

Still not implemented: per-actor RSA rotation #

Rotating an individual actor's RSA key is undefined. It means republishing publicKey in that actor's document and having every peer that cached it re-fetch, with no grace-overlap mechanism in the code to publish two keys at once. ap_actors.rsa_key_version exists for exactly this rotation (internal/db/migrations/017_ap_actors.sql) and nothing reads it yet.

The bridge's own service-actor RSA key is also still stored as plaintext PKCS#8 PEM (service_keys, row service-actor), so it is outside the KEK entirely — rotating it is a separate, unbuilt procedure, and the KEK walk correctly refuses to touch it.

Back up BRIDGE_KEK somewhere that survives the loss of the server. Rotation is no longer a reason to treat it as immutable, but losing it — with no previous key to fall back to — is still unrecoverable.

Backup and restore #

BUILT: nightly pg_dump + a throwaway-container restore drill. Two scripts, both exercised end-to-end before landing (backup → archive verification → restore → inventory gates):

  • scripts/pg-backup.sh — run by a host systemd timer. Dumps custom-format into the ./backups mount the compose file already provides, verifies the archive with a full pg_restore -f /dev/null read before it gets its final name (a dump that cannot be read end to end is not a backup; --list reads only the TOC and would pass a truncated dump), chmod 600s it, then ages out completed dumps older than RETENTION_DAYS (default 14). A failed run keeps only the most recent failed partial, under the fixed name backups/tidepool-last-failed.dump.partial, so retries overwrite it rather than pile up ~2GB files; partials are not aged out, and ones older than a day are warned about, because a partial is the corpse of a failed run and worth seeing. The production host has no cron daemon, so it runs from the systemd units in scripts/systemd/ (02:17 UTC, as root, output in the journal). A failed run retries every 5 minutes, up to 4 starts in 2 hours, which covers a boot-time catch-up run that fires before the containers are up. The unit runs a root-owned copy at /usr/local/sbin/tidepool-pg-backup, not the checkout's script: a checkout writable by a non-root user should not supply code that runs as root. Install, from /opt/tidepool, and re-run the install line after any change to scripts/pg-backup.sh:

    sudo install -m 755 scripts/pg-backup.sh /usr/local/sbin/tidepool-pg-backup
sudo cp scripts/systemd/tidepool-pg-backup.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now tidepool-pg-backup.timer
sudo systemctl start tidepool-pg-backup.service    # first backup now
journalctl -u tidepool-pg-backup.service            # its output

    Provisioning: chmod 700 /opt/tidepool/backups once, by hand. The script can only fix the files it creates. Every dump in that directory contains the plaintext service-actor PEM and every sealed blob in the database; the directory's own mode is the operator's to set.

  • scripts/pg-restore-drill.sh — restores the newest dump (or $1) into a throwaway postgres:16 container (no published ports, --network none, reached only by docker exec), prints the row inventory, exits nonzero on any gap, and removes the container. It never touches the production container, volume, or network. What it gates on:

    • The migration version, against the internal/db/migrations/*.sql in the checkout the script sits in — so a dump that restores cleanly but predates migrations the running code needs fails as a stale dump instead of passing. Copied out of the repo, the gate degrades to a warning plus the old any-nonzero check.
    • The irreplaceable tables below, present and (for service_keys) non-empty: bridged_actors, service_keys, ap_actors, blocks, repo_state, blobs, and the in-flight federation set — outbound_deliveries, outbound_activities, outbound_objects, outbound_votes, admissions.
    • The plc-rotation row must EXIST, exactly once and non-empty, whenever bridged_actors restored non-empty. An absent row is the dangerous case, not a present-but-empty one: it boots clean and mints a replacement key (see reason=missing above). When bridged_actors is empty too, absence is tolerated with a printed note — that is a fresh-install dump.
    • Zero-length sealed blobs, in bridged_actors.signing_key and ap_actors.rsa_key_sealed. A column that can only hold ciphertext holding nothing is corruption, distinct from NULL (which is a legitimately key-less actor).

RPO with the default schedule is 24 hours of database state. The drill validates data presence, not KEK correctness: sealed columns are checked present and non-zero-length — never opened, and never checked row by row for plausibility (a restore drill must not need to read the KEK). The full proof is drill + boot canary: restore, point a bridge at it with the real BRIDGE_KEK, and a wrong pairing fails at startup — the same canary as step 2 of the KEK runbook above. That holds even when the restore lost the plc-rotation row itself: with nothing sealed to read back, the canary proves the KEK against the actor keys that are there (ap_actors.rsa_key_sealed, bridged_actors.signing_key) and refuses to mint a replacement rotation key over a populated database it cannot read. Only a database with no sealed material anywhere — a genuinely fresh install — is allowed to mint on an unproven key.

Retention and drill cadence interact, and not in your favour. Retention ages dumps out on pg-backup.sh's own criteria, which is a full pg_restore read succeeding; a dump that reads cleanly but would fail the drill therefore keeps refreshing the retention window for up to RETENTION_DAYS while the last drilled dump ages out from under it. The drill is the deeper check and nothing runs it automatically: run it after the first backup, after any restore, and quarterly.

BRIDGE_KEK and .env are NOT in any database backup. Keep an offsite copy of .env (password manager or sealed storage, not this server), and touch backups/.env-backed-up after each copy — pg-backup.sh warns on every run where .env is newer than that marker, so a rotated or edited KEK that was never re-escrowed shows up in the backup's journal output instead of in an incident.

Residual limits, stated rather than implied: dumps live on the same host they protect (no offsite replication of ./backups yet — copying them into the same offsite routine as .env is the obvious next step), and WAL archiving / point-in-time recovery is deliberately not built at this scale.

What is at risk, in order of irreplaceability:

  1. BRIDGE_KEK — lives in /opt/tidepool/.env, not in Postgres, and is not covered by any database backup. Losing it is unrecoverable (above).
  2. bridged_actors / service_keys / ap_actors — the sealed signing keys, for bridged identities, the PLC escrow key, and every native Coves user respectively. Losing these loses the identities even if the KEK survives. ap_actors is easy to omit from a v1-era backup scope; it is the whole native-user population.
  3. Repo blocks and commits — the atproto repos themselves. Re-derivable from upstream only by re-bridging, which mints new DIDs; the old at-uris do not come back.
  4. outbound_*, admissions — in-flight federation state. Losing it double-sends or drops deliveries.

A divergence off switch #

There is none. The reconciliation sweep is wired unconditionally — the constructor and go divergence.Run(ctx) sit outside every flag at cmd/tidepool/main.go:485-506 — and Run performs one sweep immediately, before its first tick (internal/ingest/divergence.go:561-565). Setting DIVERGENCE_INTERVAL=0 does not disable it: durationVar rejects zero and negative values and the process refuses to start (internal/config/config.go:666-668).

The only available lever is a large interval — DIVERGENCE_INTERVAL=8760h quiets the background pass. The startup sweep still runs, once, on every boot, and GET /admin/divergence still works.

This is defensible: the sweep writes nothing, to peers or to our own tables, which is exactly what makes an always-on schedule safe. But if a sweep is ever implicated in an incident (lock pressure, a long multi-table scan — two of its legs scan outbound_activities in full, per FOLLOWUPS.md), the runbook answer is "raise the interval and restart", not "disable it", because disabling is not possible.

Periodic vote re-seed #

Nothing re-seeds vote aggregates on a schedule. SeedPostCounts has one caller, on the community-backfill path.

This makes one documented self-healing claim conditional in a way that matters: baseline-only voters can drift when a later flip or clear has no per-voter baseline row to retract, and the standing note is that "a re-seed heals the aggregate" (FOLLOWUPS.md, Votes). For a quiet community that is never backfilled again, the next re-seed is never. The drift is permanent, silently, and nothing reports it — the divergence sweep compares atproto state against outbound state, not vote aggregates against the origin instance.

The manual lever is a backfill of the affected community (POST /admin/communities/backfill), which re-seeds as a side effect. That is a workaround, not a scheduled heal, and it does other work besides.

7. Support matrix #

Peer Status Notes
Lemmy 0.19.20 Targeted. Strictness ceiling and e2e target Pinned in e2e/lemmy/Dockerfile:35, matching decision 19
PieFed Best-effort No pinned instance in the harness; conformance is held by captured-wire fixtures. Its votes arrive from anonymous per-user actors, which is fine for tallies but means no per-voter identity
Lemmy 1.0-beta Tracked, not targeted Vote FederationMode, inbox collapsing, and NoteWrapper all change behaviour we depend on. No harness coverage
Mastodon Incidental The security/v1 context is published so its parser accepts our publicKey, and its hosts appear in production handle subdomains. Not a target; not tested

Version pin — resolved #

e2e/lemmy/Dockerfile:35 pins ARG LEMMY_VERSION=0.19.20, matching PLAN.md decision 19 ("Lemmy 0.19.20 is the pinned strictness ceiling and e2e target"). The behaviours verified against 0.19.20 source during the v2 build — the Delete-summary convention, the check_bot_account rule, Instance-enum strictness — are the behaviours make e2e actually builds and tests against. (An earlier draft of this section described a 0.19.19/0.19.20 discrepancy; it was resolved by bumping the Dockerfile to 0.19.20.)

8. Known operational gaps carried forward #

Not new, but they shape what the runbook above can promise:

  • Per-IP rate limits degrade to global ones behind Caddy. Every in-process limiter keys on RemoteAddr and ignores X-Forwarded-For, so from behind the proxy they see only Caddy's container IP. Rate limiting at the edge is the fix; tracked in FOLLOWUPS.md.
  • ENVIRONMENT=production has never been exercised end to end. The e2e harness runs in development mode (migrations-on-start, HTTP, private fetch, strict lexicon validation). The production-only refusals — BRIDGE_SCHEME=http, ALLOW_PRIVATE_FETCH, AP_HOST_FALLTHROUGH_DEV, and the AP_USER_ORIGIN https rule — are unit-covered, not harness-covered. Section 1 exists because of this.
  • Production lexicon validation records and writes rather than failing. A strict-first rollout should wait until tidepool_lexicon_validation_failures stays at zero in production.