atstack
Personal atproto services deployment handbook
Star
Fork
Watch
atstack docs maintenance.md
2.8 kB · 122 lines
Markdown
at main

Maintenance #

Health Check #

Run:

scripts/healthcheck.sh

From the VM, also inspect the service:

sudo systemctl status pds
sudo docker ps
sudo docker logs --tail 200 pds

If the container name differs, use docker ps to find it.

Update PDS #

Use the update wrapper so a fresh backup is attempted first:

sudo scripts/update-pds.sh

The script expects scripts/backup-pds.sh to work with production backup credentials.

If the update fails:

  • keep the VM running for inspection
  • collect systemctl status pds and recent container logs
  • do not delete /pds
  • restore only if the current state is unrecoverable

Backup Operations #

Run a manual backup:

sudo /usr/local/sbin/backup-pds.sh

Inspect logs:

sudo journalctl -u pds-backup.service -n 100 --no-pager

The backup script stops PDS briefly for consistency, then restarts it. Expect a short Cloudflare 521 window immediately after a manual backup while Docker/Caddy/PDS start again.

Backup Retention #

The backup script does not delete stale R2 backups itself. Retention is configured as an R2 lifecycle rule on the <pds-backup-bucket> bucket:

Rule ID: expire-pds-backups-after-7-days
Prefix: pds/
Action: Delete completed objects
Age: 7 days

The bucket also keeps Cloudflare's default incomplete multipart upload cleanup rule:

Rule ID: Default Multipart Abort Rule
Action: Abort incomplete multipart uploads
Age: 7 days

Check timer:

systemctl list-timers pds-backup.timer

Backup Validation #

At least once after initial deployment, and after any script change:

scripts/restore-pds.sh --object '<backup-object-key>' --target /tmp/pds-restore-test

Then check that the restored tree contains:

  • pds.env
  • SQLite database files
  • service config and metadata
  • certificate references or files if stored under /pds
  • account and PLC recovery material

Delete only the temporary restore directory after validation.

Certificate Replacement #

Cloudflare Origin CA certificates are not publicly trusted certificates. Replace them before expiration:

  1. Generate a new Origin CA certificate for pds.3drx.top.
  2. Copy the certificate and key to the VM outside Git.
  3. Set restrictive key permissions.
  4. Update Caddy config if paths changed.
  5. Reload Caddy/PDS.
  6. Verify Cloudflare Full (strict) still succeeds.

Logs #

Useful commands:

sudo journalctl -u pds -n 200 --no-pager
sudo docker logs --tail 200 pds
sudo docker logs -f pds

Avoid copying logs into public places without reviewing them for account identifiers, tokens, email addresses, and operational details.

Secret Rotation #

See Secrets. Rotate one secret type at a time and run health checks after each change.