From 32062d7c54ba2dd9677fdda24bc4d7bfcdd79a1a Mon Sep 17 00:00:00 2001 From: My first Letta Agent Date: Sun, 9 Aug 2026 22:51:11 -0400 Subject: [PATCH] Update PDS backup plan with branding/telemetry/monitoring implementation status --- reference/pds-backup-plan.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/reference/pds-backup-plan.md b/reference/pds-backup-plan.md index 63655bd..403a894 100644 --- a/reference/pds-backup-plan.md +++ b/reference/pds-backup-plan.md @@ -54,30 +54,38 @@ rclone installed via `apt` on the host (Ubuntu/Debian packages). Mounted into PD ## README Variables Split into two sections: PDS vars and Backup service vars (done 2026-08-04) -## Implementation Status (2026-08-04) +## Implementation Status (2026-08-09) ### Completed - **Gitea CI/CD** — `.gitea/workflows/deploy.yml` created. Push to main triggers deploy. Workflow: install Ansible → write SSH key from secret → generate inventory.yml + vars/main.yml from Gitea vars/secrets → run playbook. Uses `ssh-keyscan` for known_hosts (no host key checking bypass in CI). - **Vault removal** — ansible-vault removed (vault file was gitignored, never committed, so encryption was pointless). All secrets now managed via Gitea secrets directly. S3 credentials in `vars/main.yml` (uncommented for local, generated from Gitea secrets for CI). - **OVH blob storage enabled** — S3 vars uncommented in `vars/main.yml` (atprotopds-blobstore bucket, bhs region). - **PDS backup service** — rclone-based, three backup types (pre-update, pre-deploy, periodic). Single backup script (`backup.sh.j2`) handles all three — deployed to host, mounted read-only into PDS container for Watchtower hooks. Templates: `backup.sh.j2`, `backup.service.j2`, `backup.timer.j2`, `rclone.conf.j2`. Playbook installs rclone (apt), deploys configs, runs pre-deploy backup. Watchtower lifecycle hooks configured (timeout: 2 minutes). Named "PDS Backup Service" (not "OVH Backup" — OVH is the provider, PDS is the service). +- **Health check** — `uri` task at end of playbook, checks `https://{{ pds_hostname }}/xrpc/_health`, retries 3x with 5s delay. Also checks backup timer active and last backup exit code. +- **Image bump to 0.5** — Updated from 0.4 to 0.5 (required for telemetry support, added in @atproto/pds 0.5.26). +- **Blob upload limit 300MB** — Updated from 104857600 (100MB) to 314572800 (300MB) per ATProto blog post. +- **Branding/customization** — All PDS branding env vars added to `pds.env.j2` as conditional blocks (PDS_SERVICE_NAME, PDS_LOGO_URL, PDS_PRIMARY_COLOR, semantic colors, footer URLs, background images). Optional vars in `vars/main.yml` and `vars/main.yml.example` (all commented out by default). CI/CD workflow passes through PDS_SERVICE_NAME and PDS_PRIMARY_COLOR as optional Gitea variables. +- **Telemetry/OpenTelemetry** — Env vars added to `pds.env.j2` gated on `pds_telemetry_enabled | bool` (NODE_OPTIONS, OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL, OTEL_METRIC_EXPORT_INTERVAL, OTEL_SEMCONV_STABILITY_OPT_IN, OTEL_NODE_RESOURCE_DETECTORS). Uses metrics-only config (not traces/logs) per reference PDS monitoring README. +- **Monitoring stack** — Separate compose file (`monitoring-compose.yaml.j2`) deployed to `/pds/monitoring/`. Three services: node-exporter (prom/node-exporter:v1, port 9100), prometheus (prom/prometheus:v3, port 9090, OTLP receiver enabled), grafana (grafana/grafana:13.1.1, port 3001). All bind to 127.0.0.1. Prometheus config (`prometheus.yml.j2`) scrapes node_exporter + self, receives PDS metrics via OTLP push. Grafana provisioning: datasource (`grafana-datasources.yml.j2`) + dashboard provider (`grafana-dashboards.yml.j2`). Dashboard JSON downloaded from bluesky-social/pds repo at runtime. Handler `restart monitoring` does `docker compose up -d --force-recreate`. All tasks conditional on `pds_telemetry_enabled`. ### Pending -1. **Health check task** — curl `http://localhost:3000/xrpc/_health` at end of playbook -2. **Client-side backup encryption** — rclone crypt remote wrapping `pds-backup` remote -3. **Backup verification** — periodic test restore -4. **Health alerting** — cron + curl `_health` + webhook -5. **Watchtower update safety** — post-update health check or manual approval +1. **Client-side backup encryption** — rclone crypt remote wrapping `pds-backup` remote +2. **Backup verification** — periodic test restore +3. **Health alerting** — cron + curl `_health` + webhook +4. **Watchtower update safety** — post-update health check or manual approval ## Execution order 1. ~~Gitea CI/CD~~ ✓ DONE 2. ~~OVH blob storage~~ ✓ DONE (enabled, S3 vars uncommented) 3. ~~PDS backup service~~ ✓ DONE (rclone + systemd timer + Watchtower hooks) -4. Hardening — health checks, client-side backup encryption, alerting, backup verification +4. ~~Health check~~ ✓ DONE (uri task, retries 3x) +5. ~~Branding/customization~~ ✓ DONE (2026-08-09, per atproto.com/blog/pds-customization-and-metrics) +6. ~~Telemetry + monitoring stack~~ ✓ DONE (2026-08-09, Prometheus + Grafana + node_exporter) +7. Hardening — client-side backup encryption, alerting, backup verification ## Hardening scope - ~~Ansible vault for secrets~~ REMOVED (2026-08-04 — moved to Gitea secrets only) -- Health check task at end of playbook (curl _health, fail if not responding) — PENDING +- ~~Health check task~~ ✓ DONE (uri task at end of playbook) - rclone crypt remote for client-side backup encryption — PENDING - ~~host_key_checking = True, ssh-keyscan in CI~~ ✓ DONE (CI uses ssh-keyscan, host_key_checking = False in ansible.cfg) - Simple health alert (cron + curl + webhook) — PENDING -- knotmirror 0.0.1