# Maintenance

## Health Check

Run:

```sh
scripts/healthcheck.sh
```

From the VM, also inspect the service:

```sh
sudo systemctl status pds
sudo docker ps
sudo docker logs --tail 200 pds
```

If the container name differs, use `docker ps` to find it.

## Update PDS

Use the update wrapper so a fresh backup is attempted first:

```sh
sudo scripts/update-pds.sh
```

The script expects `scripts/backup-pds.sh` to work with production backup credentials.

If the update fails:

- keep the VM running for inspection
- collect `systemctl status pds` and recent container logs
- do not delete `/pds`
- restore only if the current state is unrecoverable

## Backup Operations

Run a manual backup:

```sh
sudo /usr/local/sbin/backup-pds.sh
```

Inspect logs:

```sh
sudo journalctl -u pds-backup.service -n 100 --no-pager
```

The backup script stops PDS briefly for consistency, then restarts it. Expect a short Cloudflare 521 window immediately after a manual backup while Docker/Caddy/PDS start again.

## Backup Retention

The backup script does not delete stale R2 backups itself. Retention is configured as an R2 lifecycle rule on the `<pds-backup-bucket>` bucket:

```text
Rule ID: expire-pds-backups-after-7-days
Prefix: pds/
Action: Delete completed objects
Age: 7 days
```

The bucket also keeps Cloudflare's default incomplete multipart upload cleanup rule:

```text
Rule ID: Default Multipart Abort Rule
Action: Abort incomplete multipart uploads
Age: 7 days
```

Check timer:

```sh
systemctl list-timers pds-backup.timer
```

## Backup Validation

At least once after initial deployment, and after any script change:

```sh
scripts/restore-pds.sh --object '<backup-object-key>' --target /tmp/pds-restore-test
```

Then check that the restored tree contains:

- `pds.env`
- SQLite database files
- service config and metadata
- certificate references or files if stored under `/pds`
- account and PLC recovery material

Delete only the temporary restore directory after validation.

## Certificate Replacement

Cloudflare Origin CA certificates are not publicly trusted certificates. Replace them before expiration:

1. Generate a new Origin CA certificate for `pds.3drx.top`.
2. Copy the certificate and key to the VM outside Git.
3. Set restrictive key permissions.
4. Update Caddy config if paths changed.
5. Reload Caddy/PDS.
6. Verify Cloudflare Full (strict) still succeeds.

## Logs

Useful commands:

```sh
sudo journalctl -u pds -n 200 --no-pager
sudo docker logs --tail 200 pds
sudo docker logs -f pds
```

Avoid copying logs into public places without reviewing them for account identifiers, tokens, email addresses, and operational details.

## Secret Rotation

See [Secrets](secrets.md). Rotate one secret type at a time and run health checks after each change.