93 lines
4 KiB
Markdown
93 lines
4 KiB
Markdown
|
# `pgbackrest` role — Postgres backup + WAL archive to MinIO/S3
|
||
|
|
|
||
|
|
Wires pgBackRest into the pg_auto_failover formation: full backup weekly, differential daily, WAL continuously to a MinIO bucket. Backups encrypted at rest with `aes-256-cbc`. The dr-drill script (`scripts/dr-drill.sh`) restores into an ephemeral Incus container and asserts the data round-trips — runs weekly via systemd timer, exposes a textfile metric for Prometheus.
|
||
|
|
|
||
|
|
## Cadence
|
||
|
|
|
||
|
|
| job | when | schedule (defaults) | metric source |
|
||
|
|
| ---- | --------------------- | -------------------------------- | ------------------------- |
|
||
|
|
| full | weekly Sun 02:00 UTC | `pgbackrest_schedule_full` | systemd journald |
|
||
|
|
| diff | daily Mon-Sat 02:00 | `pgbackrest_schedule_diff` | systemd journald |
|
||
|
|
| WAL | continuous (per file) | postgres `archive_command` | postgres logs + pgBackRest |
|
||
|
|
| drill| weekly Sun 04:00 UTC | `pgbackrest_drill_schedule` | textfile collector .prom |
|
||
|
|
|
||
|
|
## RPO / RTO
|
||
|
|
|
||
|
|
- **RPO** ≈ 1 minute. `archive_timeout=60` forces a WAL switch + push every minute even when traffic is low. Worst-case data loss on a primary's death: the last 60s of WAL that hadn't shipped yet.
|
||
|
|
- **RTO** ≤ 30 min. The dr-drill restore runs end-to-end in ~10-20 min on the lab; production should match given the same backup size.
|
||
|
|
|
||
|
|
## Vault setup
|
||
|
|
|
||
|
|
Three secrets — never committed:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# group_vars/postgres_ha.vault.yml (encrypted)
|
||
|
|
vault_pgbackrest_s3_key: "<MinIO access key>"
|
||
|
|
vault_pgbackrest_s3_key_secret: "<MinIO secret key>"
|
||
|
|
vault_pgbackrest_cipher_pass: "<random 64-char passphrase>"
|
||
|
|
```
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ansible-vault encrypt infra/ansible/group_vars/postgres_ha.vault.yml
|
||
|
|
```
|
||
|
|
|
||
|
|
The role's first task asserts the placeholders are gone — applying with the placeholder defaults aborts loud rather than rolling out a misconfigured archive.
|
||
|
|
|
||
|
|
## Repo wiring
|
||
|
|
|
||
|
|
```ini
|
||
|
|
repo1-type = s3
|
||
|
|
repo1-s3-endpoint = minio.lxd:9000
|
||
|
|
repo1-s3-bucket = veza-pgbackrest
|
||
|
|
repo1-s3-uri-style = path # MinIO speaks path-style by default
|
||
|
|
repo1-cipher-type = aes-256-cbc
|
||
|
|
```
|
||
|
|
|
||
|
|
Bucket created by the `minio_distributed` role (W3 day 12). Until then operators bootstrap with:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
mc alias set veza-minio https://minio.lxd:9000 <ACCESS> <SECRET>
|
||
|
|
mc mb veza-minio/veza-pgbackrest
|
||
|
|
```
|
||
|
|
|
||
|
|
## Operations
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Backup status — most recent full + diff + WAL window:
|
||
|
|
sudo -u postgres pgbackrest --stanza=veza info
|
||
|
|
|
||
|
|
# Manual full backup (use sparingly — it's bandwidth-heavy):
|
||
|
|
sudo systemctl start pgbackrest-full.service
|
||
|
|
|
||
|
|
# Tail the most recent backup log:
|
||
|
|
sudo journalctl -u pgbackrest-full.service -n 200 --no-pager
|
||
|
|
|
||
|
|
# Verify the archive pipeline is healthy (last WAL ship time):
|
||
|
|
sudo -u postgres pgbackrest --stanza=veza check
|
||
|
|
```
|
||
|
|
|
||
|
|
## Restore — the dr-drill
|
||
|
|
|
||
|
|
```bash
|
||
|
|
bash scripts/dr-drill.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
Sequence:
|
||
|
|
|
||
|
|
1. Read latest backup label via `pgbackrest info`
|
||
|
|
2. Launch ephemeral Incus container `pg-restore-drill`
|
||
|
|
3. Install postgres + pgbackrest inside, render the same `pgbackrest.conf` (read-only mode against the same bucket)
|
||
|
|
4. `pgbackrest --stanza=veza restore` to recover
|
||
|
|
5. Start postgres
|
||
|
|
6. Connect, run `SELECT count(*) FROM users` — must be > 0 (proves the seed data round-tripped)
|
||
|
|
7. Write `veza_backup_drill_*` metrics to `pgbackrest_drill_metrics_file`
|
||
|
|
8. Tear down the container (or keep it for inspection if `--keep` is passed)
|
||
|
|
|
||
|
|
The metrics file is scraped by node_exporter's `--collector.textfile.directory`. Prometheus alert `BackupRestoreDrillFailed` (added in `config/prometheus/alert_rules.yml`) fires when the last successful drill is older than 8 days, OR when the most recent run reported a non-zero exit code.
|
||
|
|
|
||
|
|
## What this role does NOT cover
|
||
|
|
|
||
|
|
- **Off-site replica** — the bucket is single-region MinIO. v1.1+ adds Bunny.net or B2 as a secondary repo (`repo2-*`).
|
||
|
|
- **Point-in-time UI** — restore is CLI-only via `--type=time`. Operator-driven, no admin dashboard.
|
||
|
|
- **Logical export** — for legal/RGPD requests, `pg_dump` of the relevant rows is a separate path; the binary backups in this role aren't designed to be partially extracted.
|