veza/infra/ansible/roles/pgbackrest

pgbackrest role — Postgres backup + WAL archive to MinIO/S3

Wires pgBackRest into the pg_auto_failover formation: full backup weekly, differential daily, WAL continuously to a MinIO bucket. Backups encrypted at rest with aes-256-cbc. The dr-drill script (scripts/dr-drill.sh) restores into an ephemeral Incus container and asserts the data round-trips — runs weekly via systemd timer, exposes a textfile metric for Prometheus.

Cadence

job	when	schedule (defaults)	metric source
full	weekly Sun 02:00 UTC	pgbackrest_schedule_full	systemd journald
diff	daily Mon-Sat 02:00	pgbackrest_schedule_diff	systemd journald
WAL	continuous (per file)	postgres archive_command	postgres logs + pgBackRest
drill	weekly Sun 04:00 UTC	pgbackrest_drill_schedule	textfile collector .prom

RPO / RTO

• RPO ≈ 1 minute. archive_timeout=60 forces a WAL switch + push every minute even when traffic is low. Worst-case data loss on a primary's death: the last 60s of WAL that hadn't shipped yet.
• RTO ≤ 30 min. The dr-drill restore runs end-to-end in ~10-20 min on the lab; production should match given the same backup size.

Vault setup

Three secrets — never committed:

# group_vars/postgres_ha.vault.yml (encrypted)
vault_pgbackrest_s3_key: "<MinIO access key>"
vault_pgbackrest_s3_key_secret: "<MinIO secret key>"
vault_pgbackrest_cipher_pass: "<random 64-char passphrase>"

ansible-vault encrypt infra/ansible/group_vars/postgres_ha.vault.yml

The role's first task asserts the placeholders are gone — applying with the placeholder defaults aborts loud rather than rolling out a misconfigured archive.

Repo wiring

repo1-type        = s3
repo1-s3-endpoint = minio.lxd:9000
repo1-s3-bucket   = veza-pgbackrest
repo1-s3-uri-style = path             # MinIO speaks path-style by default
repo1-cipher-type = aes-256-cbc

Bucket created by the minio_distributed role (W3 day 12). Until then operators bootstrap with:

mc alias set veza-minio https://minio.lxd:9000 <ACCESS> <SECRET>
mc mb veza-minio/veza-pgbackrest

Operations

# Backup status — most recent full + diff + WAL window:
sudo -u postgres pgbackrest --stanza=veza info

# Manual full backup (use sparingly — it's bandwidth-heavy):
sudo systemctl start pgbackrest-full.service

# Tail the most recent backup log:
sudo journalctl -u pgbackrest-full.service -n 200 --no-pager

# Verify the archive pipeline is healthy (last WAL ship time):
sudo -u postgres pgbackrest --stanza=veza check

Restore — the dr-drill

bash scripts/dr-drill.sh

Sequence:

1. Read latest backup label via pgbackrest info
2. Launch ephemeral Incus container pg-restore-drill
3. Install postgres + pgbackrest inside, render the same pgbackrest.conf (read-only mode against the same bucket)
4. pgbackrest --stanza=veza restore to recover
5. Start postgres
6. Connect, run SELECT count(*) FROM users — must be > 0 (proves the seed data round-tripped)
7. Write veza_backup_drill_* metrics to pgbackrest_drill_metrics_file
8. Tear down the container (or keep it for inspection if --keep is passed)

The metrics file is scraped by node_exporter's --collector.textfile.directory. Prometheus alert BackupRestoreDrillFailed (added in config/prometheus/alert_rules.yml) fires when the last successful drill is older than 8 days, OR when the most recent run reported a non-zero exit code.

What this role does NOT cover

• Off-site replica — the bucket is single-region MinIO. v1.1+ adds Bunny.net or B2 as a secondary repo (repo2-*).
• Point-in-time UI — restore is CLI-only via --type=time. Operator-driven, no admin dashboard.
• Logical export — for legal/RGPD requests, pg_dump of the relevant rows is a separate path; the binary backups in this role aren't designed to be partially extracted.