bf31a91ae6
Some checks failed
Veza CI / Frontend (Web) (push) Failing after 16m6s
Veza CI / Notify on failure (push) Successful in 11s
E2E Playwright / e2e (full) (push) Successful in 19m59s
Veza CI / Rust (Stream Server) (push) Successful in 4m57s
Security Scan / Secret Scanning (gitleaks) (push) Successful in 49s
Veza CI / Backend (Go) (push) Successful in 6m4s
ROADMAP_V1.0_LAUNCH.md §Semaine 2 day 8 deliverable:
- Postgres backups land in MinIO via pgbackrest
- dr-drill restores them weekly into an ephemeral Incus container
and asserts the data round-trips
- Prometheus alerts fire when the drill fails OR when the timer
has stopped firing for >8 days
Cadence:
full — weekly (Sun 02:00 UTC, systemd timer)
diff — daily (Mon-Sat 02:00 UTC, systemd timer)
WAL — continuous (postgres archive_command, archive_timeout=60s)
drill — weekly (Sun 04:00 UTC — runs 2h after the Sun full so
the restore exercises fresh data)
RPO ≈ 1 min (archive_timeout). RTO ≤ 30 min (drill measures actual
restore wall-clock).
Files:
infra/ansible/roles/pgbackrest/
defaults/main.yml — repo1-* config (MinIO/S3, path-style,
aes-256-cbc encryption, vault-backed creds), retention 4 full
/ 7 diff / 4 archive cycles, zstd@3 compression. The role's
first task asserts the placeholder secrets are gone — refuses
to apply until the vault carries real keys.
tasks/main.yml — install pgbackrest, render
/etc/pgbackrest/pgbackrest.conf, set archive_command on the
postgres instance via ALTER SYSTEM, detect role at runtime
via `pg_autoctl show state --json`, stanza-create from primary
only, render + enable systemd timers (full + diff + drill).
templates/pgbackrest.conf.j2 — global + per-stanza sections;
pg1-path defaults to the pg_auto_failover state dir so the
role plugs straight into the Day 6 formation.
templates/pgbackrest-{full,diff,drill}.{service,timer}.j2 —
systemd units. Backup services run as `postgres`,
drill service runs as `root` (needs `incus`).
RandomizedDelaySec on every timer to absorb clock skew + node
collision risk.
README.md — RPO/RTO guarantees, vault setup, repo wiring,
operational cheatsheet (info / check / manual backup),
restore procedure documented separately as the dr-drill.
scripts/dr-drill.sh
Acceptance script for the day. Sequence:
0. pre-flight: required tools, latest backup metadata visible
1. launch ephemeral `pg-restore-drill` Incus container
2. install postgres + pgbackrest inside, push the SAME
pgbackrest.conf as the host (read-only against the bucket
by pgbackrest semantics — the same s3 keys get reused so
the drill exercises the production credential path)
3. `pgbackrest restore` — full + WAL replay
4. start postgres, wait for pg_isready
5. smoke query: SELECT count(*) FROM users — must be ≥ MIN_USERS_EXPECTED
6. write veza_backup_drill_* metrics to the textfile-collector
7. teardown (or --keep for postmortem inspection)
Exit codes 0/1/2 (pass / drill failure / env problem) so a
Prometheus runner can plug in directly.
config/prometheus/alert_rules.yml — new `veza_backup` group:
- BackupRestoreDrillFailed (critical, 5m): the last drill
reported success=0. Pages because a backup we haven't proved
restorable is dette technique waiting for a disaster.
- BackupRestoreDrillStale (warning, 1h after >8 days): the
drill timer has stopped firing. Catches a broken cron / unit
/ runner before the failure-mode alert above ever sees data.
Both annotations include a runbook_url stub
(veza.fr/runbooks/...) — those land alongside W2 day 10's
SLO runbook batch.
infra/ansible/playbooks/postgres_ha.yml
Two new plays:
6. apply pgbackrest role to postgres_ha_nodes (install +
config + full/diff timers on every data node;
pgbackrest's repo lock arbitrates collision)
7. install dr-drill on the incus_hosts group (push
/usr/local/bin/dr-drill.sh + render drill timer + ensure
/var/lib/node_exporter/textfile_collector exists)
Acceptance verified locally:
$ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \
--syntax-check
playbook: playbooks/postgres_ha.yml ← clean
$ python3 -c "import yaml; yaml.safe_load(open('config/prometheus/alert_rules.yml'))"
YAML OK
$ bash -n scripts/dr-drill.sh
syntax OK
Real apply + drill needs the lab R720 + a populated MinIO bucket
+ the secrets in vault — operator's call.
Out of scope (deferred per ROADMAP §2):
- Off-site backup replica (B2 / Bunny.net) — v1.1+
- Logical export pipeline for RGPD per-user dumps — separate
feature track, not a backup-system concern
- PITR admin UI — CLI-only via `--type=time` for v1.0
- pgbackrest_exporter Prometheus integration — W2 day 9
alongside the OTel collector
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
66 lines
3.1 KiB
YAML
66 lines
3.1 KiB
YAML
# pgBackRest defaults — Postgres backup + WAL archive to MinIO/S3.
|
|
# https://pgbackrest.org
|
|
#
|
|
# v1.0.9 Day 8 — RPO target ≈ 5 min (WAL archive interval), RTO
|
|
# target < 30 min (the dr-drill timing budget). Backup cadence:
|
|
# - full : weekly (Sunday 02:00 UTC)
|
|
# - diff : daily Mon-Sat 02:00 UTC
|
|
# - WAL : continuous (archive_command after every WAL file)
|
|
---
|
|
postgres_version: 16
|
|
|
|
# Repository — MinIO is S3-compatible. The bucket is provisioned by
|
|
# the minio_distributed role (W3 day 12); until then operators
|
|
# create it manually with `mc mb minio/veza-pgbackrest`.
|
|
pgbackrest_repo_type: s3
|
|
pgbackrest_repo_s3_endpoint: minio.lxd:9000
|
|
pgbackrest_repo_s3_region: us-east-1 # MinIO ignores region but pgbackrest requires the field
|
|
pgbackrest_repo_s3_bucket: veza-pgbackrest
|
|
pgbackrest_repo_s3_uri_style: path # MinIO speaks path-style by default
|
|
pgbackrest_repo_s3_verify_tls: false # lab MinIO uses self-signed certs; flip to true once Let's Encrypt is wired
|
|
pgbackrest_repo_path: /
|
|
pgbackrest_repo_cipher_type: aes-256-cbc
|
|
|
|
# Stanza — pgBackRest's name for one Postgres cluster's archive.
|
|
# Single stanza per environment for v1.0.9 (one formation = one
|
|
# stanza). Multi-cluster envs add suffixed stanzas.
|
|
pgbackrest_stanza: veza
|
|
|
|
# Retention — keep 4 full backups (≈ 1 month at weekly cadence).
|
|
# diff/incremental retention is implicit (kept until the parent
|
|
# full expires). WAL is kept as long as any in-window full needs
|
|
# it for PITR.
|
|
pgbackrest_repo_retention_full: 4
|
|
pgbackrest_repo_retention_diff: 7
|
|
pgbackrest_repo_retention_archive: 4
|
|
|
|
# Compression — zstd@3 trades CPU for ~50% smaller archive vs gz@6.
|
|
# The CPU budget is fine on the R720; bandwidth to MinIO is the
|
|
# scarcer resource.
|
|
pgbackrest_compress_type: zstd
|
|
pgbackrest_compress_level: 3
|
|
|
|
# Process count — parallel WAL push + parallel backup. 4 is right
|
|
# for the 12-core R720 with concurrent backend traffic; bump in
|
|
# `group_vars/<env>` for dedicated-host backups.
|
|
pgbackrest_process_max: 4
|
|
|
|
# Secrets — sourced from vault. The role refuses to apply when
|
|
# placeholders are still in place to prevent a "live" rollout
|
|
# pointing at the wrong S3 keys.
|
|
pgbackrest_repo_s3_key: "{{ vault_pgbackrest_s3_key | default('CHANGEME-PGBR-KEY') }}"
|
|
pgbackrest_repo_s3_key_secret: "{{ vault_pgbackrest_s3_key_secret | default('CHANGEME-PGBR-SECRET') }}"
|
|
pgbackrest_repo_cipher_pass: "{{ vault_pgbackrest_cipher_pass | default('CHANGEME-PGBR-CIPHER') }}"
|
|
|
|
# Schedule — systemd timers (preferred over cron for journald
|
|
# integration). Override per env in group_vars.
|
|
pgbackrest_schedule_full: "Sun *-*-* 02:00:00"
|
|
pgbackrest_schedule_diff: "Mon..Sat *-*-* 02:00:00"
|
|
|
|
# Drill schedule — weekly RTO check, runs from any host with
|
|
# `incus` access. `dr-drill.sh` writes a textfile metric the
|
|
# node_exporter scrapes; the Prometheus alert
|
|
# BackupRestoreDrillFailed fires when the timestamp gets stale
|
|
# (> 8 days = a week's drill missed entirely).
|
|
pgbackrest_drill_schedule: "Sun *-*-* 04:00:00"
|
|
pgbackrest_drill_metrics_file: /var/lib/node_exporter/textfile_collector/pgbackrest_drill.prom
|