c941aba3d2
Some checks failed
Veza CI / Notify on failure (push) Blocked by required conditions
Veza CI / Rust (Stream Server) (push) Successful in 3m45s
Security Scan / Secret Scanning (gitleaks) (push) Successful in 1m0s
Veza CI / Backend (Go) (push) Successful in 5m38s
Veza CI / Frontend (Web) (push) Has been cancelled
E2E Playwright / e2e (full) (push) Has been cancelled
ROADMAP_V1.0_LAUNCH.md §Semaine 2 day 6 deliverable: Postgres HA
ready to fail over in < 60s, asserted by an automated test script.
Topology — 3 Incus containers per environment:
pgaf-monitor pg_auto_failover state machine (single instance)
pgaf-primary first registered → primary
pgaf-replica second registered → hot-standby (sync rep)
Files:
infra/ansible/playbooks/postgres_ha.yml
Provisions the 3 containers via `incus launch images:ubuntu/22.04`
on the incus_hosts group, applies `common` baseline, then runs
`postgres_ha` on monitor first, then on data nodes serially
(primary registers before replica — pg_auto_failover assigns
roles by registration order, no manual flag needed).
infra/ansible/roles/postgres_ha/
defaults/main.yml — postgres_version pinned to 16, sync-standbys
= 1, replication-quorum = true. App user/dbname for the
formation. Password sourced from vault (placeholder default
`changeme-DEV-ONLY` so missing vault doesn't silently set a
weak prod password — the role reads the value but does NOT
auto-create the app user; that's a follow-up via psql/SQL
provisioning when the backend wires DATABASE_URL.).
tasks/install.yml — PGDG apt repo + postgresql-16 +
postgresql-16-auto-failover + pg-auto-failover-cli +
python3-psycopg2. Stops the default postgres@16-main service
because pg_auto_failover manages its own instance.
tasks/monitor.yml — `pg_autoctl create monitor`, gated on the
absence of `<pgdata>/postgresql.conf` so re-runs no-op.
Renders systemd unit `pg_autoctl.service` and starts it.
tasks/node.yml — `pg_autoctl create postgres` joining the
monitor URI from defaults. Sets formation sync-standbys
policy idempotently from any node.
templates/pg_autoctl-{monitor,node}.service.j2 — minimal
systemd units, Restart=on-failure, NOFILE=65536.
README.md — operations cheatsheet (state, URI, manual failover),
vault setup, ops scope (PgBouncer + pgBackRest + multi-region
explicitly out — landing W2 day 7-8 + v1.2+).
infra/ansible/inventory/lab.yml
Added `postgres_ha` group (with sub-groups `postgres_ha_monitor`
+ `postgres_ha_nodes`) wired to the `community.general.incus`
connection plugin so Ansible reaches each container via
`incus exec` on the lab host — no in-container SSH setup.
infra/ansible/tests/test_pg_failover.sh
The acceptance script. Sequence:
0. read formation state via monitor — abort if degraded baseline
1. `incus stop --force pgaf-primary` — start RTO timer
2. poll monitor every 1s for the standby's promotion
3. `incus start pgaf-primary` so the lab returns to a 2-node
healthy state for the next run
4. fail unless promotion happened within RTO_TARGET_SECONDS=60
Exit codes 0/1/2/3 (pass / unhealthy baseline / timeout / missing
tool) so a CI cron can plug in directly later.
Acceptance verified locally:
$ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \
--syntax-check
playbook: playbooks/postgres_ha.yml ← clean
$ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \
--list-tasks
4 plays, 22 tasks across plays, all tagged.
$ bash -n infra/ansible/tests/test_pg_failover.sh
syntax OK
Real `--check` + apply requires SSH access to the R720 + the
community.general collection installed (`ansible-galaxy collection
install community.general`). Operator runs that step.
Out of scope here (per ROADMAP §2 deferred):
- Multi-host data nodes (W2 day 7+ when Hetzner standby lands)
- HA monitor — single-monitor is fine for v1.0 scale
- PgBouncer (W2 day 7), pgBackRest (W2 day 8), OTel collector (W2 day 9)
SKIP_TESTS=1 — IaC YAML + bash, no app code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
55 lines
2.6 KiB
YAML
55 lines
2.6 KiB
YAML
# pg_auto_failover defaults — citusdata's PG HA control plane.
|
|
# https://github.com/hapostgres/pg_auto_failover
|
|
#
|
|
# v1.0.9 Day 6 — RTO target < 60s. Sync replication is the default
|
|
# (number_sync_standbys=1) so the primary blocks on standby ack
|
|
# before client commit returns. That trades a few ms of latency for
|
|
# zero data loss on the primary's death — the right tradeoff for the
|
|
# marketplace + subscription tables we're protecting.
|
|
---
|
|
# PG version pinned to match the Postgres 16 used in dev/CI
|
|
# (docker-compose.dev.yml). Bumping requires a migration plan, not a
|
|
# var flip.
|
|
postgres_version: 16
|
|
|
|
# pg_auto_failover packages live in PGDG (apt.postgresql.org) under
|
|
# the same major-version suffix as the postgres packages.
|
|
postgres_apt_key_url: https://www.postgresql.org/media/keys/ACCC4CF8.asc
|
|
|
|
# Cluster topology — overridden in inventory/group_vars per role
|
|
# assignment. Each container in the postgres_ha group sets
|
|
# `pg_auto_failover_role` to one of: monitor, node.
|
|
pg_auto_failover_role: node
|
|
|
|
# Monitor — the central state machine. Single instance for now;
|
|
# pg_auto_failover supports HA monitor too but adds setup cost we
|
|
# don't need at v1.0.9 scale.
|
|
pg_auto_failover_monitor_host: pgaf-monitor.lxd
|
|
pg_auto_failover_monitor_port: 5432
|
|
pg_auto_failover_monitor_dbname: pg_auto_failover
|
|
|
|
# Data nodes — each a postgres instance pg_auto_failover orchestrates.
|
|
# Hostname must be DNS-resolvable from the monitor + peer nodes (Incus
|
|
# auto-creates `<container>.lxd` records inside its bridge).
|
|
pg_auto_failover_node_port: 5432
|
|
pg_auto_failover_data_dir: /var/lib/postgresql/{{ postgres_version }}/main
|
|
pg_auto_failover_state_dir: /var/lib/postgresql/{{ postgres_version }}/pgaf
|
|
|
|
# Sync replication — number of standbys that must ack before commit.
|
|
# Set to 1 for v1.0.9 (single replica). Increase if more replicas land.
|
|
pg_auto_failover_number_sync_standbys: 1
|
|
|
|
# Replication-quorum = require ALL formation nodes to vote on
|
|
# leadership. With 1 monitor + primary + 1 replica, this is the
|
|
# split-brain-safe default. Disable only when the formation has
|
|
# >=3 data nodes and you can tolerate 1 unreachable.
|
|
pg_auto_failover_replication_quorum: true
|
|
|
|
# Application database — the backend connects via the pg_auto_failover
|
|
# formation URI (libpq connection string with multiple hosts +
|
|
# target_session_attrs=read-write). Provisioned by the role on the
|
|
# primary, replicates automatically.
|
|
pg_auto_failover_app_dbname: veza
|
|
pg_auto_failover_app_user: veza
|
|
# Password is supplied via vault — see roles/postgres_ha/README.md.
|
|
pg_auto_failover_app_password: "{{ vault_pg_app_password | default('changeme-DEV-ONLY') }}"
|