veza/infra/ansible/roles/postgres_ha/defaults/main.yml

56 lines
2.6 KiB
YAML
Raw Normal View History

feat(infra): postgres_ha role + pg_auto_failover formation + RTO test (W2 Day 6) ROADMAP_V1.0_LAUNCH.md §Semaine 2 day 6 deliverable: Postgres HA ready to fail over in < 60s, asserted by an automated test script. Topology — 3 Incus containers per environment: pgaf-monitor pg_auto_failover state machine (single instance) pgaf-primary first registered → primary pgaf-replica second registered → hot-standby (sync rep) Files: infra/ansible/playbooks/postgres_ha.yml Provisions the 3 containers via `incus launch images:ubuntu/22.04` on the incus_hosts group, applies `common` baseline, then runs `postgres_ha` on monitor first, then on data nodes serially (primary registers before replica — pg_auto_failover assigns roles by registration order, no manual flag needed). infra/ansible/roles/postgres_ha/ defaults/main.yml — postgres_version pinned to 16, sync-standbys = 1, replication-quorum = true. App user/dbname for the formation. Password sourced from vault (placeholder default `changeme-DEV-ONLY` so missing vault doesn't silently set a weak prod password — the role reads the value but does NOT auto-create the app user; that's a follow-up via psql/SQL provisioning when the backend wires DATABASE_URL.). tasks/install.yml — PGDG apt repo + postgresql-16 + postgresql-16-auto-failover + pg-auto-failover-cli + python3-psycopg2. Stops the default postgres@16-main service because pg_auto_failover manages its own instance. tasks/monitor.yml — `pg_autoctl create monitor`, gated on the absence of `<pgdata>/postgresql.conf` so re-runs no-op. Renders systemd unit `pg_autoctl.service` and starts it. tasks/node.yml — `pg_autoctl create postgres` joining the monitor URI from defaults. Sets formation sync-standbys policy idempotently from any node. templates/pg_autoctl-{monitor,node}.service.j2 — minimal systemd units, Restart=on-failure, NOFILE=65536. README.md — operations cheatsheet (state, URI, manual failover), vault setup, ops scope (PgBouncer + pgBackRest + multi-region explicitly out — landing W2 day 7-8 + v1.2+). infra/ansible/inventory/lab.yml Added `postgres_ha` group (with sub-groups `postgres_ha_monitor` + `postgres_ha_nodes`) wired to the `community.general.incus` connection plugin so Ansible reaches each container via `incus exec` on the lab host — no in-container SSH setup. infra/ansible/tests/test_pg_failover.sh The acceptance script. Sequence: 0. read formation state via monitor — abort if degraded baseline 1. `incus stop --force pgaf-primary` — start RTO timer 2. poll monitor every 1s for the standby's promotion 3. `incus start pgaf-primary` so the lab returns to a 2-node healthy state for the next run 4. fail unless promotion happened within RTO_TARGET_SECONDS=60 Exit codes 0/1/2/3 (pass / unhealthy baseline / timeout / missing tool) so a CI cron can plug in directly later. Acceptance verified locally: $ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \ --syntax-check playbook: playbooks/postgres_ha.yml ← clean $ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \ --list-tasks 4 plays, 22 tasks across plays, all tagged. $ bash -n infra/ansible/tests/test_pg_failover.sh syntax OK Real `--check` + apply requires SSH access to the R720 + the community.general collection installed (`ansible-galaxy collection install community.general`). Operator runs that step. Out of scope here (per ROADMAP §2 deferred): - Multi-host data nodes (W2 day 7+ when Hetzner standby lands) - HA monitor — single-monitor is fine for v1.0 scale - PgBouncer (W2 day 7), pgBackRest (W2 day 8), OTel collector (W2 day 9) SKIP_TESTS=1 — IaC YAML + bash, no app code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 16:27:46 +00:00
# pg_auto_failover defaults — citusdata's PG HA control plane.
# https://github.com/hapostgres/pg_auto_failover
#
# v1.0.9 Day 6 — RTO target < 60s. Sync replication is the default
# (number_sync_standbys=1) so the primary blocks on standby ack
# before client commit returns. That trades a few ms of latency for
# zero data loss on the primary's death — the right tradeoff for the
# marketplace + subscription tables we're protecting.
---
# PG version pinned to match the Postgres 16 used in dev/CI
# (docker-compose.dev.yml). Bumping requires a migration plan, not a
# var flip.
postgres_version: 16
# pg_auto_failover packages live in PGDG (apt.postgresql.org) under
# the same major-version suffix as the postgres packages.
postgres_apt_key_url: https://www.postgresql.org/media/keys/ACCC4CF8.asc
# Cluster topology — overridden in inventory/group_vars per role
# assignment. Each container in the postgres_ha group sets
# `pg_auto_failover_role` to one of: monitor, node.
pg_auto_failover_role: node
# Monitor — the central state machine. Single instance for now;
# pg_auto_failover supports HA monitor too but adds setup cost we
# don't need at v1.0.9 scale.
pg_auto_failover_monitor_host: pgaf-monitor.lxd
pg_auto_failover_monitor_port: 5432
pg_auto_failover_monitor_dbname: pg_auto_failover
# Data nodes — each a postgres instance pg_auto_failover orchestrates.
# Hostname must be DNS-resolvable from the monitor + peer nodes (Incus
# auto-creates `<container>.lxd` records inside its bridge).
pg_auto_failover_node_port: 5432
pg_auto_failover_data_dir: /var/lib/postgresql/{{ postgres_version }}/main
pg_auto_failover_state_dir: /var/lib/postgresql/{{ postgres_version }}/pgaf
# Sync replication — number of standbys that must ack before commit.
# Set to 1 for v1.0.9 (single replica). Increase if more replicas land.
pg_auto_failover_number_sync_standbys: 1
# Replication-quorum = require ALL formation nodes to vote on
# leadership. With 1 monitor + primary + 1 replica, this is the
# split-brain-safe default. Disable only when the formation has
# >=3 data nodes and you can tolerate 1 unreachable.
pg_auto_failover_replication_quorum: true
# Application database — the backend connects via the pg_auto_failover
# formation URI (libpq connection string with multiple hosts +
# target_session_attrs=read-write). Provisioned by the role on the
# primary, replicates automatically.
pg_auto_failover_app_dbname: veza
pg_auto_failover_app_user: veza
# Password is supplied via vault — see roles/postgres_ha/README.md.
pg_auto_failover_app_password: "{{ vault_pg_app_password | default('changeme-DEV-ONLY') }}"