66 lines
3 KiB
Markdown
66 lines
3 KiB
Markdown
|
# `postgres_ha` role — pg_auto_failover formation
|
||
|
|
|
||
|
|
Brings up a Postgres HA formation managed by [pg_auto_failover](https://github.com/hapostgres/pg_auto_failover) (citusdata). Three Incus containers per environment:
|
||
|
|
|
||
|
|
| container | role | purpose |
|
||
|
|
| --------------- | -------- | ------------------------------------------------ |
|
||
|
|
| `pgaf-monitor` | monitor | central state machine — primary election, health |
|
||
|
|
| `pgaf-primary` | node | first registered → becomes primary |
|
||
|
|
| `pgaf-replica` | node | second registered → becomes hot-standby (sync) |
|
||
|
|
|
||
|
|
v1.0.9 Day 6 ships the role in the lab inventory only. Staging/prod adopt it once Hetzner standby is provisioned (W2 day 7+).
|
||
|
|
|
||
|
|
## Acceptance test
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# After `ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml`,
|
||
|
|
# the failover RTO is asserted by the script:
|
||
|
|
bash infra/ansible/tests/test_pg_failover.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
Target: stop primary container → standby promoted within 60s. Script re-starts the killed container so the lab returns to a healthy 2-node formation for subsequent runs.
|
||
|
|
|
||
|
|
## Vault for secrets
|
||
|
|
|
||
|
|
The application user's password lives outside git. Create `infra/ansible/group_vars/postgres_ha.vault.yml`:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
vault_pg_app_password: "<random-32-char-secret>"
|
||
|
|
```
|
||
|
|
|
||
|
|
Encrypt:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ansible-vault encrypt infra/ansible/group_vars/postgres_ha.vault.yml
|
||
|
|
```
|
||
|
|
|
||
|
|
The vault key (`~/.ansible/vault_pass`) is operator-local — never committed. The role default `pg_auto_failover_app_password` is a `changeme-DEV-ONLY` placeholder so a missing vault doesn't silently set a real-world weak password.
|
||
|
|
|
||
|
|
## Sync replication policy
|
||
|
|
|
||
|
|
`number_sync_standbys = 1` is the v1.0.9 default — the primary blocks on the standby's WAL ack before client commit returns. Trade: a few ms of write latency for zero data loss on primary death. The monitor enforces this on the formation; bumping it requires more replicas (3+) and a config push.
|
||
|
|
|
||
|
|
## What the role does NOT do (yet)
|
||
|
|
|
||
|
|
- **No PgBouncer** — that's W2 day 7. Backend connects directly to the formation URI for now.
|
||
|
|
- **No backup** — pgBackRest lands W2 day 8. Failover ≠ disaster recovery.
|
||
|
|
- **No multi-region failover** — single region at v1.0; multi-region is v1.2+ per ROADMAP_V1.0_LAUNCH.md §2 OUT.
|
||
|
|
|
||
|
|
## Operations
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# State on the monitor:
|
||
|
|
incus exec pgaf-monitor -- sudo -u postgres \
|
||
|
|
pg_autoctl show state --pgdata /var/lib/postgresql/16/pgaf/monitor
|
||
|
|
|
||
|
|
# Connection URI (libpq multi-host with target_session_attrs=read-write):
|
||
|
|
incus exec pgaf-monitor -- sudo -u postgres \
|
||
|
|
pg_autoctl show uri --pgdata /var/lib/postgresql/16/pgaf/monitor --formation default
|
||
|
|
|
||
|
|
# Manual failover (if needed for a maintenance window):
|
||
|
|
incus exec pgaf-monitor -- sudo -u postgres \
|
||
|
|
pg_autoctl perform failover --pgdata /var/lib/postgresql/16/pgaf/monitor
|
||
|
|
```
|
||
|
|
|
||
|
|
Backend application reads the formation URI from `DATABASE_URL`; the libpq driver handles primary discovery via `target_session_attrs=read-write`. No app-level reconfiguration during a failover.
|