veza/infra/ansible
senke bf31a91ae6
Some checks failed
Veza CI / Frontend (Web) (push) Failing after 16m6s
Veza CI / Notify on failure (push) Successful in 11s
E2E Playwright / e2e (full) (push) Successful in 19m59s
Veza CI / Rust (Stream Server) (push) Successful in 4m57s
Security Scan / Secret Scanning (gitleaks) (push) Successful in 49s
Veza CI / Backend (Go) (push) Successful in 6m4s
feat(infra): pgbackrest role + dr-drill + Prometheus backup alerts (W2 Day 8)
ROADMAP_V1.0_LAUNCH.md §Semaine 2 day 8 deliverable:
  - Postgres backups land in MinIO via pgbackrest
  - dr-drill restores them weekly into an ephemeral Incus container
    and asserts the data round-trips
  - Prometheus alerts fire when the drill fails OR when the timer
    has stopped firing for >8 days

Cadence:
  full   — weekly  (Sun 02:00 UTC, systemd timer)
  diff   — daily   (Mon-Sat 02:00 UTC, systemd timer)
  WAL    — continuous (postgres archive_command, archive_timeout=60s)
  drill  — weekly  (Sun 04:00 UTC — runs 2h after the Sun full so
           the restore exercises fresh data)

RPO ≈ 1 min (archive_timeout). RTO ≤ 30 min (drill measures actual
restore wall-clock).

Files:
  infra/ansible/roles/pgbackrest/
    defaults/main.yml — repo1-* config (MinIO/S3, path-style,
      aes-256-cbc encryption, vault-backed creds), retention 4 full
      / 7 diff / 4 archive cycles, zstd@3 compression. The role's
      first task asserts the placeholder secrets are gone — refuses
      to apply until the vault carries real keys.
    tasks/main.yml — install pgbackrest, render
      /etc/pgbackrest/pgbackrest.conf, set archive_command on the
      postgres instance via ALTER SYSTEM, detect role at runtime
      via `pg_autoctl show state --json`, stanza-create from primary
      only, render + enable systemd timers (full + diff + drill).
    templates/pgbackrest.conf.j2 — global + per-stanza sections;
      pg1-path defaults to the pg_auto_failover state dir so the
      role plugs straight into the Day 6 formation.
    templates/pgbackrest-{full,diff,drill}.{service,timer}.j2 —
      systemd units. Backup services run as `postgres`,
      drill service runs as `root` (needs `incus`).
      RandomizedDelaySec on every timer to absorb clock skew + node
      collision risk.
    README.md — RPO/RTO guarantees, vault setup, repo wiring,
      operational cheatsheet (info / check / manual backup),
      restore procedure documented separately as the dr-drill.

  scripts/dr-drill.sh
    Acceptance script for the day. Sequence:
      0. pre-flight: required tools, latest backup metadata visible
      1. launch ephemeral `pg-restore-drill` Incus container
      2. install postgres + pgbackrest inside, push the SAME
         pgbackrest.conf as the host (read-only against the bucket
         by pgbackrest semantics — the same s3 keys get reused so
         the drill exercises the production credential path)
      3. `pgbackrest restore` — full + WAL replay
      4. start postgres, wait for pg_isready
      5. smoke query: SELECT count(*) FROM users — must be ≥ MIN_USERS_EXPECTED
      6. write veza_backup_drill_* metrics to the textfile-collector
      7. teardown (or --keep for postmortem inspection)
    Exit codes 0/1/2 (pass / drill failure / env problem) so a
    Prometheus runner can plug in directly.

  config/prometheus/alert_rules.yml — new `veza_backup` group:
    - BackupRestoreDrillFailed (critical, 5m): the last drill
      reported success=0. Pages because a backup we haven't proved
      restorable is dette technique waiting for a disaster.
    - BackupRestoreDrillStale (warning, 1h after >8 days): the
      drill timer has stopped firing. Catches a broken cron / unit
      / runner before the failure-mode alert above ever sees data.
    Both annotations include a runbook_url stub
    (veza.fr/runbooks/...) — those land alongside W2 day 10's
    SLO runbook batch.

  infra/ansible/playbooks/postgres_ha.yml
    Two new plays:
      6. apply pgbackrest role to postgres_ha_nodes (install +
         config + full/diff timers on every data node;
         pgbackrest's repo lock arbitrates collision)
      7. install dr-drill on the incus_hosts group (push
         /usr/local/bin/dr-drill.sh + render drill timer + ensure
         /var/lib/node_exporter/textfile_collector exists)

Acceptance verified locally:
  $ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \
      --syntax-check
  playbook: playbooks/postgres_ha.yml          ← clean
  $ python3 -c "import yaml; yaml.safe_load(open('config/prometheus/alert_rules.yml'))"
  YAML OK
  $ bash -n scripts/dr-drill.sh
  syntax OK

Real apply + drill needs the lab R720 + a populated MinIO bucket
+ the secrets in vault — operator's call.

Out of scope (deferred per ROADMAP §2):
  - Off-site backup replica (B2 / Bunny.net) — v1.1+
  - Logical export pipeline for RGPD per-user dumps — separate
    feature track, not a backup-system concern
  - PITR admin UI — CLI-only via `--type=time` for v1.0
  - pgbackrest_exporter Prometheus integration — W2 day 9
    alongside the OTel collector

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 00:51:00 +02:00
..
group_vars feat(infra): Ansible IaC scaffolding — common + incus_host roles (Day 5 v1.0.9) 2026-04-27 18:16:38 +02:00
inventory feat(infra): pgbouncer role + pgbench load test (W2 Day 7) 2026-04-27 18:35:05 +02:00
playbooks feat(infra): pgbackrest role + dr-drill + Prometheus backup alerts (W2 Day 8) 2026-04-28 00:51:00 +02:00
roles feat(infra): pgbackrest role + dr-drill + Prometheus backup alerts (W2 Day 8) 2026-04-28 00:51:00 +02:00
tests feat(infra): pgbouncer role + pgbench load test (W2 Day 7) 2026-04-27 18:35:05 +02:00
ansible.cfg feat(infra): Ansible IaC scaffolding — common + incus_host roles (Day 5 v1.0.9) 2026-04-27 18:16:38 +02:00
README.md feat(infra): Ansible IaC scaffolding — common + incus_host roles (Day 5 v1.0.9) 2026-04-27 18:16:38 +02:00

Veza Ansible IaC

Infrastructure-as-code for the Veza self-hosted platform. Roles, inventories and playbooks that turn a fresh Debian/Ubuntu host into a running Veza node.

Scope at v1.0.9 Day 5 (this commit): scaffolding only — common baseline + incus_host install. Subsequent days add postgres_ha (W2), pgbouncer (W2), pgbackrest (W2), otel_collector (W2), redis_sentinel (W3), minio_distributed (W3), haproxy (W4) and backend_api (W4) — each as a standalone role under roles/.

Layout

infra/ansible/
├── ansible.cfg                 # pinned defaults (inventory path, ControlMaster)
├── inventory/
│   ├── lab.yml                 # R720 lab Incus container — dry-run target
│   ├── staging.yml             # Hetzner staging (TODO IP — W2 provision)
│   └── prod.yml                # R720 prod (TODO IP — DNS at EX-5)
├── group_vars/
│   └── all.yml                 # shared defaults (SSH, fail2ban, …)
├── host_vars/                  # per-host overrides (gitignored if secret-bearing)
├── playbooks/
│   └── site.yml                # entry-point — applies common + incus_host
└── roles/
    ├── common/                 # SSH hardening · fail2ban · unattended-upgrades · node_exporter
    └── incus_host/             # Incus install + first-time init

Quickstart

Lab dry-run (syntax + dry-execute, no remote changes)

cd infra/ansible
ansible-playbook -i inventory/lab.yml playbooks/site.yml --check

--check is the acceptance gate for v1.0.9 Day 5 — must pass clean before merging any role change.

Lab apply

ansible-playbook -i inventory/lab.yml playbooks/site.yml

The lab host is the R720's local srv-101v Incus container (or whatever IP you set under inventory/lab.yml::veza-lab.ansible_host). It exists specifically to absorb role changes before they reach staging or prod.

Staging / prod

Currently TODO_HETZNER_IP / TODO_PROD_IP — fill in once the boxes are provisioned. Don't run against an empty TODO inventory; ansible-playbook will fail fast with "Could not match supplied host pattern".

Tags — apply a single concern

# Re-render only the SSH hardening drop-in
ansible-playbook -i inventory/lab.yml playbooks/site.yml --tags ssh

# Bump node_exporter to a newer pinned version (after editing group_vars/all.yml)
ansible-playbook -i inventory/lab.yml playbooks/site.yml --tags node_exporter

Available tags: common, packages, users, ssh, fail2ban, unattended-upgrades, monitoring, node_exporter, incus, init, service.

Roles

common — host baseline

  • ssh.yml — drops /etc/ssh/sshd_config.d/50-veza-hardening.conf from a Jinja template. Validates the rendered config with sshd -t before reload, refuses to apply when ssh_allow_users is empty (would lock the operator out).
  • fail2ban.yml/etc/fail2ban/jail.local with the sshd jail enabled, defaults to bantime=1h / findtime=10min / maxretry=5.
  • unattended_upgrades.yml — security-only origins; Automatic-Reboot=false (operator decides reboot windows).
  • node_exporter.yml — installs Prometheus node_exporter pinned to the version in group_vars/all.yml::monitoring_node_exporter_version, runs as a systemd unit on :9100.

Variables in group_vars/all.yml:

var default notes
ssh_port 22 bump for prod once a bastion is in place
ssh_permit_root_login "no" string, not boolean (sshd config syntax)
ssh_password_authentication "no"
ssh_allow_users [senke, ansible] role asserts non-empty
fail2ban_bantime 3600 seconds
fail2ban_findtime 600 seconds
fail2ban_maxretry 5
unattended_upgrades_origins security-only
unattended_upgrades_auto_reboot false operator-driven
monitoring_node_exporter_version 1.8.2 upstream pin
monitoring_node_exporter_port 9100

incus_host — Incus server install

  • Adds the upstream zabbly Incus apt repo.
  • Installs incus + incus-client.
  • Adds the ansible user to incus-admin so subsequent roles can run incus non-sudo.
  • First-time incus admin init via preseed if the host has never been initialised. Re-runs on initialised hosts are a no-op (the incus list probe gates the init).

Bridge config:

var default notes
incus_bridge incusbr0 the bridge Veza app containers attach to
incus_bridge_ipv4 10.99.0.1/24 NAT'd via Incus by default

Conventions

  • Roles are idempotent — running site.yml twice produces no changes. CI eventually validates this with a --check after a real apply.
  • No secrets in git. host_vars/<host>.yml is fine for non-secrets; secrets go in host_vars/<host>.vault.yml encrypted with ansible-vault. The vault key lives outside the repo.
  • Tags are mandatory on every task so a partial apply (--tags ssh,monitoring) is always possible. A new role missing tags fails its own commit's --check review.
  • Comment the why, not the what. Role tasks should answer "why this knob, why this default, why this guard" — the task name + module already say what.

See also

  • ROADMAP_V1.0_LAUNCH.md §Semaine 1 day 5 — original scope brief
  • docs/runbooks/ — once roles for production services land, each gets a runbook
  • docker-compose.dev.yml — the dev-host equivalent of these roles (kept for now; Ansible takes over for staging/prod once W2 lands)