3 commits

Author	SHA1	Message	Date
senke	d86815561c	feat(infra): MinIO distributed EC:2 + migration script (W3 Day 12) ... Four-node distributed MinIO cluster, single erasure set EC:2, tolerates 2 simultaneous node losses. 50% storage efficiency. Pinned to RELEASE.2025-09-07T16-13-09Z to match docker-compose so dev/prod parity is preserved. - infra/ansible/roles/minio_distributed/ : install pinned binary, systemd unit pointed at MINIO_VOLUMES with bracket-expansion form, EC:2 forced via MINIO_STORAGE_CLASS_STANDARD. Vault assertion blocks shipping placeholder credentials to staging/prod. - bucket init : creates veza-prod-tracks, enables versioning, applies lifecycle.json (30d noncurrent expiry + 7d abort-multipart). Cold-tier transition ready but inert until minio_remote_tier_name is set. - infra/ansible/playbooks/minio_distributed.yml : provisions the 4 containers, applies common baseline + role. - infra/ansible/inventory/lab.yml : new minio_nodes group. - infra/ansible/tests/test_minio_resilience.sh : kill 2 nodes, verify EC:2 reconstruction (read OK + checksum matches), restart, wait for self-heal. - scripts/minio-migrate-from-single.sh : mc mirror --preserve from the single-node bucket to the new cluster, count-verifies, prints rollout next-steps. - config/prometheus/alert_rules.yml : MinIODriveOffline (warn) + MinIONodesUnreachable (page) — page fires at >= 2 nodes unreachable because that's the redundancy ceiling for EC:2. - docs/ENV_VARIABLES.md §12 : MinIO migration cross-ref. Acceptance (Day 12) : EC:2 survives 2 concurrent kills + self-heals. Lab apply pending. No backend code change — interface stays AWS S3. W3 progress : Redis Sentinel ✓ (Day 11), MinIO distribué ✓ (this), CDN ⏳ Day 13, DMCA ⏳ Day 14, embed ⏳ Day 15. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-28 13:46:42 +02:00
senke	bf31a91ae6	feat(infra): pgbackrest role + dr-drill + Prometheus backup alerts (W2 Day 8) ... ROADMAP_V1.0_LAUNCH.md §Semaine 2 day 8 deliverable: - Postgres backups land in MinIO via pgbackrest - dr-drill restores them weekly into an ephemeral Incus container and asserts the data round-trips - Prometheus alerts fire when the drill fails OR when the timer has stopped firing for >8 days Cadence: full — weekly (Sun 02:00 UTC, systemd timer) diff — daily (Mon-Sat 02:00 UTC, systemd timer) WAL — continuous (postgres archive_command, archive_timeout=60s) drill — weekly (Sun 04:00 UTC — runs 2h after the Sun full so the restore exercises fresh data) RPO ≈ 1 min (archive_timeout). RTO ≤ 30 min (drill measures actual restore wall-clock). Files: infra/ansible/roles/pgbackrest/ defaults/main.yml — repo1-* config (MinIO/S3, path-style, aes-256-cbc encryption, vault-backed creds), retention 4 full / 7 diff / 4 archive cycles, zstd@3 compression. The role's first task asserts the placeholder secrets are gone — refuses to apply until the vault carries real keys. tasks/main.yml — install pgbackrest, render /etc/pgbackrest/pgbackrest.conf, set archive_command on the postgres instance via ALTER SYSTEM, detect role at runtime via `pg_autoctl show state --json`, stanza-create from primary only, render + enable systemd timers (full + diff + drill). templates/pgbackrest.conf.j2 — global + per-stanza sections; pg1-path defaults to the pg_auto_failover state dir so the role plugs straight into the Day 6 formation. templates/pgbackrest-{full,diff,drill}.{service,timer}.j2 — systemd units. Backup services run as `postgres`, drill service runs as `root` (needs `incus`). RandomizedDelaySec on every timer to absorb clock skew + node collision risk. README.md — RPO/RTO guarantees, vault setup, repo wiring, operational cheatsheet (info / check / manual backup), restore procedure documented separately as the dr-drill. scripts/dr-drill.sh Acceptance script for the day. Sequence: 0. pre-flight: required tools, latest backup metadata visible 1. launch ephemeral `pg-restore-drill` Incus container 2. install postgres + pgbackrest inside, push the SAME pgbackrest.conf as the host (read-only against the bucket by pgbackrest semantics — the same s3 keys get reused so the drill exercises the production credential path) 3. `pgbackrest restore` — full + WAL replay 4. start postgres, wait for pg_isready 5. smoke query: SELECT count(*) FROM users — must be ≥ MIN_USERS_EXPECTED 6. write veza_backup_drill_* metrics to the textfile-collector 7. teardown (or --keep for postmortem inspection) Exit codes 0/1/2 (pass / drill failure / env problem) so a Prometheus runner can plug in directly. config/prometheus/alert_rules.yml — new `veza_backup` group: - BackupRestoreDrillFailed (critical, 5m): the last drill reported success=0. Pages because a backup we haven't proved restorable is dette technique waiting for a disaster. - BackupRestoreDrillStale (warning, 1h after >8 days): the drill timer has stopped firing. Catches a broken cron / unit / runner before the failure-mode alert above ever sees data. Both annotations include a runbook_url stub (veza.fr/runbooks/...) — those land alongside W2 day 10's SLO runbook batch. infra/ansible/playbooks/postgres_ha.yml Two new plays: 6. apply pgbackrest role to postgres_ha_nodes (install + config + full/diff timers on every data node; pgbackrest's repo lock arbitrates collision) 7. install dr-drill on the incus_hosts group (push /usr/local/bin/dr-drill.sh + render drill timer + ensure /var/lib/node_exporter/textfile_collector exists) Acceptance verified locally: $ ansible-playbook -i inventory/lab.yml playbooks/postgres_ha.yml \ --syntax-check playbook: playbooks/postgres_ha.yml ← clean $ python3 -c "import yaml; yaml.safe_load(open('config/prometheus/alert_rules.yml'))" YAML OK $ bash -n scripts/dr-drill.sh syntax OK Real apply + drill needs the lab R720 + a populated MinIO bucket + the secrets in vault — operator's call. Out of scope (deferred per ROADMAP §2): - Off-site backup replica (B2 / Bunny.net) — v1.1+ - Logical export pipeline for RGPD per-user dumps — separate feature track, not a backup-system concern - PITR admin UI — CLI-only via `--type=time` for v1.0 - pgbackrest_exporter Prometheus integration — W2 day 9 alongside the OTel collector Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-28 00:51:00 +02:00
senke	6b25ccc9da	feat(monitoring): add Prometheus alerting rules for critical conditions ... INF-08: Alert rules for service_down, high_error_rate (>5%), high_latency (P99>2s), and redis_unreachable. Enabled rule_files in prometheus.yml.	2026-02-22 17:36:07 +01:00