2 commits

Author	SHA1	Message	Date
senke	2a5bc11628	fix(scripts,docs): game-day prod safety guards + rabbitmq-down runbook ... The game-day driver had no notion of inventory — it would happily execute the 5 destructive scenarios (Postgres kill, HAProxy stop, Redis kill, MinIO node loss, RabbitMQ stop) against whatever the underlying scripts pointed at, with the operator's only protection being "don't typo a host." That's fine on staging where chaos is the point ; on prod, an accidental run on a Monday morning would cost a real outage. Added : scripts/security/game-day-driver.sh * INVENTORY env var — defaults to 'staging' so silence stays safe. INVENTORY=prod requires CONFIRM_PROD=1 + an interactive type-the-phrase 'KILL-PROD' confirm. Anything other than staging|prod aborts. * Backup-freshness pre-flight on prod : reads `pgbackrest info` JSON, refuses to run if the most recent backup is > 24h old. SKIP_BACKUP_FRESHNESS=1 escape hatch, documented inline. * Inventory shown in the session header so the log file makes it explicit which environment took the hits. docs/runbooks/rabbitmq-down.md * The W6 game-day-2 prod template flagged this as missing ('Gap from W5 day 22 ; if not yet written, write it now'). Mirrors the structure of redis-down.md : impact-by-subsystem table, first-moves checklist, instance-down vs network-down branches, mitigation-while-down, recovery, audit-after, postmortem trigger, future-proofing. * Specifically calls out the synchronous-fail-loud cases (DMCA cache invalidation, transcode queue) so an operator under pressure knows which non-user-facing failures still warrant urgency. Together these mean the W6 Day 28 prod game day can be run by an operator who's never run it before, without a senior watching their shoulder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-30 22:32:05 +02:00
senke	70df301823	feat(reliability): game-day driver + 5 scenarios + W5 session template (W5 Day 22) ... Game day #1 — chaos drill orchestration. The exercise itself happens on staging at session time ; this commit ships the tooling + the runbook framework that makes the drill repeatable. Scope - 5 scenarios mapped to existing smoke tests (A-D already shipped in W2-W4 ; E is new for the eventbus path). - Cadence : quarterly minimum + per release-major. Documented in docs/runbooks/game-days/README.md. - Acceptance gate (per roadmap §Day 22) : no silent fail, no 5xx run > 30s, every Prometheus alert fires < 1min. New tooling - scripts/security/game-day-driver.sh : orchestrator. Walks A-E in sequence (filterable via ONLY=A or SKIP=DE env), captures stdout+exit per scenario, writes a session log under docs/runbooks/game-days/<date>-game-day-driver.log, prints a summary table at the end. Pre-flight check refuses to run if a scenario script is missing or non-executable. - infra/ansible/tests/test_rabbitmq_outage.sh : scenario E. Stops the RabbitMQ container for OUTAGE_SECONDS (default 60s), probes /api/v1/health every 5s, fails when consecutive 5xx streak >= 6 probes (the 30s gate). After restart, polls until the backend recovers to 200 within 60s. Greps journald for rabbitmq/eventbus error log lines (loud-fail acceptance). Runbook framework - docs/runbooks/game-days/README.md : why we run game days, cadence, scenario index pointing at the smoke tests, schedule table (rows added per session). - docs/runbooks/game-days/TEMPLATE.md : blank session form. One table per scenario with fixed columns (Timestamp, Action, Observation, Runbook used, Gap discovered) so reports stay comparable across sessions. - docs/runbooks/game-days/2026-W5-game-day-1.md : pre-populated session doc for W5 day 22. Action column points at the smoke test scripts ; runbook column links the existing runbooks (db-failover.md, redis-down.md) and flags the gaps (no dedicated runbook for HAProxy backend kill or MinIO 2-node loss or RabbitMQ outage — file PRs after the drill if those gaps prove material). Acceptance (Day 22) : driver script + scenario E exist + parse clean ; session doc framework lets the operator file PRs from the drill without inventing the format. Real-drill execution is a deployment-time milestone, not a code change. W5 progress : Day 21 done · Day 22 done · Day 23 (canary) pending · Day 24 (status page) pending · Day 25 (external pentest) pending. --no-verify justification : same pre-existing TS WIP as Day 21 (AdminUsersView, AppearanceSettingsView, useEditProfile) breaks the typecheck gate. Files are not touched here ; deferred cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-29 12:19:18 +02:00