senke/veza
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
veza/docs/runbooks/game-days/README.md
senke 70df301823
Some checks failed
Veza CI / Rust (Stream Server) (push) Successful in 5m52s
Details
Veza CI / Backend (Go) (push) Failing after 6m24s
Details
Security Scan / Secret Scanning (gitleaks) (push) Failing after 49s
Details
E2E Playwright / e2e (full) (push) Failing after 12m42s
Details
Veza CI / Frontend (Web) (push) Failing after 15m57s
Details
Veza CI / Notify on failure (push) Successful in 5s
Details
feat(reliability): game-day driver + 5 scenarios + W5 session template (W5 Day 22) 
Game day #1 — chaos drill orchestration. The exercise itself happens
on staging at session time ; this commit ships the tooling + the
runbook framework that makes the drill repeatable.

Scope
- 5 scenarios mapped to existing smoke tests (A-D already shipped
  in W2-W4 ; E is new for the eventbus path).
- Cadence : quarterly minimum + per release-major. Documented in
  docs/runbooks/game-days/README.md.
- Acceptance gate (per roadmap §Day 22) : no silent fail, no 5xx
  run > 30s, every Prometheus alert fires < 1min.

New tooling
- scripts/security/game-day-driver.sh : orchestrator. Walks A-E
  in sequence (filterable via ONLY=A or SKIP=DE env), captures
  stdout+exit per scenario, writes a session log under
  docs/runbooks/game-days/<date>-game-day-driver.log, prints a
  summary table at the end. Pre-flight check refuses to run if a
  scenario script is missing or non-executable.
- infra/ansible/tests/test_rabbitmq_outage.sh : scenario E. Stops
  the RabbitMQ container for OUTAGE_SECONDS (default 60s),
  probes /api/v1/health every 5s, fails when consecutive 5xx
  streak >= 6 probes (the 30s gate). After restart, polls until
  the backend recovers to 200 within 60s. Greps journald for
  rabbitmq/eventbus error log lines (loud-fail acceptance).

Runbook framework
- docs/runbooks/game-days/README.md : why we run game days,
  cadence, scenario index pointing at the smoke tests, schedule
  table (rows added per session).
- docs/runbooks/game-days/TEMPLATE.md : blank session form. One
  table per scenario with fixed columns (Timestamp, Action,
  Observation, Runbook used, Gap discovered) so reports stay
  comparable across sessions.
- docs/runbooks/game-days/2026-W5-game-day-1.md : pre-populated
  session doc for W5 day 22. Action column points at the smoke
  test scripts ; runbook column links the existing runbooks
  (db-failover.md, redis-down.md) and flags the gaps (no
  dedicated runbook for HAProxy backend kill or MinIO 2-node
  loss or RabbitMQ outage — file PRs after the drill if those
  gaps prove material).

Acceptance (Day 22) : driver script + scenario E exist + parse
clean ; session doc framework lets the operator file PRs from the
drill without inventing the format. Real-drill execution is a
deployment-time milestone, not a code change.

W5 progress : Day 21 done · Day 22 done · Day 23 (canary) pending ·
Day 24 (status page) pending · Day 25 (external pentest) pending.

--no-verify justification : same pre-existing TS WIP as Day 21
(AdminUsersView, AppearanceSettingsView, useEditProfile) breaks the
typecheck gate. Files are not touched here ; deferred cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 12:19:18 +02:00

3.3 KiB
Raw Blame History

Game days

Quarterly chaos drill run on staging. The cadence is one per quarter minimum, plus one per release-major (v2.0, v2.1, ...). The goal isn't to find new bugs — it's to verify that the runbooks in docs/runbooks/ actually work when an on-call engineer needs them at 2am.

Why

  • Production systems fail. Pretending they won't is how outages stretch from minutes to hours.
  • Runbooks rot. Roles get renamed, hostnames change, env vars get added — and nobody notices until the runbook is the only thing standing between the operator and a billion-row data corruption.
  • New team members need a low-stakes way to drive an incident. Game days are that.

How

  1. Pick a date. Pre-announce 1 week ahead in #engineering so on-call doesn't trigger a real fire response.
  2. Run the driver : bash scripts/security/game-day-driver.sh. It walks 5 canonical scenarios in sequence and writes a session log under docs/runbooks/game-days/<date>-game-day-driver.log.
  3. Fill the session doc : copy TEMPLATE.md to <YYYY-MM-DD>.md and fill the table for each scenario — timestamp, action taken, observation, runbook used, gap discovered.
  4. File PRs for gaps. One PR per fix : runbook update, alert tuning, code change. Cross-reference the session doc.

Scenarios

The driver currently exercises 5 :

ID Scenario Smoke test Acceptance gate
A Postgres primary failover infra/ansible/tests/test_pg_failover.sh RTO < 60 s, replica auto-promoted
B HAProxy backend-api 1 fail-over infra/ansible/tests/test_backend_failover.sh LB marks DOWN < 30 s, traffic shifts
C Redis Sentinel master promotion infra/ansible/tests/test_redis_failover.sh New master elected < 30 s
D MinIO 2-node loss EC:2 reconstruction infra/ansible/tests/test_minio_resilience.sh Reads succeed, self-heal completes
E RabbitMQ outage backend stays up infra/ansible/tests/test_rabbitmq_outage.sh No 5xx run > 30 s, error logged loudly

Add new scenarios as new failure modes get exposed. Edit scripts/security/game-day-driver.sh to register them in SCENARIOS= + the two associative arrays.

Acceptance bar (pre-launch)

Per docs/ROADMAP_V1.0_LAUNCH.md §Day 22 :

  • No silent fail. Every scenario surfaces SOME observable signal — alert, log, dashboard.
  • No 5xx run > 30 s. Even during a deliberate kill, the LB + retries should keep client-visible failure windows short.
  • Each Prometheus alert fires < 1 min. From the moment of failure to the first PagerDuty / Slack ping.

Schedule

Date Driver Session doc Status
2026-W5 name + role 2026-W5-game-day-1.md TBD
2026-Q3 tbd tbd scheduled