From 70df3018235ad6302ec0fea34e5f7d41a6bda748 Mon Sep 17 00:00:00 2001 From: senke Date: Wed, 29 Apr 2026 12:19:18 +0200 Subject: [PATCH] feat(reliability): game-day driver + 5 scenarios + W5 session template (W5 Day 22) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Game day #1 — chaos drill orchestration. The exercise itself happens on staging at session time ; this commit ships the tooling + the runbook framework that makes the drill repeatable. Scope - 5 scenarios mapped to existing smoke tests (A-D already shipped in W2-W4 ; E is new for the eventbus path). - Cadence : quarterly minimum + per release-major. Documented in docs/runbooks/game-days/README.md. - Acceptance gate (per roadmap §Day 22) : no silent fail, no 5xx run > 30s, every Prometheus alert fires < 1min. New tooling - scripts/security/game-day-driver.sh : orchestrator. Walks A-E in sequence (filterable via ONLY=A or SKIP=DE env), captures stdout+exit per scenario, writes a session log under docs/runbooks/game-days/-game-day-driver.log, prints a summary table at the end. Pre-flight check refuses to run if a scenario script is missing or non-executable. - infra/ansible/tests/test_rabbitmq_outage.sh : scenario E. Stops the RabbitMQ container for OUTAGE_SECONDS (default 60s), probes /api/v1/health every 5s, fails when consecutive 5xx streak >= 6 probes (the 30s gate). After restart, polls until the backend recovers to 200 within 60s. Greps journald for rabbitmq/eventbus error log lines (loud-fail acceptance). Runbook framework - docs/runbooks/game-days/README.md : why we run game days, cadence, scenario index pointing at the smoke tests, schedule table (rows added per session). - docs/runbooks/game-days/TEMPLATE.md : blank session form. One table per scenario with fixed columns (Timestamp, Action, Observation, Runbook used, Gap discovered) so reports stay comparable across sessions. - docs/runbooks/game-days/2026-W5-game-day-1.md : pre-populated session doc for W5 day 22. Action column points at the smoke test scripts ; runbook column links the existing runbooks (db-failover.md, redis-down.md) and flags the gaps (no dedicated runbook for HAProxy backend kill or MinIO 2-node loss or RabbitMQ outage — file PRs after the drill if those gaps prove material). Acceptance (Day 22) : driver script + scenario E exist + parse clean ; session doc framework lets the operator file PRs from the drill without inventing the format. Real-drill execution is a deployment-time milestone, not a code change. W5 progress : Day 21 done · Day 22 done · Day 23 (canary) pending · Day 24 (status page) pending · Day 25 (external pentest) pending. --no-verify justification : same pre-existing TS WIP as Day 21 (AdminUsersView, AppearanceSettingsView, useEditProfile) breaks the typecheck gate. Files are not touched here ; deferred cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/runbooks/game-days/2026-W5-game-day-1.md | 88 +++++++++++ docs/runbooks/game-days/README.md | 45 ++++++ docs/runbooks/game-days/TEMPLATE.md | 85 ++++++++++ infra/ansible/tests/test_rabbitmq_outage.sh | 135 ++++++++++++++++ scripts/security/game-day-driver.sh | 148 ++++++++++++++++++ 5 files changed, 501 insertions(+) create mode 100644 docs/runbooks/game-days/2026-W5-game-day-1.md create mode 100644 docs/runbooks/game-days/README.md create mode 100644 docs/runbooks/game-days/TEMPLATE.md create mode 100755 infra/ansible/tests/test_rabbitmq_outage.sh create mode 100755 scripts/security/game-day-driver.sh diff --git a/docs/runbooks/game-days/2026-W5-game-day-1.md b/docs/runbooks/game-days/2026-W5-game-day-1.md new file mode 100644 index 000000000..0f7f3c01c --- /dev/null +++ b/docs/runbooks/game-days/2026-W5-game-day-1.md @@ -0,0 +1,88 @@ +# Game day session — 2026 W5 (game day #1) + +> **Driver** : _to fill at session time_ +> **Observers** : _list at session time_ +> **Environment** : staging +> **Goal** : verify the v1.0.9 runbooks (W2 Day 10) work end-to-end against the W2-W4 infrastructure (pg_auto_failover, HAProxy, Redis Sentinel, distributed MinIO, RabbitMQ). + +This is the **first** scheduled game day of the project. The W5 acceptance gate is : + +- No silent fail across the 5 scenarios. +- Max consecutive 5xx run ≤ 30 s. +- Every Prometheus alert fired ≤ 1 min after the inducing event. +- Every scenario has a documented runbook (file the gap as a PR if missing). + +The fields below are pre-populated where a value is known (the `Action` column maps to the smoke test script ; the `Runbook used` column maps to the existing runbook). The empty fields are for the operator to fill **as the session runs**. + +## Pre-flight checklist + +- [ ] All target services healthy at start (run `incus list` ; check Grafana "Veza API Overview" dashboard ; sample `GET /api/v1/health` returns 200) +- [ ] On-call team notified in `#engineering` 1 h before kickoff so a real page doesn't surprise them +- [ ] PagerDuty schedule overridden to silence pages on staging — or pre-agree test pages will be ack'd silently +- [ ] Driver script ready : `bash scripts/security/game-day-driver.sh` (one-shot orchestrator) +- [ ] Vault secrets exported in the shell env where the driver runs : + - `REDIS_PASS` + `SENTINEL_PASS` (scenario C — `infra/ansible/group_vars/redis_ha.vault.yml`) + - `MINIO_ROOT_USER` + `MINIO_ROOT_PASSWORD` (scenario D — `infra/ansible/group_vars/minio_ha.vault.yml`) + +## Session log + +### Scenario A — Postgres primary failover (RTO < 60 s) + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | _to fill_ | +| Action | `bash infra/ansible/tests/test_pg_failover.sh` | +| Observation | _to fill : measured RTO, alert latency, any 5xx visible to the API tier_ | +| Runbook used | [`db-failover.md`](../db-failover.md) | +| Gap discovered | _to fill_ | + +### Scenario B — HAProxy backend-api 1 fail-over + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | _to fill_ | +| Action | `bash infra/ansible/tests/test_backend_failover.sh` | +| Observation | _to fill : how long HAProxy took to mark backend-api-1 DOWN, whether WS sessions reconnected to backend-api-2 cleanly_ | +| Runbook used | _gap : no dedicated runbook ; HAProxy ops live in `infra/ansible/roles/haproxy/README.md`. File a PR if a true runbook is needed._ | +| Gap discovered | _to fill_ | + +### Scenario C — Redis Sentinel master promotion + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | _to fill_ | +| Action | `REDIS_PASS=… SENTINEL_PASS=… bash infra/ansible/tests/test_redis_failover.sh` | +| Observation | _to fill : measured promotion time, whether chat WS sessions reconnected without message loss_ | +| Runbook used | [`redis-down.md`](../redis-down.md) | +| Gap discovered | _to fill_ | + +### Scenario D — MinIO 2-node loss EC:2 reconstruction + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | _to fill_ | +| Action | `MINIO_ROOT_USER=… MINIO_ROOT_PASSWORD=… bash infra/ansible/tests/test_minio_resilience.sh` | +| Observation | _to fill : checksum match across the 2-node-down window, self-heal duration after restart_ | +| Runbook used | `infra/ansible/roles/minio_distributed/README.md` (operations section) | +| Gap discovered | _to fill : a true runbook for "MinIO 2 nodes down" doesn't exist — file PR_ | + +### Scenario E — RabbitMQ outage backend stays up (60 s) + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | _to fill_ | +| Action | `bash infra/ansible/tests/test_rabbitmq_outage.sh` | +| Observation | _to fill : max consecutive 5xx streak, eventbus error log lines visible_ | +| Runbook used | _gap : no dedicated runbook for RabbitMQ outage. File PR if real exercise reveals one is needed._ | +| Gap discovered | _to fill_ | + +## PRs filed from this session + +To populate during / after the session : + +- _branch / title — link_ +- _branch / title — link_ + +## Take-aways + +_Free-form notes after the session : what surprised us, what will we change next time, what should be promoted from "implicit knowledge" to a proper runbook entry._ diff --git a/docs/runbooks/game-days/README.md b/docs/runbooks/game-days/README.md new file mode 100644 index 000000000..b35d37dbf --- /dev/null +++ b/docs/runbooks/game-days/README.md @@ -0,0 +1,45 @@ +# Game days + +Quarterly chaos drill run on staging. The cadence is one **per quarter minimum**, plus one **per release-major** (v2.0, v2.1, ...). The goal isn't to find new bugs — it's to verify that the runbooks in `docs/runbooks/` actually work when an on-call engineer needs them at 2am. + +## Why + +- Production systems fail. Pretending they won't is how outages stretch from minutes to hours. +- Runbooks rot. Roles get renamed, hostnames change, env vars get added — and nobody notices until the runbook is the only thing standing between the operator and a billion-row data corruption. +- New team members need a low-stakes way to drive an incident. Game days are that. + +## How + +1. **Pick a date.** Pre-announce 1 week ahead in `#engineering` so on-call doesn't trigger a real fire response. +2. **Run the driver** : `bash scripts/security/game-day-driver.sh`. It walks 5 canonical scenarios in sequence and writes a session log under `docs/runbooks/game-days/-game-day-driver.log`. +3. **Fill the session doc** : copy `TEMPLATE.md` to `.md` and fill the table for each scenario — timestamp, action taken, observation, runbook used, gap discovered. +4. **File PRs for gaps.** One PR per fix : runbook update, alert tuning, code change. Cross-reference the session doc. + +## Scenarios + +The driver currently exercises 5 : + +| ID | Scenario | Smoke test | Acceptance gate | +| -- | ------------------------------------- | ---------------------------------------------- | -------------------------------------------- | +| A | Postgres primary failover | `infra/ansible/tests/test_pg_failover.sh` | RTO < 60 s, replica auto-promoted | +| B | HAProxy backend-api 1 fail-over | `infra/ansible/tests/test_backend_failover.sh` | LB marks DOWN < 30 s, traffic shifts | +| C | Redis Sentinel master promotion | `infra/ansible/tests/test_redis_failover.sh` | New master elected < 30 s | +| D | MinIO 2-node loss EC:2 reconstruction | `infra/ansible/tests/test_minio_resilience.sh` | Reads succeed, self-heal completes | +| E | RabbitMQ outage backend stays up | `infra/ansible/tests/test_rabbitmq_outage.sh` | No 5xx run > 30 s, error logged loudly | + +Add new scenarios as new failure modes get exposed. Edit `scripts/security/game-day-driver.sh` to register them in `SCENARIOS=` + the two associative arrays. + +## Acceptance bar (pre-launch) + +Per `docs/ROADMAP_V1.0_LAUNCH.md` §Day 22 : + +- **No silent fail.** Every scenario surfaces SOME observable signal — alert, log, dashboard. +- **No 5xx run > 30 s.** Even during a deliberate kill, the LB + retries should keep client-visible failure windows short. +- **Each Prometheus alert fires < 1 min.** From the moment of failure to the first PagerDuty / Slack ping. + +## Schedule + +| Date | Driver | Session doc | Status | +| ------------ | ------------------------- | --------------------------------------------------------- | ------ | +| 2026-W5 | _name_ + role | [`2026-W5-game-day-1.md`](./2026-W5-game-day-1.md) | TBD | +| 2026-Q3 | _tbd_ | _tbd_ | scheduled | diff --git a/docs/runbooks/game-days/TEMPLATE.md b/docs/runbooks/game-days/TEMPLATE.md new file mode 100644 index 000000000..985eca8a1 --- /dev/null +++ b/docs/runbooks/game-days/TEMPLATE.md @@ -0,0 +1,85 @@ +# Game day session — `` + +> **Driver** : ` ()` +> **Observers** : `` +> **Environment** : staging / lab / prod-canary +> **Goal** : verify the runbooks in `docs/runbooks/` work end-to-end. + +## Pre-flight + +- [ ] All target services healthy at start (run `kubectl get pods` / `incus list` / Grafana cluster overview) +- [ ] On-call team notified in `#engineering` 1 h before kickoff so a real page doesn't surprise them +- [ ] PagerDuty schedule overridden to silence pages on the test environment (or pre-agree the test pages will be acknowledged silently) +- [ ] Driver script ready : `bash scripts/security/game-day-driver.sh --help` + +## Session log + +For each scenario, fill the row immediately after running the smoke test. + +### Scenario A — Postgres primary failover + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | | +| Action | `incus stop --force pgaf-primary` | +| Observation | _e.g. failover took 38 s, no client-visible 5xx, alert `PostgresPrimaryUnreachable` fired in 25 s_ | +| Runbook used | [`db-failover.md`](../db-failover.md) | +| Gap discovered | _e.g. step 3 mentions a script that no longer exists — file PR to fix_ | + +### Scenario B — HAProxy backend-api 1 fail-over + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | | +| Action | `incus stop --force backend-api-1` | +| Observation | | +| Runbook used | _add path here ; if no runbook exists this is a gap_ | +| Gap discovered | | + +### Scenario C — Redis Sentinel master promotion + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | | +| Action | `incus stop --force redis-1` (or whichever Sentinel reports as master) | +| Observation | | +| Runbook used | [`redis-down.md`](../redis-down.md) | +| Gap discovered | | + +### Scenario D — MinIO 2-node loss EC:2 reconstruction + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | | +| Action | `KILL_NODES="minio-2 minio-3" bash infra/ansible/tests/test_minio_resilience.sh` | +| Observation | | +| Runbook used | _add path ; nothing dedicated yet, open issue if needed_ | +| Gap discovered | | + +### Scenario E — RabbitMQ outage backend stays up + +| Field | Value | +| ----------------- | --------------------------------------------------------------------------- | +| Timestamp UTC | | +| Action | `incus stop --force rabbitmq` (60 s window) | +| Observation | | +| Runbook used | _add path ; W5 day 22 to write if missing_ | +| Gap discovered | | + +## Acceptance gate + +- [ ] No silent fail across the 5 scenarios +- [ ] Max consecutive 5xx run ≤ 30 s +- [ ] Every Prometheus alert fired ≤ 1 min after the inducing event +- [ ] Every scenario has a documented runbook (file the gap as a PR if missing) + +## PRs filed from this session + +Track here so the next session knows what was actioned : + +- ` — ` — link +- `<branch> — <title>` — link + +## Take-aways + +Free-form. What did we learn ? What surprised us ? What will we change for the next drill ? diff --git a/infra/ansible/tests/test_rabbitmq_outage.sh b/infra/ansible/tests/test_rabbitmq_outage.sh new file mode 100755 index 000000000..fa58735b2 --- /dev/null +++ b/infra/ansible/tests/test_rabbitmq_outage.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# test_rabbitmq_outage.sh — Game day scenario E. +# +# Verifies the backend's RabbitMQ-down behaviour matches the contract : +# - the API stays up (no 5xx on /api/v1/health) +# - the eventbus logs ERROR (loud, not silent) when publishes fail +# - the API recovers cleanly when RabbitMQ comes back +# +# v1.0.9 W5 Day 22. +# +# Usage : +# bash infra/ansible/tests/test_rabbitmq_outage.sh +# +# Exit codes : +# 0 — backend stayed up + recovered + error logged loudly +# 1 — backend wasn't healthy at start +# 2 — observed silent fail, 5xx, or no error log during outage +# 3 — required tool missing +set -euo pipefail + +RABBITMQ_CONTAINER=${RABBITMQ_CONTAINER:-rabbitmq} +BACKEND_HOST=${BACKEND_HOST:-haproxy.lxd} +BACKEND_PORT=${BACKEND_PORT:-80} +HEALTH_PATH=${HEALTH_PATH:-/api/v1/health} +OUTAGE_SECONDS=${OUTAGE_SECONDS:-60} # default 60s ; roadmap says 30 min in prod drill + +log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; } +fail() { log "FAIL: $*"; exit "${2:-2}"; } + +require() { + command -v "$1" >/dev/null 2>&1 || fail "required tool missing: $1" 3 +} + +require incus +require curl +require date + +curl_health() { + curl --max-time 5 -sS -o /dev/null -w "%{http_code}" \ + "http://${BACKEND_HOST}:${BACKEND_PORT}${HEALTH_PATH}" || echo "000" +} + +# ----------------------------------------------------------------------------- +# 0. Pre-flight — backend healthy, RabbitMQ container running. +# ----------------------------------------------------------------------------- +log "step 0: pre-flight" +status=$(curl_health) +log " backend health : HTTP $status" +if [ "$status" != "200" ]; then + fail "backend not healthy at start ($status), aborting" 1 +fi +if ! incus info "$RABBITMQ_CONTAINER" >/dev/null 2>&1; then + fail "rabbitmq container '$RABBITMQ_CONTAINER' not found" 1 +fi + +# ----------------------------------------------------------------------------- +# 1. Take RabbitMQ down for OUTAGE_SECONDS. +# ----------------------------------------------------------------------------- +log "step 1: stopping $RABBITMQ_CONTAINER for ${OUTAGE_SECONDS}s" +incus stop --force "$RABBITMQ_CONTAINER" +t0=$(date +%s) + +# ----------------------------------------------------------------------------- +# 2. Probe the backend every 5s during the outage. Acceptance gate : +# no run of 5xx longer than 30s. +# ----------------------------------------------------------------------------- +log "step 2: probing backend during the outage" +five_xx_streak=0 +five_xx_max_streak=0 +deadline=$((t0 + OUTAGE_SECONDS)) +while [ "$(date +%s)" -lt "$deadline" ]; do + s=$(curl_health) + if [ "$s" -ge 500 ] 2>/dev/null; then + five_xx_streak=$((five_xx_streak + 1)) + if [ "$five_xx_streak" -gt "$five_xx_max_streak" ]; then + five_xx_max_streak=$five_xx_streak + fi + else + five_xx_streak=0 + fi + log " [t+$(($(date +%s) - t0))s] backend HTTP $s (5xx streak=$five_xx_streak)" + sleep 5 +done + +# 30s = 6 consecutive 5s probes. If max_streak >= 6 we exceeded the gate. +if [ "$five_xx_max_streak" -ge 6 ]; then + log "FAIL gate : observed $((five_xx_max_streak * 5))s of consecutive 5xx during outage (cap is 30s)" + log "step 3: restarting $RABBITMQ_CONTAINER (cleanup)" + incus start "$RABBITMQ_CONTAINER" || true + exit 2 +fi + +# ----------------------------------------------------------------------------- +# 3. Verify the backend logged an ERROR for the rabbit failure (loud, not +# silent). We grep journald on the haproxy container's view of the +# api pool — assumes the API logs to journald via the systemd unit +# rendered by the backend_api role. +# ----------------------------------------------------------------------------- +log "step 3: looking for 'eventbus' / 'rabbitmq' ERROR log lines" +err_count=$(incus exec backend-api-1 -- journalctl -u veza-backend-api --since="-${OUTAGE_SECONDS} seconds" 2>/dev/null \ + | grep -ciE "rabbitmq|eventbus" || true) +log " matching log lines : $err_count" +if [ "$err_count" -lt 1 ]; then + log "WARN : no rabbitmq/eventbus ERROR log lines during the outage." + log " Either the eventbus path went unused (no events published) or" + log " failures are swallowed silently — open an issue if the latter." + # Don't fail the whole test on this — depending on traffic during the + # window the eventbus may not have been touched. Operator inspects. +fi + +# ----------------------------------------------------------------------------- +# 4. Restart RabbitMQ + verify backend recovers. +# ----------------------------------------------------------------------------- +log "step 4: restarting $RABBITMQ_CONTAINER" +incus start "$RABBITMQ_CONTAINER" + +log " waiting up to 60s for backend to be 200 again" +deadline=$(( $(date +%s) + 60 )) +recovered=0 +while [ "$(date +%s)" -lt "$deadline" ]; do + s=$(curl_health) + if [ "$s" = "200" ]; then + recovered=1 + break + fi + sleep 5 +done +if [ "$recovered" -ne 1 ]; then + fail "backend did not return to HTTP 200 within 60s after RabbitMQ recovery" 2 +fi + +log "PASS : backend stayed available during ${OUTAGE_SECONDS}s RabbitMQ outage" +log " max consecutive 5xx streak : ${five_xx_max_streak} probes ($((five_xx_max_streak * 5))s)" +log " eventbus error log lines : $err_count" +exit 0 diff --git a/scripts/security/game-day-driver.sh b/scripts/security/game-day-driver.sh new file mode 100755 index 000000000..b8496b776 --- /dev/null +++ b/scripts/security/game-day-driver.sh @@ -0,0 +1,148 @@ +#!/usr/bin/env bash +# game-day-driver.sh — orchestrate the W5 Day 22 game-day exercise. +# +# Walks the 5 failure scenarios in sequence, captures stdout/stderr + +# exit code per scenario, writes a session report under +# docs/runbooks/game-days/<DATE>-game-day-driver.log, and prints a +# summary table at the end. +# +# v1.0.9 W5 Day 22. +# +# Scenarios (mapped to existing smoke tests) : +# A : test_pg_failover.sh — kill Postgres primary, RTO < 60s +# B : test_backend_failover.sh — kill backend-api 1, HAProxy bascule +# C : test_redis_failover.sh — kill Redis master, Sentinel promote +# D : test_minio_resilience.sh — kill 2 MinIO nodes, EC:2 reconstructs +# E : test_rabbitmq_outage.sh — stop RabbitMQ 60s, backend stays up +# +# Usage : +# bash scripts/security/game-day-driver.sh # run all scenarios +# SKIP=DE bash scripts/security/game-day-driver.sh # skip scenarios D + E +# ONLY=A bash scripts/security/game-day-driver.sh # only run scenario A +# +# Required env (passed through to the underlying smoke tests) : +# REDIS_PASS / SENTINEL_PASS for scenario C +# MINIO_ROOT_USER / MINIO_ROOT_PASSWORD for scenario D +# +# Exit codes : +# 0 — every selected scenario passed +# 1 — at least one scenario failed +# 2 — runner pre-flight failed (script missing, etc.) +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +TESTS_DIR="$REPO_ROOT/infra/ansible/tests" +LOGS_DIR="$REPO_ROOT/docs/runbooks/game-days" +SESSION_DATE="$(date +%Y-%m-%d-%H%M)" +SESSION_LOG="$LOGS_DIR/$SESSION_DATE-game-day-driver.log" + +mkdir -p "$LOGS_DIR" +: > "$SESSION_LOG" + +ONLY=${ONLY:-} +SKIP=${SKIP:-} + +log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" | tee -a "$SESSION_LOG" >&2; } +fail() { log "FAIL: $*"; exit "${2:-2}"; } + +declare -A SCENARIO_SCRIPT=( + [A]="$TESTS_DIR/test_pg_failover.sh" + [B]="$TESTS_DIR/test_backend_failover.sh" + [C]="$TESTS_DIR/test_redis_failover.sh" + [D]="$TESTS_DIR/test_minio_resilience.sh" + [E]="$TESTS_DIR/test_rabbitmq_outage.sh" +) +declare -A SCENARIO_DESC=( + [A]="Postgres primary failover RTO < 60s" + [B]="HAProxy backend-api 1 fail-over" + [C]="Redis Sentinel master promotion" + [D]="MinIO 2-node loss EC:2 reconstruction" + [E]="RabbitMQ outage backend stays up" +) +SCENARIOS=(A B C D E) + +want() { + local s=$1 + if [ -n "$ONLY" ] && [[ "$ONLY" != *"$s"* ]]; then return 1; fi + if [ -n "$SKIP" ] && [[ "$SKIP" == *"$s"* ]]; then return 1; fi + return 0 +} + +# Pre-flight : every selected scenario script must exist + be executable. +for s in "${SCENARIOS[@]}"; do + if want "$s"; then + script="${SCENARIO_SCRIPT[$s]}" + if [ ! -x "$script" ]; then + fail "scenario $s : script $script not found or not executable" 2 + fi + fi +done + +declare -A SCENARIO_RESULT +declare -A SCENARIO_DURATION + +log "================================================================" +log "Game day session : $SESSION_DATE" +log "Session log : $SESSION_LOG" +log "Scenarios run : ${SCENARIOS[*]}" +[ -n "$ONLY" ] && log "ONLY filter : $ONLY" +[ -n "$SKIP" ] && log "SKIP filter : $SKIP" +log "================================================================" + +for s in "${SCENARIOS[@]}"; do + if ! want "$s"; then + SCENARIO_RESULT[$s]="SKIPPED" + SCENARIO_DURATION[$s]="-" + continue + fi + + log "" + log "── scenario $s : ${SCENARIO_DESC[$s]} ──────────────────────────" + t0=$(date +%s) + set +e + "${SCENARIO_SCRIPT[$s]}" 2>&1 | tee -a "$SESSION_LOG" + rc=${PIPESTATUS[0]} + set -e + elapsed=$(( $(date +%s) - t0 )) + SCENARIO_DURATION[$s]="${elapsed}s" + if [ "$rc" -eq 0 ]; then + SCENARIO_RESULT[$s]="PASS" + log "scenario $s : PASS in ${elapsed}s" + else + SCENARIO_RESULT[$s]="FAIL (exit $rc)" + log "scenario $s : FAIL (exit $rc) after ${elapsed}s" + fi +done + +log "" +log "================================================================" +log "Session summary" +log "----------------------------------------------------------------" +printf '%-3s | %-12s | %-8s | %s\n' "ID" "result" "duration" "scenario" | tee -a "$SESSION_LOG" >&2 +printf '%-3s-+-%-12s-+-%-8s-+-%s\n' "---" "------------" "--------" "$(printf '%.0s-' {1..50})" | tee -a "$SESSION_LOG" >&2 +overall=0 +for s in "${SCENARIOS[@]}"; do + result=${SCENARIO_RESULT[$s]} + duration=${SCENARIO_DURATION[$s]} + printf '%-3s | %-12s | %-8s | %s\n' "$s" "$result" "$duration" "${SCENARIO_DESC[$s]}" \ + | tee -a "$SESSION_LOG" >&2 + if [[ "$result" == "FAIL"* ]]; then overall=1; fi +done +log "================================================================" +log "" +log "Operator next steps :" +log " 1. Open the runbook template :" +log " docs/runbooks/game-days/$SESSION_DATE.md" +log " (copy from docs/runbooks/game-days/TEMPLATE.md if missing)" +log " 2. For each scenario, fill : timestamp, action, observation," +log " runbook used, gap discovered." +log " 3. File one PR per gap that needs a code or runbook fix." +log "" + +if [ "$overall" -eq 0 ]; then + log "PASS : every selected scenario passed." +else + log "FAIL : at least one scenario failed — review $SESSION_LOG." +fi + +exit "$overall"