# Runbook — rollback a Veza deploy Three rollback paths, ordered from fastest to slowest. Pick based on what's still alive and what you're rolling back from. | Path | Time | Use when | | --------------------- | ---- | -------------------------------------------------- | | 1. HAProxy fast-flip | ~5s | The previous color's containers are still alive. | | 2. Re-deploy old SHA | ~10m | Previous color destroyed, but the old tarball is still in the Forgejo registry. | | 3. Manual emergency | ad-hoc | Both above failed (registry purged, infra broken). | > **Before you rollback, breathe and read this first.** The default > instinct under fire is "smash the rollback button". Often the right > call is to fix-forward — see "When NOT to rollback" at the bottom. --- ## Decision flowchart ``` Did the new color come up at all? │ ┌───────────┴────────────┐ │NO (HAProxy still on │YES (HAProxy switched, but │ old color, deploy job │ public probe failing or app │ went red in Phase D) │ broken in user reports) ▼ ▼ Phase F's auto-revert Use Path 1 (HAProxy fast-flip) already flipped HAProxy to flip BACK to the prior color. for you. No action The prior color is still alive needed beyond reading until the next deploy recycles it. logs. If the prior color was already cleaned up, use Path 2. ``` --- ## Path 1 — HAProxy fast-flip (~5s) Use when the prior color's containers are still alive. Triggered via the `Veza rollback` workflow with `mode=fast`. ### Pre-checks ```bash # What's the current active color? incus exec veza-staging-haproxy -- cat /var/lib/veza/active-color # (or veza-haproxy in prod) # What's the prior color (last entry of the history)? incus exec veza-staging-haproxy -- head -2 /var/lib/veza/active-color.history # Are the prior color's containers RUNNING? incus list 'veza-staging-{backend,stream,web}-blue' --format csv -c n,s ``` ### Trigger In the Forgejo UI: **Actions → Veza rollback → Run workflow**: | input | value | | ------------- | -------------------------- | | env | staging (or prod) | | mode | fast | | target_color | (the PRIOR color, eg blue if green is currently active) | | release_sha | (leave empty) | The workflow runs `infra/ansible/playbooks/rollback.yml -e mode=fast -e target_color=blue` which : 1. Verifies all three target-color containers are RUNNING (fails loud if not — switch to Path 2). 2. Re-templates `haproxy.cfg` with `veza_active_color=blue`, validates with `haproxy -c`, atomic-mv-swaps, HUPs. 3. Updates `/var/lib/veza/active-color`. Wall time: ~5s. Zero connection drop (HAProxy reload is graceful). ### Post-rollback - Verify externally: `curl https://staging.veza.fr/api/v1/health` - Check logs of the bad color (kept alive for forensics): `incus exec veza-staging-backend-green -- journalctl -u veza-backend -n 200` - Once root cause is understood, run the **Veza cleanup** workflow with `color=green` to reclaim the slot. --- ## Path 2 — Re-deploy older SHA (~10 minutes) Use when the prior color's containers were already destroyed (next deploy recycled them) but the old tarball is still in the Forgejo package registry. ### Pre-checks ```bash # Pick the SHA you want to roll back TO. # Look at the active-color.history for SHAs the pipeline knows about : incus exec veza-staging-haproxy -- head -10 /var/lib/veza/active-color.history # Or `git log --oneline main` for any commit ; just confirm the # tarball still exists in the registry (default retention 30 SHAs # per component) : curl -fsSL -I -H "Authorization: token $TOKEN" \ "https://forgejo.veza.fr/api/packages/talas/generic/veza-backend/$SHA/veza-backend-$SHA.tar.zst" ``` ### Trigger In the Forgejo UI: **Actions → Veza rollback → Run workflow**: | input | value | | ------------ | ------------------------------- | | env | staging (or prod) | | mode | full | | target_color | (leave empty) | | release_sha | the 40-char SHA you're rolling TO | The workflow runs `playbooks/rollback.yml -e mode=full -e veza_release_sha=$SHA` which `import_playbook`s the full `deploy_app.yml` pipeline. Same Phase A → Phase F sequence as a normal deploy, but with the older SHA. Wall time: ~5–10 minutes (build artefacts already exist, only the deploy half runs). ### Caveat — schema migrations Migrations are **not** rolled back automatically. The schema after `Path 2` is the post-deploy schema, not the pre-deploy schema. Per **MIGRATIONS.md**'s expand-contract discipline, this should be fine for one deploy back. If it isn't (i.e., the failed deploy included a destructive migration), see **Path 3**. --- ## Path 3 — Manual emergency (ad hoc) You're here when: - Forgejo registry has been purged of the SHA you need. - The schema migration is destructive and the app crashes against the post-migration schema. - The Incus host itself is in a bad state. ### Tarball missing — rebuild and push manually ```bash # Build the artefact locally (you'll need the toolchain) : cd veza-backend-api GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" \ -o ./bin/veza-api ./cmd/api/main.go tar --use-compress-program=zstd -cf "/tmp/veza-backend-$SHA.tar.zst" \ -C ./bin veza-api migrate_tool # Push to the registry : curl -fsSL --fail-with-body -X PUT \ -H "Authorization: token $TOKEN" \ --upload-file "/tmp/veza-backend-$SHA.tar.zst" \ "https://forgejo.veza.fr/api/packages/talas/generic/veza-backend/$SHA/veza-backend-$SHA.tar.zst" # Then run Path 2. ``` ### Schema is poisoned — manual SQL The destructive migration's PR description should document the inverse SQL (per MIGRATIONS.md "When you must violate the rule"). Apply it inside the postgres container : ```bash incus exec veza-staging-postgres -- psql -U veza veza < /tmp/inverse.sql ``` Then run Path 2 to deploy the older binary. ### Incus host broken — rollback ZFS snapshot `deploy_data.yml` snapshots every data container's dataset before mutating anything (`@pre-deploy-`). To restore : ```bash # First, stop the container : incus stop veza-staging-postgres # Roll the dataset back to the pre-deploy snapshot : zfs rollback -r rpool/incus/containers/veza-staging-postgres@pre-deploy- # Restart the container : incus start veza-staging-postgres ``` This loses any data written after the snapshot. Last-resort only. --- ## When NOT to rollback - **Single user reports a bug**. Triage first ; rolling back affects 100% of users to fix something hitting <1%. - **Performance regression**. If the new SHA is up but slow, scale horizontally before rolling back. (Future Hetzner offload covers this ; for now, accept the regression and prep a fix-forward.) - **Cosmetic UI bug**. Hot-fix the frontend and let the deploy pipeline ship it as a normal commit. - **You're not on-call and didn't get a page**. Don't rollback "to be safe". The on-call's call. The rollback button's existence isn't a license to use it preemptively. Each rollback resets the team's confidence in the pipeline ; over-rolling-back makes the next real deploy feel risky. --- ## Post-incident After ANY rollback (path 1, 2, or 3) : 1. Update **docs/POSTMORTEMS.md** (or `docs/runbooks/incidents/.md`) with what happened, why the deploy failed, and what triggered the rollback. 2. File the fix as a normal PR ; do NOT skip CI. 3. If the failed deploy left containers behind (Path 1's "old color kept alive"), run **Veza cleanup** workflow with the failed color once forensics are done. 4. Verify the alert `VezaDeployFailed` cleared (next successful deploy will reset `last_success_timestamp > last_failure_timestamp`). --- ## Workflows referenced - `.forgejo/workflows/deploy.yml` — push:main → staging, tag → prod. - `.forgejo/workflows/rollback.yml` — workflow_dispatch only, modes fast and full. - `.forgejo/workflows/cleanup-failed.yml` — workflow_dispatch only, destroys a specific color's app containers. ## Playbooks referenced - `infra/ansible/playbooks/deploy_app.yml` - `infra/ansible/playbooks/rollback.yml` - `infra/ansible/playbooks/cleanup_failed.yml` - `infra/ansible/playbooks/deploy_data.yml` ## Roles referenced - `infra/ansible/roles/veza_app/` - `infra/ansible/roles/veza_haproxy_switch/` - `infra/ansible/roles/haproxy/` (template `haproxy.cfg.j2` with blue/green topology toggle).