veza/docs/runbooks/api-availability-slo-burn.md

Runbook — API availability SLO burn

SLO : 99.5% of GET requests on /api/v1/* return non-5xx (monthly window). Alerts : APIAvailabilitySLOFastBurn (page) · APIAvailabilitySLOSlowBurn (ticket) Owner : backend on-call.

What tripped me

The 5xx ratio on read endpoints is consuming the monthly error budget faster than the steady-state rate allows :

• Fast burn (page=true) : 14.4× over 1h ⇒ entire monthly budget gone in ~3.5 days.
• Slow burn (page=false) : 6× over 6h ⇒ entire budget gone in ~7 days.

First moves (under 5 minutes)

1. Confirm the alert is real, not a metric-pipeline glitch :

   # Live error rate on the GETs we measure :
   sum(rate(veza_gin_http_requests_total{job="veza-backend",method="GET",status=~"5.."}[5m]))
   /
   sum(rate(veza_gin_http_requests_total{job="veza-backend",method="GET"}[5m]))
   

   Open Grafana → "Veza API Overview" dashboard, panel "Request rate by path".

2. Identify the affected endpoint. The fastest pivot :

   topk(5, sum by (path, status) (
     rate(veza_gin_http_requests_total{job="veza-backend",method="GET",status=~"5.."}[5m])
   ))
   

3. Drop into traces. Open the "Veza Service Map (Tempo)" dashboard and filter the slowest-spans table for the offending path. If the failures correlate with one downstream (Redis, Postgres, Hyperswitch), the trace will show it.

Common causes

Symptom	Likely cause	Fix
5xx concentrated on /feed, /library	Postgres slow / connection pool exhausted	See db-failover.md — check pg_auto_failover state
5xx concentrated on /search, /tracks	Postgres FTS index churn or autovacuum holding row locks	SELECT pid, query FROM pg_stat_activity WHERE state='active' ORDER BY xact_start LIMIT 5;
5xx across all paths, sudden	Pod just restarted / migration broken / DB unreachable	kubectl get pods -n veza or systemctl status veza-backend-api
5xx slowly climbing	Memory leak; container approaching OOMKill	kubectl top pod -n veza and bounce the leaking pod
5xx confined to one instance	Single bad replica (config, certs, networking)	Drain that instance from the load balancer

If the page is real

1. Page the secondary on-call if the primary fix is going to take > 15 min.
2. Update the status page (status.veza.fr) with "Investigating elevated error rates."
3. Post in #incident-response with the alert link + first hypothesis.

When to silence

• Confirmed degradation is a known maintenance window already announced : silence for the maintenance window's duration.
• Single-instance issue, that instance has been drained : silence for 1h.
• Otherwise, do not silence — let the alert keep firing until the burn rate drops below threshold naturally.

Recovery verification

After mitigation, both burn-rate windows must drop below threshold for the alert to clear (1h and 5m for fast burn, 6h and 30m for slow burn). The 6h window means the slow-burn alert can stay green for hours after the issue is fixed — don't be surprised.

Postmortem trigger

A page-grade alert that fires for > 15 minutes triggers a postmortem doc (docs/postmortems/YYYY-MM-DD-<slug>.md). Include the timeline, the trace IDs, and the metric query screenshots.