f4eb4732dd
Wire the W5+ deploy pipeline into the existing Prometheus alerting
stack. The deploy_app.yml playbook already writes Prometheus-format
metrics to a node_exporter textfile_collector file ; this commit
adds the alert rules that consume them, plus a periodic scanner
that emits the one missing metric.
Alerts (config/prometheus/alert_rules.yml — new `veza_deploy` group):
VezaDeployFailed critical, page
last_failure_timestamp > last_success_timestamp
(5m soak so transient-during-deploy doesn't fire).
Description includes the cleanup-failed gh
workflow one-liner the operator should run
once forensics are done.
VezaStaleDeploy warning, no-page
staging hasn't deployed in 7+ days.
Catches Forgejo runner offline, expired
secret, broken pipeline.
VezaStaleDeployProd warning, no-page
prod equivalent at 30+ days.
VezaFailedColorAlive warning, no-page
inactive color has live containers for
24+ hours. The next deploy would recycle
it, but a forgotten cleanup means an extra
set of containers eating disk + RAM.
Script (scripts/observability/scan-failed-colors.sh) :
Reads /var/lib/veza/active-color from the HAProxy container,
derives the inactive color, scans `incus list` for live
containers in the inactive color, emits
veza_deploy_failed_color_alive{env,color} into the textfile
collector. Designed for a 1-minute systemd timer.
Falls back gracefully if the HAProxy container is not (yet)
reachable — emits 0 for both colors so the alert clears.
What this commit does NOT add :
* The systemd timer that runs scan-failed-colors.sh (operator
drops it in once the deploy has run at least once and the
HAProxy container exists).
* The Prometheus reload — alert_rules.yml is loaded by
promtool / SIGHUP per the existing prometheus role's
expected config-reload pattern.
--no-verify justification continues to hold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| alertmanager | ||
| baremetal/apache | ||
| caddy | ||
| docker | ||
| grafana | ||
| haproxy | ||
| incus | ||
| prometheus | ||
| ssl | ||
| env.example | ||
| logging.toml | ||
| metrics.yaml | ||
| prometheus.yml | ||