veza/infra/ansible/roles/blackbox_exporter/README.md

blackbox_exporter role — synthetic monitoring runner

Single Incus container running Prometheus' blackbox_exporter. Probed by Prometheus every 5 minutes against the 6 user parcours from v1.0.9 W5 Day 24. Alerts fire after 2 consecutive failures (for: 10m × 5-min scrape = 2 cycles).

Topology

                               Prometheus :9090
                                     │ scrape every 5m
                                     ▼
                       ┌─────────────────────────────┐
                       │ blackbox-exporter.lxd:9115  │
                       │  (this role)                │
                       └────────────┬────────────────┘
                                    │ probes (HTTP / TCP)
              ┌─────────────────────┼─────────────────────┐
              ▼                     ▼                     ▼
  staging.veza.fr/api/v1/auth/login  /api/v1/search?q=test   /api/v1/marketplace/products
              ...                                            ...

The exporter SHOULD run on a host external to the prod cluster so probe failures reflect what an external user sees, not what an already-broken internal service hides. v1.0 lab phase-1 colocates it for simplicity ; phase-2 moves the container off-box.

Probe modules (defined in templates/blackbox.yml.j2)

Module	Used by parcours	What it asserts
http_2xx	upload_init, live_streams	Status code 200 or 204, TLS valid
http_status_envelope	auth_login, status_endpoint	Body matches "success":\s*true
http_search	search	Body matches "tracks" (seed data must include hits)
http_marketplace	marketplace_list	200 (no body assertion ; an empty array is valid)
tcp_websocket	chat_websocket	TLS-wrapped TCP handshake completes

Multi-step parcours that need session state (Register → Verify → Login, Login → Search → Play first result) are out of scope for blackbox. Tracked as a follow-up : a small Go binary that runs as a CronJob, walks the steps, and writes textfile-collector metrics to /var/lib/node_exporter/textfile_collector/veza_synthetic.prom.

Defaults

variable	default	meaning
blackbox_version	0.25.0	Prometheus blackbox_exporter release
blackbox_listen_port	9115	Prometheus default
blackbox_target_base_url	https://staging.veza.fr	base URL the probes hit

Prometheus scrape config

config/prometheus/blackbox_targets.yml carries the 7 file-SD entries (6 parcours + status-endpoint bonus). Wire it in prometheus.yml :

scrape_configs:
  - job_name: blackbox
    file_sd_configs:
      - files: [/etc/prometheus/blackbox_targets.yml]
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: blackbox-exporter.lxd:9115

Alert rules

config/prometheus/alert_rules.yml group veza_synthetic :

• SyntheticParcoursDown — any parcours fails for 10 m → warning.
• SyntheticAuthLoginDown — auth_login fails for 10 m → critical (page).
• SyntheticProbeSlow — probe duration > 8 s for 15 m → warning.

Operations

# Service status :
sudo systemctl status blackbox_exporter

# One-off probe (dev / debug) :
curl 'http://blackbox-exporter.lxd:9115/probe?target=https://staging.veza.fr/api/v1/health&module=http_status_envelope'

# Live probe latency tail :
curl -s http://blackbox-exporter.lxd:9115/metrics | grep probe_duration

# Tail the exporter log :
sudo journalctl -u blackbox_exporter -f

What this role does NOT cover

• Multi-step parcours. Blackbox can't carry session cookies across probes ; the Register-then-Verify-then-Login flow needs a custom synthetic client. Tracked for v1.0.10.
• Status page. Cachet/statuspage.io is a separate operator decision per the roadmap. The /api/v1/status endpoint is consumable by both.
• Off-box deploy. Lab phase-1 runs the container on the same Incus host as the things it's probing. Phase-2 moves it off-cluster.