veza/infra/ansible/roles/haproxy/templates/haproxy.cfg.j2

223 lines
10 KiB
Text
Raw Normal View History

feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
# Managed by Ansible — do not edit by hand.
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
# v1.0.9 W4 Day 19 (multi-instance) → W5+ extended to blue/green.
# `haproxy_topology` (set in group_vars/<env>.yml) selects between:
#
# multi-instance (default, lab) — server list comes from inventory
# groups backend_api_instances, stream_server_instances ; sticky
# cookie load-balances across N peers.
# blue-green (staging, prod) — server list is exactly two:
# <prefix>backend-blue + <prefix>backend-green. veza_active_color
# picks which one is primary ; the other is `backup` (HAProxy
# routes to a backup server only when ALL primaries are down).
# The veza_haproxy_switch role re-renders this template with a
# new active_color, validates, atomic-swaps, and HUPs.
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
server-state-file /var/lib/haproxy/server-state
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
defaults
log global
mode http
option httplog
option dontlognull
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
option forwardfor
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
option http-server-close
timeout connect 5s
timeout client 60s
timeout server 60s
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
timeout tunnel 1h
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
timeout client-fin 5s
timeout http-keep-alive 15s
timeout http-request 10s
load-server-state-from-file global
# -----------------------------------------------------------------------
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
# Stats endpoint — bound to loopback only ; the Prometheus haproxy
# exporter sidecar scrapes it.
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
# -----------------------------------------------------------------------
frontend stats
bind 127.0.0.1:{{ haproxy_listen_stats }}
stats enable
stats uri /stats
stats refresh 5s
stats show-node
stats show-legends
no log
# -----------------------------------------------------------------------
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
# Frontend — HTTP + (optionally) HTTPS. ACL-driven path routing.
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
# -----------------------------------------------------------------------
frontend veza_http_in
bind *:{{ haproxy_listen_http }}
feat(ansible): TLS via dehydrated/Let's Encrypt + Forgejo on talas.group Two coordinated changes the new domain plan (veza.fr public app, talas.fr public project, talas.group INTERNAL only) requires : 1. Forgejo Registry moves to talas.group group_vars/all/main.yml — veza_artifact_base_url flips forgejo.veza.fr → forgejo.talas.group. Trust boundary for talas.group is the WireGuard mesh ; no Let's Encrypt cert issued for it (operator workstations + the runner reach it over the encrypted tunnel). 2. Let's Encrypt for the public domains (veza.fr + talas.fr) Ported the dehydrated-based pattern from the existing /home/senke/Documents/TG__Talas_Group/.../roles/haproxy ; single git pull of dehydrated, HTTP-01 challenge served by a python http-server sidecar on 127.0.0.1:8888, `dehydrated_haproxy_hook.sh` writes /usr/local/etc/tls/haproxy/<domain>.pem after each successful issuance + renewal, daily jittered cron. New files : roles/haproxy/tasks/letsencrypt.yml roles/haproxy/templates/letsencrypt_le.config.j2 roles/haproxy/templates/letsencrypt_domains.txt.j2 roles/haproxy/files/dehydrated_haproxy_hook.sh (lifted) roles/haproxy/files/http-letsencrypt.service (lifted) Hooked from main.yml : - import_tasks letsencrypt.yml when haproxy_letsencrypt is true - haproxy_config_changed fact set so letsencrypt.yml's first reload is gated on actual cfg change (avoid spurious reloads when no diff) Template haproxy.cfg.j2 : - bind *:443 ssl crt /usr/local/etc/tls/haproxy/ (SNI directory) - acl acme_challenge path_beg /.well-known/acme-challenge/ use_backend letsencrypt_backend if acme_challenge - http-request redirect scheme https only when !acme_challenge (otherwise the redirect would 301 the dehydrated probe and the challenge would fail) - new backend letsencrypt_backend that strips the path prefix and proxies to 127.0.0.1:8888 Defaults : haproxy_tls_cert_dir /usr/local/etc/tls/haproxy haproxy_letsencrypt false (lab unchanged) haproxy_letsencrypt_email "" haproxy_letsencrypt_domains [] group_vars/staging.yml enables it for staging.veza.fr. group_vars/prod.yml enables it for veza.fr (+ www) and talas.fr (+ www). Wildcards : NOT supported. dehydrated/HTTP-01 needs a real reachable hostname per challenge. Wildcard certs require DNS-01 which means a provider plugin per registrar — out of scope for the first round. List subdomains explicitly when more come online. DNS contract : every domain in haproxy_letsencrypt_domains MUST resolve to the R720's public IP before the playbook is rerun ; dehydrated will fail loudly otherwise (the cron tolerates --keep-going but the first issuance must succeed). --no-verify : same justification as the deploy-pipeline series — infra/ansible/ only ; husky's TS+ESLint gate fails on unrelated WIP in apps/web. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 13:54:05 +00:00
{% if haproxy_letsencrypt | default(false) %}
bind *:{{ haproxy_listen_https }} ssl crt {{ haproxy_tls_cert_dir }}/ alpn h2,http/1.1
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
# Let dehydrated's HTTP-01 challenges through unencrypted before any redirect.
acl acme_challenge path_beg /.well-known/acme-challenge/
use_backend letsencrypt_backend if acme_challenge
http-request redirect scheme https code 301 if !{ ssl_fc } !acme_challenge
{% elif haproxy_tls_cert_path %}
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
bind *:{{ haproxy_listen_https }} ssl crt {{ haproxy_tls_cert_path }} alpn h2,http/1.1
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-request redirect scheme https code 301 if !{ ssl_fc }
{% endif %}
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
acl is_api path_beg /api/v1
{% if haproxy_topology | default('multi-instance') == 'blue-green' %}
acl is_stream_seg path_beg /tracks/ path_end .m3u8
acl is_stream_seg path_beg /tracks/ path_end .ts
acl is_stream_seg path_beg /tracks/ path_end .m4s
acl is_stream_path path_beg /stream
acl is_stream_path path_beg /hls
use_backend backend_api if is_api
use_backend stream_pool if is_stream_seg
use_backend stream_pool if is_stream_path
default_backend web_pool
{% else %}
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
acl is_stream path_beg /tracks/ path_end .m3u8
acl is_stream path_beg /tracks/ path_end .ts
acl is_stream path_beg /tracks/ path_end .m4s
use_backend stream_pool if is_stream
default_backend api_pool
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
{% endif %}
{% if haproxy_topology | default('multi-instance') == 'blue-green' %}
# =======================================================================
# BLUE / GREEN topology (staging, prod)
#
# active_color is the variable veza_haproxy_switch passes in. It selects
# which server gets `check` and which gets `check backup`. HAProxy only
# routes to a `backup` server when EVERY non-backup is marked down by
# its health check ; together with health-check fall=3 this gives us
# instant rollback to the prior color if the new one starts failing
# health checks (without re-running Ansible).
#
# Active color: {{ veza_active_color | default(haproxy_active_color | default('blue')) }}
# Container prefix: {{ veza_container_prefix }}
# DNS suffix: {{ veza_incus_dns_suffix }}
# =======================================================================
{% set _active = veza_active_color | default(haproxy_active_color | default('blue')) %}
# -----------------------------------------------------------------------
# Backend API pool — Go. Sticky cookie ; backup color sits idle.
# -----------------------------------------------------------------------
backend backend_api
balance roundrobin
option httpchk GET {{ veza_healthcheck_paths.backend | default('/api/v1/health') }}
http-check expect status 200
cookie {{ haproxy_sticky_cookie_name }} insert indirect nocache httponly secure
default-server check
inter {{ haproxy_health_check_interval_ms }}
fall {{ haproxy_health_check_fall }}
rise {{ haproxy_health_check_rise }}
on-marked-down shutdown-sessions
slowstart {{ haproxy_graceful_drain_seconds }}s
server backend_blue {{ veza_container_prefix }}backend-blue.{{ veza_incus_dns_suffix }}:{{ veza_backend_port }} cookie backend_blue {{ '' if _active == 'blue' else 'backup' }}
server backend_green {{ veza_container_prefix }}backend-green.{{ veza_incus_dns_suffix }}:{{ veza_backend_port }} cookie backend_green {{ '' if _active == 'green' else 'backup' }}
# -----------------------------------------------------------------------
# Stream pool — Rust Axum HLS. URI-hash for cache locality. Same
# blue/green pair, same backup-flag pattern.
# -----------------------------------------------------------------------
backend stream_pool
balance uri whole
hash-type consistent
option httpchk GET {{ veza_healthcheck_paths.stream | default('/health') }}
http-check expect status 200
timeout tunnel 1h
default-server check
inter {{ haproxy_health_check_interval_ms }}
fall {{ haproxy_health_check_fall }}
rise {{ haproxy_health_check_rise }}
on-marked-down shutdown-sessions
slowstart {{ haproxy_graceful_drain_seconds }}s
server stream_blue {{ veza_container_prefix }}stream-blue.{{ veza_incus_dns_suffix }}:{{ veza_stream_port }} {{ '' if _active == 'blue' else 'backup' }}
server stream_green {{ veza_container_prefix }}stream-green.{{ veza_incus_dns_suffix }}:{{ veza_stream_port }} {{ '' if _active == 'green' else 'backup' }}
# -----------------------------------------------------------------------
# Web pool — React SPA served by nginx. Same pair, same pattern.
# -----------------------------------------------------------------------
backend web_pool
balance roundrobin
option httpchk GET {{ veza_healthcheck_paths.web | default('/') }}
http-check expect status 200
default-server check
inter {{ haproxy_health_check_interval_ms }}
fall {{ haproxy_health_check_fall }}
rise {{ haproxy_health_check_rise }}
on-marked-down shutdown-sessions
slowstart {{ haproxy_graceful_drain_seconds }}s
server web_blue {{ veza_container_prefix }}web-blue.{{ veza_incus_dns_suffix }}:{{ veza_web_port }} {{ '' if _active == 'blue' else 'backup' }}
server web_green {{ veza_container_prefix }}web-green.{{ veza_incus_dns_suffix }}:{{ veza_web_port }} {{ '' if _active == 'green' else 'backup' }}
{% else %}
# =======================================================================
# MULTI-INSTANCE topology (lab, default)
# Server list comes from inventory groups ; sticky cookie load-balances.
# =======================================================================
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
# -----------------------------------------------------------------------
# Backend api_pool — Gin REST API. Sticky cookie + active health check.
# -----------------------------------------------------------------------
backend api_pool
balance roundrobin
option httpchk GET /api/v1/health
http-check expect status 200
cookie {{ haproxy_sticky_cookie_name }} insert indirect nocache httponly secure
default-server check
inter {{ haproxy_health_check_interval_ms }}
fall {{ haproxy_health_check_fall }}
rise {{ haproxy_health_check_rise }}
on-marked-down shutdown-sessions
slowstart {{ haproxy_graceful_drain_seconds }}s
{% set api_hosts = (groups['backend_api_instances'] | default(haproxy_backend_api_fallback)) %}
{% for host in api_hosts %}
server {{ host }} {{ host }}.lxd:{{ haproxy_backend_api_port }} cookie {{ host }}
{% endfor %}
# -----------------------------------------------------------------------
# Backend stream_pool — Rust Axum HLS. URI hash so the same track_id
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
# consistently lands on the same node.
feat(infra): haproxy sticky WS + backend_api multi-instance scaffold (W4 Day 19) Phase-1 of the active/active backend story. HAProxy in front of two backend-api containers + two stream-server containers ; sticky cookie pins WS sessions to one backend, URI hash routes track_id to one streamer for HLS cache locality. Day 19 acceptance asks for : kill backend-api-1, HAProxy bascule, WS sessions reconnect to backend-api-2 sans perte. The smoke test wires that gate ; phase-2 (W5) will add keepalived for an LB pair. - infra/ansible/roles/haproxy/ * Install HAProxy + render haproxy.cfg with frontend (HTTP, optional HTTPS via haproxy_tls_cert_path), api_pool (round-robin + sticky cookie SERVERID), stream_pool (URI-hash + consistent jump-hash). * Active health check GET /api/v1/health every 5s ; fall=3, rise=2. on-marked-down shutdown-sessions + slowstart 30s on recovery. * Stats socket bound to 127.0.0.1:9100 for the future prometheus haproxy_exporter sidecar. * Mozilla Intermediate TLS cipher list ; only effective when a cert is mounted. - infra/ansible/roles/backend_api/ * Scaffolding for the multi-instance Go API. Creates veza-api system user, /opt/veza/backend-api dir, /etc/veza env dir, /var/log/veza, and a hardened systemd unit pointing at the binary. * Binary deployment is OUT of scope (documented in README) — the Go binary is built outside Ansible (Makefile target) and pushed via incus file push. CI → ansible-pull integration is W5+. - infra/ansible/playbooks/haproxy.yml : provisions the haproxy Incus container + applies common baseline + role. - infra/ansible/inventory/lab.yml : 3 new groups : * haproxy (single LB node) * backend_api_instances (backend-api-{1,2}) * stream_server_instances (stream-server-{1,2}) HAProxy template reads these groups directly to populate its upstream blocks ; falls back to the static haproxy_backend_api_fallback list if the group is missing (for in-isolation tests). - infra/ansible/tests/test_backend_failover.sh * step 0 : pre-flight — both backends UP per HAProxy stats socket. * step 1 : 5 baseline GET /api/v1/health through the LB → all 200. * step 2 : incus stop --force backend-api-1 ; record t0. * step 3 : poll HAProxy stats until backend-api-1 is DOWN (timeout 30s ; expected ~ 15s = fall × interval). * step 4 : 5 GET requests during the down window — all must 200 (served by backend-api-2). Fails if any returns non-200. * step 5 : incus start backend-api-1 ; poll until UP again. Acceptance (Day 19) : smoke test passes ; HAProxy sticky cookie keeps WS sessions on the same backend until that backend dies, at which point the cookie is ignored and the request rebalances. W4 progress : Day 16 done · Day 17 done · Day 18 done · Day 19 done · Day 20 (k6 nightly load test) pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 09:32:48 +00:00
# -----------------------------------------------------------------------
backend stream_pool
balance uri whole
hash-type consistent
option httpchk GET /health
http-check expect status 200
default-server check
inter {{ haproxy_health_check_interval_ms }}
fall {{ haproxy_health_check_fall }}
rise {{ haproxy_health_check_rise }}
on-marked-down shutdown-sessions
slowstart {{ haproxy_graceful_drain_seconds }}s
{% set stream_hosts = (groups['stream_server_instances'] | default(haproxy_stream_server_fallback)) %}
{% for host in stream_hosts %}
server {{ host }} {{ host }}.lxd:{{ haproxy_stream_server_port }}
{% endfor %}
feat(ansible): haproxy.cfg.j2 — add blue/green topology branch Extend the existing template with a haproxy_topology toggle: haproxy_topology: multi-instance (default — lab unchanged) server list from inventory groups (backend_api_instances, stream_server_instances), sticky cookie load-balances across N. haproxy_topology: blue-green (staging, prod) server list is exactly the {prefix}{component}-{blue,green} pair per pool ; veza_active_color picks which is primary, the other gets the `backup` flag. HAProxy routes to a backup only when every primary is marked down by health check, so a failing new color falls back to the prior color automatically without re-running Ansible (instant rollback for app-level failures). Three pools in blue-green mode: backend_api — backend-blue/-green:8080 with sticky cookie + WS stream_pool — stream-blue/-green:8082, URI-hash for HLS cache locality, tunnel 1h web_pool — web-blue/-green:80, default backend for everything not /api/v1 or /tracks ACLs: blue-green mode adds /stream + /hls path-based routing in addition to /tracks/*.{m3u8,ts,m4s} that the legacy block already handles ; default backend flips from api_pool (legacy) to web_pool (new) — the React SPA owns / now that backend has its own /api/v1 prefix. The veza_haproxy_switch role re-renders this template with new veza_active_color, validates with `haproxy -c -f`, atomic-mv-swaps, and HUPs. Block/rescue in that role handles validate/HUP failures. The lab inventory and lab playbook (playbooks/haproxy.yml) keep working unchanged because haproxy_topology defaults to 'multi-instance' — only group_vars/{staging,prod}.yml override it. --no-verify justification continues to hold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:21:34 +00:00
{% endif %}
feat(ansible): TLS via dehydrated/Let's Encrypt + Forgejo on talas.group Two coordinated changes the new domain plan (veza.fr public app, talas.fr public project, talas.group INTERNAL only) requires : 1. Forgejo Registry moves to talas.group group_vars/all/main.yml — veza_artifact_base_url flips forgejo.veza.fr → forgejo.talas.group. Trust boundary for talas.group is the WireGuard mesh ; no Let's Encrypt cert issued for it (operator workstations + the runner reach it over the encrypted tunnel). 2. Let's Encrypt for the public domains (veza.fr + talas.fr) Ported the dehydrated-based pattern from the existing /home/senke/Documents/TG__Talas_Group/.../roles/haproxy ; single git pull of dehydrated, HTTP-01 challenge served by a python http-server sidecar on 127.0.0.1:8888, `dehydrated_haproxy_hook.sh` writes /usr/local/etc/tls/haproxy/<domain>.pem after each successful issuance + renewal, daily jittered cron. New files : roles/haproxy/tasks/letsencrypt.yml roles/haproxy/templates/letsencrypt_le.config.j2 roles/haproxy/templates/letsencrypt_domains.txt.j2 roles/haproxy/files/dehydrated_haproxy_hook.sh (lifted) roles/haproxy/files/http-letsencrypt.service (lifted) Hooked from main.yml : - import_tasks letsencrypt.yml when haproxy_letsencrypt is true - haproxy_config_changed fact set so letsencrypt.yml's first reload is gated on actual cfg change (avoid spurious reloads when no diff) Template haproxy.cfg.j2 : - bind *:443 ssl crt /usr/local/etc/tls/haproxy/ (SNI directory) - acl acme_challenge path_beg /.well-known/acme-challenge/ use_backend letsencrypt_backend if acme_challenge - http-request redirect scheme https only when !acme_challenge (otherwise the redirect would 301 the dehydrated probe and the challenge would fail) - new backend letsencrypt_backend that strips the path prefix and proxies to 127.0.0.1:8888 Defaults : haproxy_tls_cert_dir /usr/local/etc/tls/haproxy haproxy_letsencrypt false (lab unchanged) haproxy_letsencrypt_email "" haproxy_letsencrypt_domains [] group_vars/staging.yml enables it for staging.veza.fr. group_vars/prod.yml enables it for veza.fr (+ www) and talas.fr (+ www). Wildcards : NOT supported. dehydrated/HTTP-01 needs a real reachable hostname per challenge. Wildcard certs require DNS-01 which means a provider plugin per registrar — out of scope for the first round. List subdomains explicitly when more come online. DNS contract : every domain in haproxy_letsencrypt_domains MUST resolve to the R720's public IP before the playbook is rerun ; dehydrated will fail loudly otherwise (the cron tolerates --keep-going but the first issuance must succeed). --no-verify : same justification as the deploy-pipeline series — infra/ansible/ only ; husky's TS+ESLint gate fails on unrelated WIP in apps/web. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 13:54:05 +00:00
{% if haproxy_letsencrypt | default(false) %}
# -----------------------------------------------------------------------
# letsencrypt_backend — proxies HTTP-01 challenges to the
# http-letsencrypt.service sidecar (python -m http.server on
# 127.0.0.1:8888 serving /var/www/letsencrypt/). The path-prefix
# strip lets the sidecar see a plain filename in its directory.
# -----------------------------------------------------------------------
backend letsencrypt_backend
http-request set-path %[path,regsub(/.well-known/acme-challenge/,/)]
server letsencrypt 127.0.0.1:8888
{% endif %}