Commit graph

3 commits

Author SHA1 Message Date
senke
a36d9b2d59 feat(redis): Sentinel HA + cache hit rate metrics (W3 Day 11)
Some checks failed
Veza CI / Backend (Go) (push) Failing after 8m56s
Veza CI / Frontend (Web) (push) Has been cancelled
E2E Playwright / e2e (full) (push) Has been cancelled
Veza CI / Notify on failure (push) Blocked by required conditions
Veza CI / Rust (Stream Server) (push) Successful in 5m3s
Security Scan / Secret Scanning (gitleaks) (push) Failing after 53s
Three Incus containers, each running redis-server + redis-sentinel
(co-located). redis-1 = master at first boot, redis-2/3 = replicas.
Sentinel quorum=2 of 3 ; failover-timeout=30s satisfies the W3
acceptance criterion.

- internal/config/redis_init.go : initRedis branches on
  REDIS_SENTINEL_ADDRS ; non-empty -> redis.NewFailoverClient with
  MasterName + SentinelAddrs + SentinelPassword. Empty -> existing
  single-instance NewClient (dev/local stays parametric).
- internal/config/config.go : 3 new fields (RedisSentinelAddrs,
  RedisSentinelMasterName, RedisSentinelPassword) read from env.
  parseRedisSentinelAddrs trims+filters CSV.
- internal/metrics/cache_hit_rate.go : new RecordCacheHit / Miss
  counters, labelled by subsystem. Cardinality bounded.
- internal/middleware/rate_limiter.go : instrument 3 Eval call sites
  (DDoS, frontend log throttle, upload throttle). Hit = Redis answered,
  Miss = error -> in-memory fallback.
- internal/services/chat_pubsub.go : instrument Publish + PublishPresence.
- internal/websocket/chat/presence_service.go : instrument SetOnline /
  SetOffline / Heartbeat / GetPresence. redis.Nil counts as a hit
  (legitimate empty result).
- infra/ansible/roles/redis_sentinel/ : install Redis 7 + Sentinel,
  render redis.conf + sentinel.conf, systemd units. Vault assertion
  prevents shipping placeholder passwords to staging/prod.
- infra/ansible/playbooks/redis_sentinel.yml : provisions the 3
  containers + applies common baseline + role.
- infra/ansible/inventory/lab.yml : new groups redis_ha + redis_ha_master.
- infra/ansible/tests/test_redis_failover.sh : kills the master
  container, polls Sentinel for the new master, asserts elapsed < 30s.
- config/grafana/dashboards/redis-cache-overview.json : 3 hit-rate
  stats (rate_limiter / chat_pubsub / presence) + ops/s breakdown.
- docs/ENV_VARIABLES.md §3 : 3 new REDIS_SENTINEL_* env vars.
- veza-backend-api/.env.template : 3 placeholders (empty default).

Acceptance (Day 11) : Sentinel failover < 30s ; cache hit-rate
dashboard populated. Lab test pending Sentinel deployment.

W3 verification gate progress : Redis Sentinel ✓ (this commit),
MinIO EC4+2  Day 12, CDN  Day 13, DMCA  Day 14, embed  Day 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 13:36:55 +02:00
senke
97ca5209a1 fix(chat,config): require REDIS_URL in prod + error on in-memory fallback
Two connected failure modes that silently break multi-pod deployments:

  1. `RedisURL` has a struct-level default (`redis://<appDomain>:6379`)
     that makes `c.RedisURL == ""` always false. An operator forgetting
     to set `REDIS_URL` booted against a phantom host — every Redis call
     would then fail, and `ChatPubSubService` would quietly fall back to
     an in-memory map. On a single-pod deploy that "works"; on two pods
     it silently partitions chat (messages on pod A never reach
     subscribers on pod B).
  2. The fallback itself was logged at `Warn` level, buried under normal
     traffic. Operators only noticed when users reported stuck chats.

Changes:

  * `config.go` (`ValidateForEnvironment` prod branch): new check that
    `os.Getenv("REDIS_URL")` is non-empty. The struct field is left
    alone (dev + test still use the default); we inspect the raw env so
    the check is "explicitly set" rather than "non-empty after defaults".
  * `chat_pubsub.go` `NewChatPubSubService`: if `redisClient == nil`,
    emit an `ERROR` at construction time naming the failure mode
    ("cross-instance messages will be lost"). Same `Warn`→`Error`
    promotion for the `Publish` fallback path — runbook-worthy.

Tests: new `chat_pubsub_test.go` with a `zaptest/observer` that asserts
the ERROR-level log fires exactly once when Redis is nil, plus an
in-memory fan-out happy-path so single-pod dev behaviour stays covered.
New `TestValidateForEnvironment_RedisURLRequiredInProduction` mirrors
the Hyperswitch guard test shape.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:56:47 +02:00
senke
4d4d07836c feat(chat): Sprint 1 -- migrations, models, repositories for chat rewrite
- Add migrations 109-112: read_receipts, delivered_status, message_reactions, messages extra columns
- Create ReadReceipt, DeliveredStatus, MessageReaction GORM models
- Update Message model with EditedAt, Status, IsPinned, Metadata fields
- Enrich ChatMessageRepository with cursor pagination, search, soft delete
- Create ReadReceiptRepository, DeliveredStatusRepository, ReactionRepository
- Create ChatPubSubService with Redis PubSub and in-memory fallback
2026-02-22 20:38:20 +01:00