veza/veza-backend-api/cmd/api/main.go

349 lines
12 KiB
Go
Raw Normal View History

2025-12-03 19:29:37 +00:00
package main
import (
"context"
"fmt"
"log"
"net/http"
fix(v0.12.6): apply all pentest remediations — 36 findings across 36 files CRITICAL fixes: - Race condition (TOCTOU) in payout/refund with SELECT FOR UPDATE (CRITICAL-001/002) - IDOR on analytics endpoint — ownership check enforced (CRITICAL-003) - CSWSH on all WebSocket endpoints — origin whitelist (CRITICAL-004) - Mass assignment on user self-update — strip privileged fields (CRITICAL-005) HIGH fixes: - Path traversal in marketplace upload — UUID filenames (HIGH-001) - IP spoofing — use Gin trusted proxy c.ClientIP() (HIGH-002) - Popularity metrics (followers, likes) set to json:"-" (HIGH-003) - bcrypt cost hardened to 12 everywhere (HIGH-004) - Refresh token lock made mandatory (HIGH-005) - Stream token replay prevention with access_count (HIGH-006) - Subscription trial race condition fixed (HIGH-007) - License download expiration check (HIGH-008) - Webhook amount validation (HIGH-009) - pprof endpoint removed from production (HIGH-010) MEDIUM fixes: - WebSocket message size limit 64KB (MEDIUM-010) - HSTS header in nginx production (MEDIUM-001) - CORS origin restricted in nginx-rtmp (MEDIUM-002) - Docker alpine pinned to 3.21 (MEDIUM-003/004) - Redis authentication enforced (MEDIUM-005) - GDPR account deletion expanded (MEDIUM-006) - .gitignore hardened (MEDIUM-007) LOW/INFO fixes: - GitHub Actions SHA pinning on all workflows (LOW-001) - .env.example security documentation (INFO-001) - Production CORS set to HTTPS (LOW-002) All tests pass. Go and Rust compile clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 23:44:46 +00:00
// SECURITY(REM-027): pprof removed from production — use build tag or dedicated debug binary instead.
// To enable: go build -tags debug ./cmd/api
2025-12-03 19:29:37 +00:00
"os"
"os/signal"
"syscall"
"time"
P0: stabilisation backend/chat/stream + nouvelle base migrations v1 Backend Go: - Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN. - Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError). - Sécurisation de config.go, CORS, statuts de santé et monitoring. - Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles). - Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés. - Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*. Chat server (Rust): - Refonte du pipeline JWT + sécurité, audit et rate limiting avancé. - Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing). - Nettoyage des panics, gestion d’erreurs robuste, logs structurés. - Migrations chat alignées sur le schéma UUID et nouvelles features. Stream server (Rust): - Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core. - Transactions P0 pour les jobs et segments, garanties d’atomicité. - Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION). Documentation & audits: - TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services. - Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3). - Scripts de reset et de cleanup pour la lab DB et la V1. Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
2025-12-06 10:14:38 +00:00
"github.com/getsentry/sentry-go"
2025-12-03 19:29:37 +00:00
"github.com/gin-gonic/gin"
"github.com/joho/godotenv"
"go.uber.org/zap"
"veza-backend-api/internal/api"
"veza-backend-api/internal/config"
"veza-backend-api/internal/core/marketplace"
fix(backend): J4 — GDPR-compliant hard delete with Redis and ES cleanup Closes TODO(HIGH-007). When the hard-delete worker anonymizes a user past their recovery deadline, it now also cleans the user's residual data from Redis and Elasticsearch, not just PostgreSQL. Without this, a user who invoked their right to erasure would still appear in cached feed/profile responses and in ES search results for up to the next reindex cycle. Worker changes (internal/workers/hard_delete_worker.go): WithRedis / WithElasticsearch builder methods inject the clients. Both are optional: if either is nil (feature disabled or unreachable), the corresponding cleanup is skipped with a debug log and the worker keeps going. Partial progress beats panic. cleanRedisKeys uses SCAN with a cursor loop (COUNT 100), NEVER KEYS — KEYS would block the Redis server on multi-million-key deployments. Pattern is user:{id}:*. Transient SCAN errors retry up to 3 times with 100ms * retry linear backoff; persistent errors return without panic. DEL errors on a batch are logged but non-fatal so subsequent batches are still attempted. cleanESDocs hits three indices independently: - users index: DELETE doc by _id (the user UUID); 404 treated as success (already gone = desired state) - tracks index: DeleteByQuery with a terms filter on _id, using the list of track IDs collected from PostgreSQL BEFORE anonymization - playlists index: same pattern as tracks A failure on one index does not prevent the others from being tried; the first error is returned so the caller can log. Track/playlist IDs are pre-collected (collectTrackIDs, collectPlaylistIDs) before the UPDATE anonymization runs, because the anonymization does NOT cascade (no DELETE on users), so tracks and playlists rows remain with their creator_id / user_id intact and resolvable at query time. Wiring (cmd/api/main.go): The worker now receives cfg.RedisClient directly, and an optional ES client built from elasticsearch.LoadConfig() + NewClient. If ES is disabled or unreachable at startup, the worker logs a warning and proceeds with Redis-only cleanup. Tests (internal/workers/hard_delete_worker_test.go, +260 lines): Pure-function unit tests: - TestUUIDsToStrings - TestEsIndexNameFor Nil-client safety tests: - TestCleanRedisKeys_NilClientIsNoop - TestCleanESDocs_NilClientIsNoop ES mock-server tests (httptest.Server mimicking /_doc and /_delete_by_query endpoints with valid ES 8.11 responses): - TestCleanESDocs_CallsAllThreeIndices — verifies the three expected HTTP calls land with the right paths and request bodies containing the provided UUIDs - TestCleanESDocs_SkipsEmptyIDLists — verifies no DeleteByQuery is issued when the ID lists are empty Redis testcontainer integration test (gated by VEZA_SKIP_INTEGRATION): - TestCleanRedisKeys_Integration — seeds 154 keys (4 fixed + 150 bulk to force the SCAN loop past a single batch) plus 4 unrelated keys from another user / global, runs cleanRedisKeys, asserts all 154 own keys are gone and all 4 unrelated keys remain. Verification: go build ./... OK go vet ./... OK VEZA_SKIP_INTEGRATION=1 go test ./internal/workers/... short OK go test ./internal/workers/ -run TestCleanRedisKeys_Integration → testcontainers spins redis:7-alpine, test passes in 1.34s Out of J4 scope (noted for a follow-up): - No "activity" ES index exists in the codebase today (the audit plan mentioned it as a possible target). The three real indices with user data — users, tracks, playlists — are all now cleaned. - Track artist strings (free-form) may still contain the user's display name as a cached value in the tracks index after this cleanup. Actual user-owned tracks are deleted here, but if a third party's track referenced the removed user in its artist field, that reference is not touched. Strict RGPD on that edge case is a separate ticket. Refs: AUDIT_REPORT.md §8.5, §10 P5, §12 item 1
2026-04-15 10:25:39 +00:00
vezaes "veza-backend-api/internal/elasticsearch"
"veza-backend-api/internal/jobs"
2025-12-13 02:34:34 +00:00
"veza-backend-api/internal/metrics"
"veza-backend-api/internal/services"
"veza-backend-api/internal/shutdown"
"veza-backend-api/internal/workers"
2025-12-03 19:29:37 +00:00
_ "veza-backend-api/docs" // Import docs for swagger
)
// @title Veza Backend API
// @version 1.2.0
// @description Backend API for Veza platform.
// @termsOfService http://swagger.io/terms/
// @contact.name API Support
// @contact.url http://www.veza.app/support
// @contact.email support@veza.app
// @license.name Apache 2.0
// @license.url http://www.apache.org/licenses/LICENSE-2.0.html
// @host localhost:18080
2025-12-03 19:29:37 +00:00
// @BasePath /api/v1
// @securityDefinitions.apikey BearerAuth
// @in header
// @name Authorization
// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name X-API-Key
// @description Developer API key (obtain from Developer Portal). Format: vza_xxxxx
2025-12-03 19:29:37 +00:00
func main() {
// Charger les variables d'environnement
// NOTE: Do not write to stderr to avoid broken pipe errors with systemd journald
// The message will be logged by the logger once it's initialized
_ = godotenv.Load()
2025-12-03 19:29:37 +00:00
// FIX #1: Supprimer l'initialisation dupliquée du logger
// Le logger sera initialisé dans config.NewConfig() avec le bon LOG_LEVEL
// Charger la configuration (qui initialise le logger)
2025-12-03 19:29:37 +00:00
cfg, err := config.NewConfig()
if err != nil {
// CRITICAL: Do not write to stderr or files to avoid broken pipe errors
// Just exit silently - systemd will capture the exit code
// The error details will be in the application logs if the logger was initialized
os.Exit(1)
}
// Utiliser le logger de la config
logger := cfg.Logger
if logger == nil {
log.Fatal("❌ Logger non initialisé dans la configuration")
2025-12-03 19:29:37 +00:00
}
logger.Info("🚀 Démarrage de Veza Backend API")
2025-12-03 19:29:37 +00:00
// Valider la configuration
if err := cfg.Validate(); err != nil {
logger.Fatal("❌ Configuration invalide", zap.Error(err))
}
P0: stabilisation backend/chat/stream + nouvelle base migrations v1 Backend Go: - Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN. - Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError). - Sécurisation de config.go, CORS, statuts de santé et monitoring. - Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles). - Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés. - Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*. Chat server (Rust): - Refonte du pipeline JWT + sécurité, audit et rate limiting avancé. - Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing). - Nettoyage des panics, gestion d’erreurs robuste, logs structurés. - Migrations chat alignées sur le schéma UUID et nouvelles features. Stream server (Rust): - Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core. - Transactions P0 pour les jobs et segments, garanties d’atomicité. - Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION). Documentation & audits: - TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services. - Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3). - Scripts de reset et de cleanup pour la lab DB et la V1. Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
2025-12-06 10:14:38 +00:00
// Initialiser Sentry si DSN configuré
if cfg.SentryDsn != "" {
err := sentry.Init(sentry.ClientOptions{
Dsn: cfg.SentryDsn,
Environment: cfg.SentryEnvironment,
TracesSampleRate: cfg.SentrySampleRateTransactions,
SampleRate: cfg.SentrySampleRateErrors,
// AttachStacktrace pour capturer les stack traces
AttachStacktrace: true,
})
if err != nil {
logger.Warn("❌ Impossible d'initialiser Sentry", zap.Error(err))
} else {
logger.Info("✅ Sentry initialisé", zap.String("environment", cfg.SentryEnvironment))
}
// Flush les événements Sentry avant shutdown
defer sentry.Flush(2 * time.Second)
} else {
logger.Info(" Sentry non configuré (SENTRY_DSN non défini)")
}
2025-12-03 19:29:37 +00:00
// Initialisation de la base de données
db := cfg.Database
if db == nil {
logger.Fatal("❌ Base de données non initialisée")
}
defer db.Close()
if err := db.Initialize(); err != nil {
logger.Fatal("❌ Impossible d'initialiser la base de données", zap.Error(err))
}
2025-12-13 02:34:34 +00:00
// MOD-P2-004: Démarrer le collecteur de métriques DB pool
// Collecte les stats DB pool toutes les 10 secondes et les expose via Prometheus
metrics.StartDBPoolStatsCollector(db.DB, 10*time.Second)
logger.Info("✅ Collecteur de métriques DB pool démarré")
// Fail-Fast: Vérifier RabbitMQ si activé
if cfg.RabbitMQEnable {
if cfg.RabbitMQEventBus == nil {
logger.Fatal("❌ RabbitMQ activé (RABBITMQ_ENABLE=true) mais non initialisé (problème de connexion?)")
} else {
// Optionnel: Check connection status if RabbitMQEventBus exposes it
// For now, assume if initialized it's connected or retrying.
// If we want STRICT fail fast, we would need to verify connection is Open here.
logger.Info("✅ RabbitMQ actif")
}
} else {
logger.Info(" RabbitMQ désactivé")
}
// BE-SVC-017: Créer le gestionnaire de shutdown gracieux
shutdownManager := shutdown.NewShutdownManager(logger)
// Démarrer le Job Worker avec contexte pour shutdown gracieux
var workerCtx context.Context
var workerCancel context.CancelFunc
P0: stabilisation backend/chat/stream + nouvelle base migrations v1 Backend Go: - Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN. - Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError). - Sécurisation de config.go, CORS, statuts de santé et monitoring. - Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles). - Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés. - Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*. Chat server (Rust): - Refonte du pipeline JWT + sécurité, audit et rate limiting avancé. - Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing). - Nettoyage des panics, gestion d’erreurs robuste, logs structurés. - Migrations chat alignées sur le schéma UUID et nouvelles features. Stream server (Rust): - Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core. - Transactions P0 pour les jobs et segments, garanties d’atomicité. - Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION). Documentation & audits: - TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services. - Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3). - Scripts de reset et de cleanup pour la lab DB et la V1. Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
2025-12-06 10:14:38 +00:00
if cfg.JobWorker != nil {
workerCtx, workerCancel = context.WithCancel(context.Background())
P0: stabilisation backend/chat/stream + nouvelle base migrations v1 Backend Go: - Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN. - Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError). - Sécurisation de config.go, CORS, statuts de santé et monitoring. - Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles). - Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés. - Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*. Chat server (Rust): - Refonte du pipeline JWT + sécurité, audit et rate limiting avancé. - Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing). - Nettoyage des panics, gestion d’erreurs robuste, logs structurés. - Migrations chat alignées sur le schéma UUID et nouvelles features. Stream server (Rust): - Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core. - Transactions P0 pour les jobs et segments, garanties d’atomicité. - Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION). Documentation & audits: - TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services. - Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3). - Scripts de reset et de cleanup pour la lab DB et la V1. Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
2025-12-06 10:14:38 +00:00
cfg.JobWorker.Start(workerCtx)
logger.Info("✅ Job Worker démarré")
// Enregistrer le Job Worker pour shutdown gracieux
shutdownManager.Register(shutdown.NewShutdownFunc("job_worker", func(ctx context.Context) error {
if workerCancel != nil {
workerCancel()
// Attendre un peu pour que les workers se terminent
time.Sleep(2 * time.Second)
}
return nil
}))
P0: stabilisation backend/chat/stream + nouvelle base migrations v1 Backend Go: - Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN. - Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError). - Sécurisation de config.go, CORS, statuts de santé et monitoring. - Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles). - Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés. - Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*. Chat server (Rust): - Refonte du pipeline JWT + sécurité, audit et rate limiting avancé. - Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing). - Nettoyage des panics, gestion d’erreurs robuste, logs structurés. - Migrations chat alignées sur le schéma UUID et nouvelles features. Stream server (Rust): - Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core. - Transactions P0 pour les jobs et segments, garanties d’atomicité. - Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION). Documentation & audits: - TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services. - Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3). - Scripts de reset et de cleanup pour la lab DB et la V1. Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
2025-12-06 10:14:38 +00:00
} else {
logger.Warn("⚠️ Job Worker non initialisé")
}
// v0.701: Start Transfer Retry Worker
if cfg.TransferRetryEnabled && cfg.StripeConnectEnabled && cfg.StripeConnectSecretKey != "" {
stripeConnectSvc := services.NewStripeConnectService(db.GormDB, cfg.StripeConnectSecretKey, logger)
retryWorker := marketplace.NewTransferRetryWorker(
db.GormDB, stripeConnectSvc, logger, cfg.TransferRetryInterval, cfg.TransferRetryMaxAttempts,
)
retryCtx, retryCancel := context.WithCancel(context.Background())
go retryWorker.Start(retryCtx)
logger.Info("Transfer Retry Worker started",
zap.Duration("interval", cfg.TransferRetryInterval),
zap.Int("max_retries", cfg.TransferRetryMaxAttempts))
shutdownManager.Register(shutdown.NewShutdownFunc("transfer_retry_worker", func(ctx context.Context) error {
retryCancel()
return nil
}))
} else if cfg.TransferRetryEnabled {
logger.Info("Transfer Retry Worker skipped — Stripe Connect not enabled")
}
// v0.802: Start Cloud Backup Worker (copies cloud files to backup prefix every 24h)
if cfg.S3StorageService != nil {
backupWorker := services.NewCloudBackupWorker(db.GormDB, cfg.S3StorageService, logger)
backupCtx, backupCancel := context.WithCancel(context.Background())
go backupWorker.Start(backupCtx)
logger.Info("Cloud Backup Worker started (24h interval)")
shutdownManager.Register(shutdown.NewShutdownFunc("cloud_backup_worker", func(ctx context.Context) error {
backupCancel()
return nil
}))
}
// v0.802: Start Gear Warranty Notifier (sends notifications when warranty expires in 30 days)
notificationService := services.NewNotificationService(db, logger)
warrantyNotifier := services.NewGearWarrantyNotifier(db.GormDB, notificationService, logger)
warrantyCtx, warrantyCancel := context.WithCancel(context.Background())
go warrantyNotifier.Start(warrantyCtx)
logger.Info("Gear Warranty Notifier started (24h interval)")
shutdownManager.Register(shutdown.NewShutdownFunc("gear_warranty_notifier", func(ctx context.Context) error {
warrantyCancel()
return nil
}))
// v0.10.5 F552: Weekly notification digest (runs on Sunday)
if cfg.JobWorker != nil {
digestWorker := services.NewNotificationDigestWorker(db.GormDB, cfg.JobWorker, logger)
digestCtx, digestCancel := context.WithCancel(context.Background())
go digestWorker.Start(digestCtx)
logger.Info("Notification digest worker started (weekly on Sunday)")
shutdownManager.Register(shutdown.NewShutdownFunc("notification_digest_worker", func(ctx context.Context) error {
digestCancel()
return nil
}))
}
// v0.10.8 F065: Hard delete worker (GDPR - final anonymization after 30 days)
if os.Getenv("HARD_DELETE_CRON_ENABLED") != "false" {
fix(backend): J4 — GDPR-compliant hard delete with Redis and ES cleanup Closes TODO(HIGH-007). When the hard-delete worker anonymizes a user past their recovery deadline, it now also cleans the user's residual data from Redis and Elasticsearch, not just PostgreSQL. Without this, a user who invoked their right to erasure would still appear in cached feed/profile responses and in ES search results for up to the next reindex cycle. Worker changes (internal/workers/hard_delete_worker.go): WithRedis / WithElasticsearch builder methods inject the clients. Both are optional: if either is nil (feature disabled or unreachable), the corresponding cleanup is skipped with a debug log and the worker keeps going. Partial progress beats panic. cleanRedisKeys uses SCAN with a cursor loop (COUNT 100), NEVER KEYS — KEYS would block the Redis server on multi-million-key deployments. Pattern is user:{id}:*. Transient SCAN errors retry up to 3 times with 100ms * retry linear backoff; persistent errors return without panic. DEL errors on a batch are logged but non-fatal so subsequent batches are still attempted. cleanESDocs hits three indices independently: - users index: DELETE doc by _id (the user UUID); 404 treated as success (already gone = desired state) - tracks index: DeleteByQuery with a terms filter on _id, using the list of track IDs collected from PostgreSQL BEFORE anonymization - playlists index: same pattern as tracks A failure on one index does not prevent the others from being tried; the first error is returned so the caller can log. Track/playlist IDs are pre-collected (collectTrackIDs, collectPlaylistIDs) before the UPDATE anonymization runs, because the anonymization does NOT cascade (no DELETE on users), so tracks and playlists rows remain with their creator_id / user_id intact and resolvable at query time. Wiring (cmd/api/main.go): The worker now receives cfg.RedisClient directly, and an optional ES client built from elasticsearch.LoadConfig() + NewClient. If ES is disabled or unreachable at startup, the worker logs a warning and proceeds with Redis-only cleanup. Tests (internal/workers/hard_delete_worker_test.go, +260 lines): Pure-function unit tests: - TestUUIDsToStrings - TestEsIndexNameFor Nil-client safety tests: - TestCleanRedisKeys_NilClientIsNoop - TestCleanESDocs_NilClientIsNoop ES mock-server tests (httptest.Server mimicking /_doc and /_delete_by_query endpoints with valid ES 8.11 responses): - TestCleanESDocs_CallsAllThreeIndices — verifies the three expected HTTP calls land with the right paths and request bodies containing the provided UUIDs - TestCleanESDocs_SkipsEmptyIDLists — verifies no DeleteByQuery is issued when the ID lists are empty Redis testcontainer integration test (gated by VEZA_SKIP_INTEGRATION): - TestCleanRedisKeys_Integration — seeds 154 keys (4 fixed + 150 bulk to force the SCAN loop past a single batch) plus 4 unrelated keys from another user / global, runs cleanRedisKeys, asserts all 154 own keys are gone and all 4 unrelated keys remain. Verification: go build ./... OK go vet ./... OK VEZA_SKIP_INTEGRATION=1 go test ./internal/workers/... short OK go test ./internal/workers/ -run TestCleanRedisKeys_Integration → testcontainers spins redis:7-alpine, test passes in 1.34s Out of J4 scope (noted for a follow-up): - No "activity" ES index exists in the codebase today (the audit plan mentioned it as a possible target). The three real indices with user data — users, tracks, playlists — are all now cleaned. - Track artist strings (free-form) may still contain the user's display name as a cached value in the tracks index after this cleanup. Actual user-owned tracks are deleted here, but if a third party's track referenced the removed user in its artist field, that reference is not touched. Strict RGPD on that edge case is a separate ticket. Refs: AUDIT_REPORT.md §8.5, §10 P5, §12 item 1
2026-04-15 10:25:39 +00:00
// Optional ES client for the worker's RGPD cleanup (users/tracks/playlists indices).
// Non-fatal if ES is disabled or unreachable — the worker will just skip ES cleanup.
var hardDeleteESClient *vezaes.Client
if esCfg := vezaes.LoadConfig(); esCfg.Enabled {
if esc, esErr := vezaes.NewClient(esCfg, logger); esErr != nil {
logger.Warn("Elasticsearch unavailable for hard delete worker, ES cleanup disabled",
zap.Error(esErr))
} else {
hardDeleteESClient = esc
}
}
hardDeleteWorker := workers.NewHardDeleteWorker(db.GormDB, logger, 24*time.Hour).
WithRedis(cfg.RedisClient).
WithElasticsearch(hardDeleteESClient)
hardDeleteCtx, hardDeleteCancel := context.WithCancel(context.Background())
go hardDeleteWorker.Start(hardDeleteCtx)
fix(backend): J4 — GDPR-compliant hard delete with Redis and ES cleanup Closes TODO(HIGH-007). When the hard-delete worker anonymizes a user past their recovery deadline, it now also cleans the user's residual data from Redis and Elasticsearch, not just PostgreSQL. Without this, a user who invoked their right to erasure would still appear in cached feed/profile responses and in ES search results for up to the next reindex cycle. Worker changes (internal/workers/hard_delete_worker.go): WithRedis / WithElasticsearch builder methods inject the clients. Both are optional: if either is nil (feature disabled or unreachable), the corresponding cleanup is skipped with a debug log and the worker keeps going. Partial progress beats panic. cleanRedisKeys uses SCAN with a cursor loop (COUNT 100), NEVER KEYS — KEYS would block the Redis server on multi-million-key deployments. Pattern is user:{id}:*. Transient SCAN errors retry up to 3 times with 100ms * retry linear backoff; persistent errors return without panic. DEL errors on a batch are logged but non-fatal so subsequent batches are still attempted. cleanESDocs hits three indices independently: - users index: DELETE doc by _id (the user UUID); 404 treated as success (already gone = desired state) - tracks index: DeleteByQuery with a terms filter on _id, using the list of track IDs collected from PostgreSQL BEFORE anonymization - playlists index: same pattern as tracks A failure on one index does not prevent the others from being tried; the first error is returned so the caller can log. Track/playlist IDs are pre-collected (collectTrackIDs, collectPlaylistIDs) before the UPDATE anonymization runs, because the anonymization does NOT cascade (no DELETE on users), so tracks and playlists rows remain with their creator_id / user_id intact and resolvable at query time. Wiring (cmd/api/main.go): The worker now receives cfg.RedisClient directly, and an optional ES client built from elasticsearch.LoadConfig() + NewClient. If ES is disabled or unreachable at startup, the worker logs a warning and proceeds with Redis-only cleanup. Tests (internal/workers/hard_delete_worker_test.go, +260 lines): Pure-function unit tests: - TestUUIDsToStrings - TestEsIndexNameFor Nil-client safety tests: - TestCleanRedisKeys_NilClientIsNoop - TestCleanESDocs_NilClientIsNoop ES mock-server tests (httptest.Server mimicking /_doc and /_delete_by_query endpoints with valid ES 8.11 responses): - TestCleanESDocs_CallsAllThreeIndices — verifies the three expected HTTP calls land with the right paths and request bodies containing the provided UUIDs - TestCleanESDocs_SkipsEmptyIDLists — verifies no DeleteByQuery is issued when the ID lists are empty Redis testcontainer integration test (gated by VEZA_SKIP_INTEGRATION): - TestCleanRedisKeys_Integration — seeds 154 keys (4 fixed + 150 bulk to force the SCAN loop past a single batch) plus 4 unrelated keys from another user / global, runs cleanRedisKeys, asserts all 154 own keys are gone and all 4 unrelated keys remain. Verification: go build ./... OK go vet ./... OK VEZA_SKIP_INTEGRATION=1 go test ./internal/workers/... short OK go test ./internal/workers/ -run TestCleanRedisKeys_Integration → testcontainers spins redis:7-alpine, test passes in 1.34s Out of J4 scope (noted for a follow-up): - No "activity" ES index exists in the codebase today (the audit plan mentioned it as a possible target). The three real indices with user data — users, tracks, playlists — are all now cleaned. - Track artist strings (free-form) may still contain the user's display name as a cached value in the tracks index after this cleanup. Actual user-owned tracks are deleted here, but if a third party's track referenced the removed user in its artist field, that reference is not touched. Strict RGPD on that edge case is a separate ticket. Refs: AUDIT_REPORT.md §8.5, §10 P5, §12 item 1
2026-04-15 10:25:39 +00:00
logger.Info("Hard delete worker started (24h interval)",
zap.Bool("redis_cleanup", cfg.RedisClient != nil),
zap.Bool("es_cleanup", hardDeleteESClient != nil),
)
shutdownManager.Register(shutdown.NewShutdownFunc("hard_delete_worker", func(ctx context.Context) error {
hardDeleteWorker.Stop()
hardDeleteCancel()
return nil
}))
} else {
logger.Info("Hard delete worker disabled (HARD_DELETE_CRON_ENABLED=false)")
}
2025-12-03 19:29:37 +00:00
// Configuration du mode Gin
// Correction: Utilisation directe de la variable d'env car non exposée dans Config
appEnv := os.Getenv("APP_ENV")
if appEnv == "production" {
gin.SetMode(gin.ReleaseMode)
} else {
gin.SetMode(gin.DebugMode)
}
// Créer le router Gin
router := gin.New()
fix(v0.12.6.1): remediate 2 CRITICAL + 10 HIGH + 1 MEDIUM pentest findings Security fixes implemented: CRITICAL: - CRIT-001: IDOR on chat rooms — added IsRoomMember check before returning room data or message history (returns 404, not 403) - CRIT-002: play_count/like_count exposed publicly — changed JSON tags to "-" so they are never serialized in API responses HIGH: - HIGH-001: TOCTOU race on marketplace downloads — transaction + SELECT FOR UPDATE on GetDownloadURL - HIGH-002: HS256 in production docker-compose — replaced JWT_SECRET with JWT_PRIVATE_KEY_PATH / JWT_PUBLIC_KEY_PATH (RS256) - HIGH-003: context.Background() bypass in user repository — full context propagation from handlers → services → repository (29 files) - HIGH-004: Race condition on promo codes — SELECT FOR UPDATE - HIGH-005: Race condition on exclusive licenses — SELECT FOR UPDATE - HIGH-006: Rate limiter IP spoofing — SetTrustedProxies(nil) default - HIGH-007: RGPD hard delete incomplete — added cleanup for sessions, settings, follows, notifications, audit_logs anonymization - HIGH-008: RTMP callback auth weak — fail-closed when unconfigured, header-only (no query param), constant-time compare - HIGH-009: Co-listening host hijack — UpdateHostState now takes *Conn and verifies IsHost before processing - HIGH-010: Moderator self-strike — added issuedBy != userID check MEDIUM: - MEDIUM-001: Recovery codes used math/rand — replaced with crypto/rand - MEDIUM-005: Stream token forgeable — resolved by HIGH-002 (RS256) Updated REMEDIATION_MATRIX: 14 findings marked ✅ CORRIGÉ. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 04:40:53 +00:00
// SECURITY(HIGH-006): Restrict trusted proxies to prevent IP spoofing via X-Forwarded-For.
// Default: trust nothing (c.ClientIP() returns RemoteAddr only).
// Set TRUSTED_PROXIES="10.0.0.1,10.0.0.2" if behind a known reverse proxy/load balancer.
router.SetTrustedProxies(nil)
2025-12-03 19:29:37 +00:00
// Middleware globaux (Logger, Recovery) recommandés par ORIGIN
router.Use(gin.Logger(), gin.Recovery())
// Configuration des routes
apiRouter := api.NewAPIRouter(db, cfg) // Instantiate APIRouter
if err := apiRouter.Setup(router); err != nil {
logger.Error("Failed to setup API routes", zap.Error(err))
os.Exit(1)
}
2025-12-03 19:29:37 +00:00
// v1.0.4: Hourly cleanup of tracks stuck in `processing` whose upload file
// vanished (crash, SIGKILL, disk wipe). Keeps the tracks table honest.
jobs.ScheduleOrphanTracksCleanup(db, logger)
2025-12-03 19:29:37 +00:00
// Configuration du serveur HTTP
port := fmt.Sprintf("%d", cfg.AppPort)
if cfg.AppPort == 0 {
port = "8080"
}
server := &http.Server{
Addr: fmt.Sprintf(":%s", port),
Handler: router,
ReadTimeout: 30 * time.Second, // Standards ORIGIN
WriteTimeout: 30 * time.Second,
}
// BE-SVC-017: Enregistrer tous les services pour shutdown gracieux
// Enregistrer le serveur HTTP
shutdownManager.Register(shutdown.NewShutdownFunc("http_server", func(ctx context.Context) error {
return server.Shutdown(ctx)
}))
// Enregistrer la configuration (ferme DB, Redis, RabbitMQ, etc.)
shutdownManager.Register(shutdown.NewShutdownFunc("config", func(ctx context.Context) error {
return cfg.Close()
}))
// Enregistrer le logger pour flush final
shutdownManager.Register(shutdown.NewShutdownFunc("logger", func(ctx context.Context) error {
if logger != nil {
return logger.Sync()
}
return nil
}))
// Enregistrer Sentry pour flush final
if cfg.SentryDsn != "" {
shutdownManager.Register(shutdown.NewShutdownFunc("sentry", func(ctx context.Context) error {
sentry.Flush(2 * time.Second)
return nil
}))
}
2025-12-03 19:29:37 +00:00
// Gestion de l'arrêt gracieux
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
go func() {
logger.Info("🌐 Serveur HTTP démarré", zap.String("port", port))
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
logger.Fatal("❌ Erreur du serveur HTTP", zap.Error(err))
}
}()
// Attendre le signal d'arrêt
2025-12-03 19:29:37 +00:00
<-quit
logger.Info("🔄 Signal d'arrêt reçu, démarrage du shutdown gracieux...")
2025-12-03 19:29:37 +00:00
// BE-SVC-017: Arrêt gracieux coordonné de tous les services
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
2025-12-03 19:29:37 +00:00
if err := shutdownManager.Shutdown(shutdownCtx); err != nil {
logger.Error("❌ Erreur lors du shutdown gracieux", zap.Error(err))
2025-12-03 19:29:37 +00:00
} else {
logger.Info("✅ Shutdown gracieux terminé avec succès")
2025-12-03 19:29:37 +00:00
}
}