Closes TODO(HIGH-007). When the hard-delete worker anonymizes a user past
their recovery deadline, it now also cleans the user's residual data from
Redis and Elasticsearch, not just PostgreSQL. Without this, a user who
invoked their right to erasure would still appear in cached feed/profile
responses and in ES search results for up to the next reindex cycle.
Worker changes (internal/workers/hard_delete_worker.go):
WithRedis / WithElasticsearch builder methods inject the clients. Both
are optional: if either is nil (feature disabled or unreachable), the
corresponding cleanup is skipped with a debug log and the worker keeps
going. Partial progress beats panic.
cleanRedisKeys uses SCAN with a cursor loop (COUNT 100), NEVER KEYS —
KEYS would block the Redis server on multi-million-key deployments.
Pattern is user:{id}:*. Transient SCAN errors retry up to 3 times with
100ms * retry linear backoff; persistent errors return without panic.
DEL errors on a batch are logged but non-fatal so subsequent batches
are still attempted.
cleanESDocs hits three indices independently:
- users index: DELETE doc by _id (the user UUID); 404 treated as
success (already gone = desired state)
- tracks index: DeleteByQuery with a terms filter on _id, using the
list of track IDs collected from PostgreSQL BEFORE anonymization
- playlists index: same pattern as tracks
A failure on one index does not prevent the others from being tried;
the first error is returned so the caller can log.
Track/playlist IDs are pre-collected (collectTrackIDs, collectPlaylistIDs)
before the UPDATE anonymization runs, because the anonymization does NOT
cascade (no DELETE on users), so tracks and playlists rows remain with
their creator_id / user_id intact and resolvable at query time.
Wiring (cmd/api/main.go):
The worker now receives cfg.RedisClient directly, and an optional ES
client built from elasticsearch.LoadConfig() + NewClient. If ES is
disabled or unreachable at startup, the worker logs a warning and
proceeds with Redis-only cleanup.
Tests (internal/workers/hard_delete_worker_test.go, +260 lines):
Pure-function unit tests:
- TestUUIDsToStrings
- TestEsIndexNameFor
Nil-client safety tests:
- TestCleanRedisKeys_NilClientIsNoop
- TestCleanESDocs_NilClientIsNoop
ES mock-server tests (httptest.Server mimicking /_doc and
/_delete_by_query endpoints with valid ES 8.11 responses):
- TestCleanESDocs_CallsAllThreeIndices — verifies the three expected
HTTP calls land with the right paths and request bodies containing
the provided UUIDs
- TestCleanESDocs_SkipsEmptyIDLists — verifies no DeleteByQuery is
issued when the ID lists are empty
Redis testcontainer integration test (gated by VEZA_SKIP_INTEGRATION):
- TestCleanRedisKeys_Integration — seeds 154 keys (4 fixed + 150 bulk
to force the SCAN loop past a single batch) plus 4 unrelated keys
from another user / global, runs cleanRedisKeys, asserts all 154
own keys are gone and all 4 unrelated keys remain.
Verification:
go build ./... OK
go vet ./... OK
VEZA_SKIP_INTEGRATION=1 go test ./internal/workers/... short OK
go test ./internal/workers/ -run TestCleanRedisKeys_Integration
→ testcontainers spins redis:7-alpine, test passes in 1.34s
Out of J4 scope (noted for a follow-up):
- No "activity" ES index exists in the codebase today (the audit plan
mentioned it as a possible target). The three real indices with user
data — users, tracks, playlists — are all now cleaned.
- Track artist strings (free-form) may still contain the user's
display name as a cached value in the tracks index after this
cleanup. Actual user-owned tracks are deleted here, but if a third
party's track referenced the removed user in its artist field, that
reference is not touched. Strict RGPD on that edge case is a
separate ticket.
Refs: AUDIT_REPORT.md §8.5, §10 P5, §12 item 1
golangci-lint v2.11.4 requires Go >= 1.25. With the workflow on 1.24,
setup-go would silently trigger an in-job auto-toolchain download
(observed in run #71: 'go: github.com/golangci/golangci-lint/v2@v2.11.4
requires go >= 1.25.0; switching to go1.25.9') adding ~3 min to every
Backend (Go) run.
Bump setup-go to 1.25 in ci.yml, backend-ci.yml, go-fuzz.yml so the
prebuilt Go is already the right version.
Also lint-fix three files that golangci-lint's goimports checker
flagged — goimports sorts/groups imports and removes unused ones,
which plain gofmt leaves alone:
- veza-backend-api/cmd/api/main.go
- veza-backend-api/internal/api/handlers/chat_handlers.go
- veza-backend-api/internal/handlers/auth_integration_test.go
backend-ci.yml's `test -z "$(gofmt -l .)"` strict gate (added in
13c21ac11) failed on a backlog of unformatted files. None of the
85 files in this commit had been edited since the gate was added
because no push touched veza-backend-api/** in between, so the
gate never fired until today's CI fixes triggered it.
The diff is exclusively whitespace alignment in struct literals
and trailing-space comments. `go build ./...` and the full test
suite (with VEZA_SKIP_INTEGRATION=1 -short) pass identically.
Extract monolithic seed main.go into separate files per domain:
users, tracks, playlists, chat, analytics, marketplace, social,
content, live, moderation, notifications, and misc. Add config,
fake data helpers, and utility modules. Update Makefile targets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update RabbitMQ config and eventbus. Improve secret filter logging.
Refine presence, cloud, and social services. Update announcement and
feature flag handlers. Add track_likes updated_at migration. Rebuild
seed binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- API key rate limiting middleware (1000 reads/h, 200 writes/h par clé)
— tracking séparé read/write, par API key ID (pas par IP)
— headers X-RateLimit-Limit/Remaining/Reset sur chaque réponse
- API key scope enforcement middleware (read → GET, write → POST/PUT/DELETE)
— admin scope permet tout, CSRF skip pour API key auth
- OpenAPI spec: ajout securityDefinition ApiKeyAuth (X-API-Key header)
- Swagger annotations: ajout ApiKeyAuth dans cmd/api/main.go
- Wiring dans router.go: middlewares appliqués sur tout le groupe /api/v1
- Tests: 10 tests (5 rate limiter + 5 scope enforcement), tous PASS
Backend existant déjà en place (pré-v0.12.8):
- Swagger UI (gin-swagger + frontend SwaggerUIDoc component)
- API key CRUD (create/list/delete + X-API-Key auth dans AuthMiddleware)
- Developer Dashboard frontend (API keys, webhooks, playground)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Export: table data_exports, POST /me/export (202), GET /me/exports, messages+playback_history
- Notification email quand ZIP prêt, rate limit 3/jour
- Suppression: keep_public_tracks, anonymisation PII complète (users, user_profiles)
- HardDeleteWorker: final anonymization après 30 jours
- Frontend: POST export, checkbox keep_public_tracks
- MSW handlers pour Storybook
- PKCE (S256) in OAuth flow: code_verifier in oauth_states, code_challenge in auth URL
- CryptoService: AES-256-GCM encryption for OAuth provider tokens at rest
- OAuth redirect URL validated against OAUTH_ALLOWED_REDIRECT_DOMAINS
- CHAT_JWT_SECRET must differ from JWT_SECRET in production
- Migration script: cmd/tools/encrypt_oauth_tokens for existing tokens
- Fixes: VEZA-SEC-003, VEZA-SEC-004, VEZA-SEC-009, VEZA-SEC-010
- Add early validation in Setup() returning error if Redis nil in production
- Remove panic/Fatal from routes_core.go and router.go applyCSRFProtection
- Handle Setup() error in cmd/api/main.go and cmd/modern-server/main.go
- Mark audit item 1.4 as done
- Deleted apps/web/src/utils/optimisticStoreUpdates.ts (unused file)
- File was unused - no imports found in codebase
- Mutations already use React Query's onMutate pattern
- No TypeScript errors after deletion
- Actions 4.4.1.2 and 4.4.1.3 complete
- Created ShutdownManager for coordinated graceful shutdown of all services
- Added Shutdowner interface for services that need graceful shutdown
- Implemented parallel shutdown with individual timeouts (10s per service)
- Added global shutdown timeout (30s total)
- Integrated shutdown manager in main.go for:
- HTTP server shutdown
- JobWorker cancellation
- Config.Close() (DB, Redis, RabbitMQ)
- Logger sync
- Sentry flush
- Added comprehensive unit tests for shutdown manager
- Prevents registration of new services during shutdown
Phase: PHASE-6
Priority: P2
Progress: 113/267 (42.32%)
Backend Go:
- Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN.
- Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError).
- Sécurisation de config.go, CORS, statuts de santé et monitoring.
- Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles).
- Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés.
- Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*.
Chat server (Rust):
- Refonte du pipeline JWT + sécurité, audit et rate limiting avancé.
- Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing).
- Nettoyage des panics, gestion d’erreurs robuste, logs structurés.
- Migrations chat alignées sur le schéma UUID et nouvelles features.
Stream server (Rust):
- Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core.
- Transactions P0 pour les jobs et segments, garanties d’atomicité.
- Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION).
Documentation & audits:
- TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services.
- Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3).
- Scripts de reset et de cleanup pour la lab DB et la V1.
Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
