Closes the transcoder's read-side gap for Phase 2. HLS transcoding now
works for tracks uploaded under TRACK_STORAGE_BACKEND=s3 without
requiring the stream server pod to share a local volume.
Changes:
- internal/services/hls_transcode_service.go
- New SignedURLProvider interface (minimal: GetSignedURL).
- HLSTranscodeService gains optional s3Resolver + SetS3Resolver.
- TranscodeTrack routed through new resolveSource helper — returns
local FilePath for local tracks, a 1h-TTL signed URL for s3-backed
rows. Missing resolver for an s3 track returns a clear error.
- os.Stat check skipped for HTTP(S) sources (ffmpeg validates them).
- transcodeBitrate takes `source` explicitly so URL propagation is
obvious and ValidateExecPath is bypassed only for the known
signed-URL shape.
- isHTTPSource helper (http://, https:// prefix check).
- internal/workers/job_worker.go
- JobWorker gains optional s3Resolver + SetS3Resolver.
- processTranscodingJob skips the local-file stat when
track.StorageBackend='s3', reads via signed URL instead.
- Passes w.s3Resolver to NewHLSTranscodeService when non-nil.
- internal/config/config.go: DI wires S3StorageService into JobWorker
after instantiation (nil-safe).
- internal/core/track/service.go (copyFileAsyncS3)
- Re-enabled stream server trigger: generates a 1h-TTL signed URL
for the fresh s3 key and passes it to streamService.StartProcessing.
Rust-side ffmpeg consumes HTTPS URLs natively. Failure is logged
but does not fail the upload (track will sit in Processing until
a retry / reconcile).
- internal/core/track/track_upload_handler.go (CompleteChunkedUpload)
- Reload track after S3 migration to pick up the new storage_key.
- Compute transcodeSource = signed URL (s3 path) or finalPath (local).
- Pass transcodeSource to both streamService.StartProcessing and
jobEnqueuer.EnqueueTranscodingJob — dual-trigger preserved per
plan D2 (consolidation deferred v1.0.9).
- internal/services/hls_transcode_service_test.go
- TestHLSTranscodeService_TranscodeTrack_EmptyFilePath updated for
the expanded error message ("empty FilePath" vs "file path is empty").
Known limitation (v1.0.9): HLS segment OUTPUT still writes to the
local outputDir; only the INPUT side is S3-aware. Multi-pod HLS serving
needs the worker to upload segments to MinIO post-transcode. Acceptable
for v1.0.8 target — single-pod staging supports both local + s3 tracks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes TODO(HIGH-007). When the hard-delete worker anonymizes a user past
their recovery deadline, it now also cleans the user's residual data from
Redis and Elasticsearch, not just PostgreSQL. Without this, a user who
invoked their right to erasure would still appear in cached feed/profile
responses and in ES search results for up to the next reindex cycle.
Worker changes (internal/workers/hard_delete_worker.go):
WithRedis / WithElasticsearch builder methods inject the clients. Both
are optional: if either is nil (feature disabled or unreachable), the
corresponding cleanup is skipped with a debug log and the worker keeps
going. Partial progress beats panic.
cleanRedisKeys uses SCAN with a cursor loop (COUNT 100), NEVER KEYS —
KEYS would block the Redis server on multi-million-key deployments.
Pattern is user:{id}:*. Transient SCAN errors retry up to 3 times with
100ms * retry linear backoff; persistent errors return without panic.
DEL errors on a batch are logged but non-fatal so subsequent batches
are still attempted.
cleanESDocs hits three indices independently:
- users index: DELETE doc by _id (the user UUID); 404 treated as
success (already gone = desired state)
- tracks index: DeleteByQuery with a terms filter on _id, using the
list of track IDs collected from PostgreSQL BEFORE anonymization
- playlists index: same pattern as tracks
A failure on one index does not prevent the others from being tried;
the first error is returned so the caller can log.
Track/playlist IDs are pre-collected (collectTrackIDs, collectPlaylistIDs)
before the UPDATE anonymization runs, because the anonymization does NOT
cascade (no DELETE on users), so tracks and playlists rows remain with
their creator_id / user_id intact and resolvable at query time.
Wiring (cmd/api/main.go):
The worker now receives cfg.RedisClient directly, and an optional ES
client built from elasticsearch.LoadConfig() + NewClient. If ES is
disabled or unreachable at startup, the worker logs a warning and
proceeds with Redis-only cleanup.
Tests (internal/workers/hard_delete_worker_test.go, +260 lines):
Pure-function unit tests:
- TestUUIDsToStrings
- TestEsIndexNameFor
Nil-client safety tests:
- TestCleanRedisKeys_NilClientIsNoop
- TestCleanESDocs_NilClientIsNoop
ES mock-server tests (httptest.Server mimicking /_doc and
/_delete_by_query endpoints with valid ES 8.11 responses):
- TestCleanESDocs_CallsAllThreeIndices — verifies the three expected
HTTP calls land with the right paths and request bodies containing
the provided UUIDs
- TestCleanESDocs_SkipsEmptyIDLists — verifies no DeleteByQuery is
issued when the ID lists are empty
Redis testcontainer integration test (gated by VEZA_SKIP_INTEGRATION):
- TestCleanRedisKeys_Integration — seeds 154 keys (4 fixed + 150 bulk
to force the SCAN loop past a single batch) plus 4 unrelated keys
from another user / global, runs cleanRedisKeys, asserts all 154
own keys are gone and all 4 unrelated keys remain.
Verification:
go build ./... OK
go vet ./... OK
VEZA_SKIP_INTEGRATION=1 go test ./internal/workers/... short OK
go test ./internal/workers/ -run TestCleanRedisKeys_Integration
→ testcontainers spins redis:7-alpine, test passes in 1.34s
Out of J4 scope (noted for a follow-up):
- No "activity" ES index exists in the codebase today (the audit plan
mentioned it as a possible target). The three real indices with user
data — users, tracks, playlists — are all now cleaned.
- Track artist strings (free-form) may still contain the user's
display name as a cached value in the tracks index after this
cleanup. Actual user-owned tracks are deleted here, but if a third
party's track referenced the removed user in its artist field, that
reference is not touched. Strict RGPD on that edge case is a
separate ticket.
Refs: AUDIT_REPORT.md §8.5, §10 P5, §12 item 1
- Export: table data_exports, POST /me/export (202), GET /me/exports, messages+playback_history
- Notification email quand ZIP prêt, rate limit 3/jour
- Suppression: keep_public_tracks, anonymisation PII complète (users, user_profiles)
- HardDeleteWorker: final anonymization après 30 jours
- Frontend: POST export, checkbox keep_public_tracks
- MSW handlers pour Storybook
Backend Go:
- Remplacement complet des anciennes migrations par la base V1 alignée sur ORIGIN.
- Durcissement global du parsing JSON (BindAndValidateJSON + RespondWithAppError).
- Sécurisation de config.go, CORS, statuts de santé et monitoring.
- Implémentation des transactions P0 (RBAC, duplication de playlists, social toggles).
- Ajout d’un job worker structuré (emails, analytics, thumbnails) + tests associés.
- Nouvelle doc backend : AUDIT_CONFIG, BACKEND_CONFIG, AUTH_PASSWORD_RESET, JOB_WORKER_*.
Chat server (Rust):
- Refonte du pipeline JWT + sécurité, audit et rate limiting avancé.
- Implémentation complète du cycle de message (read receipts, delivered, edit/delete, typing).
- Nettoyage des panics, gestion d’erreurs robuste, logs structurés.
- Migrations chat alignées sur le schéma UUID et nouvelles features.
Stream server (Rust):
- Refonte du moteur de streaming (encoding pipeline + HLS) et des modules core.
- Transactions P0 pour les jobs et segments, garanties d’atomicité.
- Documentation détaillée de la pipeline (AUDIT_STREAM_*, DESIGN_STREAM_PIPELINE, TRANSACTIONS_P0_IMPLEMENTATION).
Documentation & audits:
- TRIAGE.md et AUDIT_STABILITY.md à jour avec l’état réel des 3 services.
- Cartographie complète des migrations et des transactions (DB_MIGRATIONS_*, DB_TRANSACTION_PLAN, AUDIT_DB_TRANSACTIONS, TRANSACTION_TESTS_PHASE3).
- Scripts de reset et de cleanup pour la lab DB et la V1.
Ce commit fige l’ensemble du travail de stabilisation P0 (UUID, backend, chat et stream) avant les phases suivantes (Coherence Guardian, WS hardening, etc.).
