Replaces the historical chunked-upload flow when TRACK_STORAGE_BACKEND=s3:
before: chunks → assembled file on disk → MigrateLocalToS3IfConfigured
opens the file → manager.Uploader streams in 10 MB parts
after: chunks → io.Pipe → manager.Uploader streams in 10 MB parts
(no assembled file on local disk)
Eliminates the second local copy of every upload and ~500 MB of disk
I/O per concurrent 500 MB upload. The local-storage path
(TRACK_STORAGE_BACKEND=local, default) is unchanged — it still goes
through CompleteChunkedUpload + CreateTrackFromPath because ClamAV needs
the assembled file (chunked path skips ClamAV by design, see audit).
New surface:
- TrackChunkService.StreamChunkedUpload(ctx, uploadID, dst io.Writer)
— extracted from CompleteChunkedUpload, writes chunks in order to
any io.Writer, computes SHA-256 + verifies expected size, cleans
up Redis state on success and preserves it on failure (resumable).
- TrackService.CreateTrackFromChunkedUploadToS3 — orchestrates
io.Pipe + goroutine, deletes orphan S3 objects on assembly failure,
creates the Track row with storage_backend=s3 + storage_key.
Tests: 4 chunk-service stream tests (happy / writer error / size
mismatch / delegation) + 4 service tests (happy / wrong backend /
stream error / S3 upload error). One E2E @critical-s3 spec gated on
S3 availability via /health/deep so it ships today and starts running
once MinIO is added to the e2e workflow services block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the transcoder's read-side gap for Phase 2. HLS transcoding now
works for tracks uploaded under TRACK_STORAGE_BACKEND=s3 without
requiring the stream server pod to share a local volume.
Changes:
- internal/services/hls_transcode_service.go
- New SignedURLProvider interface (minimal: GetSignedURL).
- HLSTranscodeService gains optional s3Resolver + SetS3Resolver.
- TranscodeTrack routed through new resolveSource helper — returns
local FilePath for local tracks, a 1h-TTL signed URL for s3-backed
rows. Missing resolver for an s3 track returns a clear error.
- os.Stat check skipped for HTTP(S) sources (ffmpeg validates them).
- transcodeBitrate takes `source` explicitly so URL propagation is
obvious and ValidateExecPath is bypassed only for the known
signed-URL shape.
- isHTTPSource helper (http://, https:// prefix check).
- internal/workers/job_worker.go
- JobWorker gains optional s3Resolver + SetS3Resolver.
- processTranscodingJob skips the local-file stat when
track.StorageBackend='s3', reads via signed URL instead.
- Passes w.s3Resolver to NewHLSTranscodeService when non-nil.
- internal/config/config.go: DI wires S3StorageService into JobWorker
after instantiation (nil-safe).
- internal/core/track/service.go (copyFileAsyncS3)
- Re-enabled stream server trigger: generates a 1h-TTL signed URL
for the fresh s3 key and passes it to streamService.StartProcessing.
Rust-side ffmpeg consumes HTTPS URLs natively. Failure is logged
but does not fail the upload (track will sit in Processing until
a retry / reconcile).
- internal/core/track/track_upload_handler.go (CompleteChunkedUpload)
- Reload track after S3 migration to pick up the new storage_key.
- Compute transcodeSource = signed URL (s3 path) or finalPath (local).
- Pass transcodeSource to both streamService.StartProcessing and
jobEnqueuer.EnqueueTranscodingJob — dual-trigger preserved per
plan D2 (consolidation deferred v1.0.9).
- internal/services/hls_transcode_service_test.go
- TestHLSTranscodeService_TranscodeTrack_EmptyFilePath updated for
the expanded error message ("empty FilePath" vs "file path is empty").
Known limitation (v1.0.9): HLS segment OUTPUT still writes to the
local outputDir; only the INPUT side is S3-aware. Multi-pod HLS serving
needs the worker to upload segments to MinIO post-transcode. Acceptable
for v1.0.8 target — single-pod staging supports both local + s3 tracks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the read-side gap for Phase 1 uploads. Tracks with
storage_backend='s3' now get a 302 redirect to a MinIO signed URL
from /stream and /download, letting the client fetch bytes directly
without the backend proxying. Range headers remain honored by MinIO.
Changes:
- internal/core/track/service.go
- New method `TrackService.GetStorageURL(ctx, track, ttl)` returns
(url, isS3, err). Empty + false for local-backed tracks (caller
falls back to FS). Returns a presigned URL with caller-chosen TTL
for s3-backed rows.
- Defensive: storage_backend='s3' with nil storage_key returns
(empty, false, nil) — treated as legacy/broken, falls back to FS
rather than crashing the request.
- Errors when row claims s3 but TrackService has no S3 wired
(should be prevented by Config validation rule 11).
- internal/core/track/track_hls_handler.go
- `StreamTrack`: tries GetStorageURL(ctx, track, 15*time.Minute)
before opening the local file. On s3 hit → 302 redirect. TTL 15min
fits a full track consumption with margin.
- `DownloadTrack`: same pattern with 30min TTL (downloads can be
slower on mobile; single-shot flow).
- Both endpoints keep their existing permission checks (share token,
public/owner, license) unchanged — redirect happens only after the
request is authorized to see the track.
- internal/core/track/service_async_test.go
- `TestGetStorageURL` covers 3 cases: local backend (no redirect),
s3 backend with valid key (redirect + TTL forwarded), s3 backend
with nil key (defensive fallback).
Out of scope Phase 2 remaining (A5): transcoder pulls from S3 via
signed URL, HLS segments written to MinIO.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After `CompleteChunkedUpload` lands the assembled file on local FS,
stream it to S3 and delete the local copy when TrackService is in
s3-backend mode. Symmetrical to copyFileAsyncS3 for regular uploads
(`f47141fe`), closing the Phase 1 write path.
Changes:
- internal/core/track/service.go
- New method: `TrackService.MigrateLocalToS3IfConfigured(ctx, trackID,
userID, localPath)`. Opens local file, streams to S3 at
tracks/<userID>/<trackID>.<ext>, updates DB row
(storage_backend='s3', storage_key=<key>), removes local file.
No-op when storageBackend != 's3' or s3Service == nil.
- New method: `TrackService.IsS3Backend() bool` — convenience for
handlers that need to skip path-based transcode triggers when the
file has been migrated off local FS.
- internal/core/track/track_upload_handler.go
- `CompleteChunkedUpload`: after `CreateTrackFromPath` succeeds, call
`MigrateLocalToS3IfConfigured` with a dedicated 10-min context
(S3 stream of up to 500MB can outlive the HTTP request ctx).
- Migration failure is logged but does NOT fail the HTTP response —
the track row exists locally; admin can re-migrate via
cmd/migrate_storage (Phase 3).
- When `IsS3Backend()`, skip the two path-based transcode triggers
(streamService.StartProcessing + jobEnqueuer.EnqueueTranscodingJob).
Phase 2 will re-wire them against signed URLs. For now, tracks
routed to S3 sit in Processing status until Phase 2 lands — same
trade-off as copyFileAsyncS3.
Out of scope (Phase 2 wires these): read path for S3-backed tracks,
transcoder reading from signed URL, HLS segments to MinIO.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits copyFileAsync into local vs s3 branches gated by the
TRACK_STORAGE_BACKEND flag (added in P0 d03232c8). Regular uploads
via TrackService.UploadTrack() now write to MinIO/S3 when the flag
is 's3' and a non-nil S3 service is configured, persisting the S3
object key + storage_backend='s3' on the track row atomically.
Changes:
- internal/core/track/service.go
- New S3StorageInterface (UploadStream + GetSignedURL + DeleteFile).
Narrow surface for testability; *services.S3StorageService satisfies.
- TrackService gains s3Service + storageBackend + s3Bucket fields
and a SetS3Storage setter.
- copyFileAsync is now a dispatcher; former body moved to
copyFileAsyncLocal, new copyFileAsyncS3 streams to S3 with key
tracks/<userID>/<trackID>.<ext>.
- mimeTypeForAudioExt helper.
- Stream server trigger deliberately skipped on S3 branch; wired
in Phase 2 with S3 read support.
- internal/api/routes_tracks.go: DI passes S3StorageService,
TrackStorageBackend, S3Bucket into TrackService.
- internal/core/track/service_async_test.go:
- fakeS3Storage stub (captures UploadStream payload).
- TestUploadTrack_S3Backend_UploadsToS3: end-to-end on key format,
content-type, DB row state.
- TestUploadTrack_S3Backend_NilS3Service_FallsBackToLocal:
defensive — backend='s3' + nil service must not panic.
Out of scope Phase 1: read path, transcoder. Enabling
TRACK_STORAGE_BACKEND=s3 in prod BEFORE Phase 2 ships makes S3-backed
tracks un-streamable. Keep flag 'local' until A4/A5 land.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second item of the v1.0.6 backlog. The "front 500MB vs back 100MB" mismatch
flagged in the v1.0.5 audit turned out to be a misread — every live pair
was already aligned (tracks 100/100, cloud 500/500, video 500/500). The
real bug is architectural: the same byte values were duplicated in five
places (`track/service.go`, `handlers/upload.go:GetUploadLimits`,
`handlers/education_handler.go`, `upload-modal/constants.ts`, and
`CloudUploadModal.tsx`), drifting silently as soon as anyone tuned one.
Backend — one canonical spec at `internal/config/upload_limits.go`:
* `AudioLimit`, `ImageLimit`, `VideoLimit` expose `Bytes()`, `MB()`,
`HumanReadable()`, `AllowedMIMEs` — read lazily from env
(`MAX_UPLOAD_AUDIO_MB`, `MAX_UPLOAD_IMAGE_MB`, `MAX_UPLOAD_VIDEO_MB`)
with defaults 100/10/500.
* Invalid / negative / zero env values fall back to the default;
unreadable config can't turn the limit off silently.
* `track.Service.maxFileSize`, `track_upload_handler.go` error string,
`education_handler.go` video gate, and `upload.go:GetUploadLimits`
all read from this single source. Changing `MAX_UPLOAD_AUDIO_MB`
retunes every path at once.
Frontend — new `useUploadLimits()` hook:
* Fetches GET `/api/v1/upload/limits` via react-query (5 min stale,
30 min gc), one retry, then silently falls back to baked-in
defaults that match the backend compile-time defaults so the
dropzone stays responsive even without the network round-trip.
* `useUploadModal.ts` replaces its hardcoded `MAX_FILE_SIZE`
constant with `useUploadLimits().audio.maxBytes`, and surfaces
`audioMaxHuman` up to `UploadModal` → `UploadModalDropzone` so
the "max 100 MB" label and the "too large" error toast both
display the live value.
* `MAX_FILE_SIZE` constant kept as pure fallback for pre-network
render (documented as such).
Tests
* 4 Go tests on `config.UploadLimit` (defaults, env override, invalid
env → fallback, non-empty MIME lists).
* 4 Vitest tests on `useUploadLimits` (sync fallback on first render,
typed mapping from server payload, partial-payload falls back
per-category, network failure keeps fallback).
* Existing `trackUpload.integration.test.tsx` (11 cases) still green.
Out of scope (tracked for later):
* `CloudUploadModal.tsx` still has its own 500MB hardcoded — cloud
uploads accept audio+zip+midi with a different category semantic
than the three in `/upload/limits`. Unifying those deserves its
own design pass, not a drive-by.
* No runtime refactor of admin-provided custom category limits —
the current tri-category split covers every upload we ship today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
backend-ci.yml's `test -z "$(gofmt -l .)"` strict gate (added in
13c21ac11) failed on a backlog of unformatted files. None of the
85 files in this commit had been edited since the gate was added
because no push touched veza-backend-api/** in between, so the
gate never fired until today's CI fixes triggered it.
The diff is exclusively whitespace alignment in struct literals
and trailing-space comments. `go build ./...` and the full test
suite (with VEZA_SKIP_INTEGRATION=1 -short) pass identically.
INT-02: TrackService.copyFileAsync now calls StreamService.StartProcessing
after successful file copy. Wires the stream server integration into
all track route registrations.
CLN-03: router.go, track/service.go, upload_validator.go, cors.go,
playlist_handler.go, and mfa.go now use zap.L() or local logger
for structured logging instead of fmt.Printf.
- Add DATABASE_READ_URL config and InitReadReplica in database package
- Add ForRead() helper for read-only handler routing
- Update TrackService and TrackSearchService to use read replica for reads
- Document setup in DEPLOYMENT_GUIDE.md and .env.template
