veza/infra/ansible/roles/minio_distributed/README.md

minio_distributed role — distributed MinIO with EC:2

Four Incus containers, each running one MinIO server. Single erasure set of 4 drives = 2 data + 2 parity. The cluster tolerates 2 simultaneous node failures without data loss; storage efficiency is 50% (1 GB raw → 500 MB usable).

Topology

                     S3 API on :9000
                            │
            ┌───────────────┼───────────────┐
            │               │               │
       ┌────▼────┐     ┌────▼────┐     ┌────▼────┐     ┌────▼────┐
       │ minio-1 │     │ minio-2 │     │ minio-3 │     │ minio-4 │
       │  /data  │     │  /data  │     │  /data  │     │  /data  │
       └─────────┘     └─────────┘     └─────────┘     └─────────┘
                  └─── single erasure set, EC:2 ───┘

Each node also runs the web console on :9001.

Why EC:2 (not 4 or larger)

• Recoverability ceiling. EC:N tolerates N drive losses. With 4 drives, EC:4 is a 4-way mirror — 25% efficiency, lose-3 OK but with no functional gain over EC:2 in the failure modes we care about (concurrent node losses).
• Write amplification. EC:2 writes each object to 4 nodes (2 data + 2 parity). EC:4 would write to all 4 + a copy = 4-way replication. Doubling the wire cost for marginal durability isn't worth it on a 4-node cluster.
• Future-proofing. When we go to 6+ nodes (W3+), the natural upgrade is EC:3 across a 6-drive set, NOT growing EC on the same 4 drives.

Defaults

variable	default	meaning
minio_version	RELEASE.2025-09-07T16-13-09Z	matches docker-compose.yml — keep them locked together
minio_port	9000	S3 API
minio_console_port	9001	web console
minio_data_path	/var/lib/minio	drive root on each node
minio_storage_class_standard	EC:2	parity count for STANDARD storage class
minio_bucket_tracks	veza-prod-tracks	prod bucket created on first apply
minio_noncurrent_version_expiry_days	30	delete old object versions after N days
minio_cold_tier_after_days	90	only effective if minio_remote_tier_name is set
minio_remote_tier_name	"" (none)	future remote tier (Glacier / B2). v1.1 territory.
minio_root_user / minio_root_password	(vault)	root credentials

Vault setup

# group_vars/minio_ha.vault.yml — encrypt with `ansible-vault encrypt`
minio_root_user: "<random 32-char access key>"
minio_root_password: "<random 32-char secret>"

The role asserts the placeholder values are gone before applying to anything other than lab.

Backend integration

No code change. The backend's internal/services/storage/s3* already speaks the AWS SDK v2 ; pointing it at the new cluster is a config flip :

AWS_S3_ENABLED=true
AWS_S3_BUCKET=veza-prod-tracks
AWS_S3_ENDPOINT=http://minio-1.lxd:9000   # or behind HAProxy
AWS_S3_REGION=us-east-1                   # MinIO default region
AWS_ACCESS_KEY_ID=<minio_root_user>
AWS_SECRET_ACCESS_KEY=<minio_root_password>

For prod, front the 4 nodes with HAProxy (round-robin, health-checked) so the backend sees a single endpoint and tolerates any 1-node loss without DNS edits. HAProxy config lives in infra/haproxy/ (W4 day 19 ties this in).

Migration from single-node

# On the old single-node host (or via mc on a workstation) :
mc alias set veza-current http://veza.fr:19000 <ACCESS> <SECRET>
mc alias set veza-distributed http://minio-1.lxd:9000 <NEW_ACCESS> <NEW_SECRET>

# Mirror : preserves versioning, ACLs, content-types.
mc mirror --preserve veza-current/veza-files veza-distributed/veza-prod-tracks

# Verify count + bytes match before flipping the AWS_S3_ENDPOINT in
# backend env :
mc ls --recursive veza-current/veza-files       | wc -l
mc ls --recursive veza-distributed/veza-prod-tracks | wc -l

The old bucket can be kept hot for ~ 1 week after the flip in case a rollback is needed, then mc rm --recursive --force --dangerous drops it.

Operations

# Cluster health (admin info = info about each drive) :
mc admin info veza-distributed

# Per-node verbose state :
ssh minio-1 sudo journalctl -u minio -n 100 --no-pager

# Watch heal progress (after a node was offline / drive replaced) :
mc admin heal veza-distributed --recursive

# Check lifecycle policy :
mc ilm ls veza-distributed/veza-prod-tracks

# Console UI (per-node — pick any) :
open http://minio-1.lxd:9001

Failover smoke test

MINIO_ROOT_USER=... MINIO_ROOT_PASSWORD=... \
  bash infra/ansible/tests/test_minio_resilience.sh

Sequence : upload 100 MB random file, kill 2 nodes, assert read still works, restart nodes, wait for self-heal, assert all 4 nodes report healthy.

What this role does NOT cover

• Cross-DC replication. Single-host (lab) or single-region in v1.0. v1.1+ adds bucket replication to a remote cluster.
• Site replication / federation. Multi-tenant federation is out of scope.
• Cold tier transitions. minio_remote_tier_name is empty by default — no Glacier / B2 / second-cluster behind the lifecycle yet. Wire when needed.
• mTLS. --tls-cert/key is W4. The Incus bridge is the security boundary today.