veza/veza-backend-api/docs/DEPLOYMENT_GUIDE.md

Veza Backend API - Deployment Guide

Table of Contents

1. Overview
2. Prerequisites
3. Environment Configuration
4. Local Development Deployment
5. Staging Deployment
6. Production Deployment
7. Docker Deployment
8. Ansible Deployment
9. Kubernetes Deployment
10. Database Migrations
11. Health Checks and Monitoring
12. Rollback Procedures
13. Troubleshooting
14. Maintenance

Overview

This guide covers deployment procedures for the Veza Backend API across all environments:

• Local Development: Docker Compose for local testing
• Staging: Pre-production environment for testing
• Production: Production deployment with high availability

The API supports multiple deployment methods:

• Docker and Docker Compose
• Ansible automation
• Kubernetes orchestration
• Manual deployment

Prerequisites

System Requirements

Minimum Requirements:

• CPU: 2 cores
• RAM: 4GB
• Storage: 20GB
• Network: 100Mbps

Recommended for Production:

• CPU: 4+ cores
• RAM: 8GB+
• Storage: 100GB+ (SSD recommended)
• Network: 1Gbps

Software Requirements

• Operating System: Linux (Debian 12+, Ubuntu 22.04+, RHEL 8+)
• Container Runtime: Docker 20.10+ or containerd 1.6+
• Orchestration: Kubernetes 1.24+ (optional)
• Database: PostgreSQL 12+
• Cache: Redis 6+ (optional but recommended)
• Reverse Proxy: Nginx 1.20+ or HAProxy 2.4+ (for production)

Development Tools

• Go 1.23+
• Git
• Make
• Docker and Docker Compose

Environment Configuration

Environment Variables

The API uses environment variables for configuration. Create a .env file or set environment variables:

Required Variables

# Database
DATABASE_URL=postgres://user:password@host:5432/dbname?sslmode=disable
# Optional: Read replica for scaling read-heavy workloads (same format as DATABASE_URL)
# DATABASE_READ_URL=postgres://user:password@read-replica-host:5432/dbname?sslmode=disable

# Security
JWT_SECRET=<32+ character secret>
JWT_EXPIRY=24h

# Application
APP_ENV=production  # development|staging|production
API_PORT=8080

Production Required Variables

# CORS (Required in production)
CORS_ALLOWED_ORIGINS=https://app.veza.com,https://www.veza.com

# Database
DB_MAX_RETRIES=5
DB_RETRY_INTERVAL=5s
DB_MAX_OPEN_CONNS=25
DB_MAX_IDLE_CONNS=5
DB_CONN_MAX_LIFETIME=5m

# Redis (Optional but recommended)
REDIS_URL=redis://:password@host:6379
REDIS_PASSWORD=your_redis_password

# Logging
LOG_LEVEL=INFO  # DEBUG|INFO|WARN|ERROR
LOG_FORMAT=json  # json|text

Optional Variables

# ClamAV (Antivirus scanning)
ENABLE_CLAMAV=true
CLAMAV_REQUIRED=true
CLAMAV_HOST=localhost
CLAMAV_PORT=3310

# Email
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=user@example.com
SMTP_PASSWORD=password
SMTP_FROM=noreply@veza.com

# Storage
UPLOAD_DIR=/var/lib/veza/uploads
MAX_UPLOAD_SIZE=500MB
ALLOWED_AUDIO_FORMATS=mp3,wav,flac,ogg

# Rate Limiting
RATE_LIMIT_ENABLED=true
RATE_LIMIT_REDIS_URL=redis://:password@host:6379

# Monitoring
PROMETHEUS_ENABLED=true
PROMETHEUS_PORT=9090

# Stream Server Integration
STREAM_SERVER_URL=http://stream-server:8082

# Webhooks
WEBHOOK_SECRET=<32+ character secret>
WEBHOOK_TIMEOUT=30s

Configuration Files

Production Config Example

Create config/production.env:

# Production Configuration
APP_ENV=production
LOG_LEVEL=INFO
LOG_FORMAT=json

# Database
DATABASE_URL=postgres://veza_user:secure_password@db.veza.internal:5432/veza_prod?sslmode=require
DB_MAX_OPEN_CONNS=50
DB_MAX_IDLE_CONNS=10

# Security
JWT_SECRET=<generate-strong-secret-32-chars-min>
JWT_EXPIRY=24h
CORS_ALLOWED_ORIGINS=https://app.veza.com,https://www.veza.com

# Redis
REDIS_URL=redis://:redis_password@redis.veza.internal:6379/0

# Storage
UPLOAD_DIR=/var/lib/veza/uploads
MAX_UPLOAD_SIZE=500MB

# Monitoring
PROMETHEUS_ENABLED=true

Local Development Deployment

Quick Start with Docker Compose

# Clone repository
git clone <repository-url>
cd veza

# Copy environment file
cp veza-backend-api/.env.example veza-backend-api/.env

# Edit .env with your local settings
nano veza-backend-api/.env

# Start all services
docker-compose up -d

# Check logs
docker-compose logs -f backend-api

# Stop services
docker-compose down

Manual Local Setup

# Install dependencies
cd veza-backend-api
go mod download

# Run database migrations
make migrate-up

# Build application
make build

# Run application
./bin/veza-backend-api

# Or run with hot reload (development)
make dev

Local Development with Hot Reload

# Install air for hot reload
go install github.com/cosmtrek/air@latest

# Run with hot reload
air

# Or use make
make dev

Staging Deployment

Staging Environment Setup

Staging environment should mirror production as closely as possible:

# Set environment
export APP_ENV=staging

# Use staging database
export DATABASE_URL=postgres://user:pass@staging-db:5432/veza_staging

# Use staging Redis
export REDIS_URL=redis://:pass@staging-redis:6379

# Deploy with Docker Compose
docker-compose -f docker-compose.staging.yml up -d

# Run migrations
docker-compose -f docker-compose.staging.yml exec backend-api ./veza-api migrate up

# Check health
curl https://staging-api.veza.com/healthz

Staging Docker Compose

Create docker-compose.staging.yml:

version: '3.8'

services:
  backend-api:
    image: veza/backend-api:staging
    environment:
      APP_ENV: staging
      DATABASE_URL: ${DATABASE_URL}
      JWT_SECRET: ${JWT_SECRET}
      CORS_ALLOWED_ORIGINS: https://staging.veza.com
      LOG_LEVEL: DEBUG
    ports:
      - "8080:8080"
    restart: unless-stopped

Production Deployment

Pre-Deployment Checklist

• All environment variables configured
• Database backups created
• SSL certificates valid
• DNS records configured
• Firewall rules configured
• Monitoring alerts configured
• Rollback plan prepared
• Team notified of deployment

Production Deployment Methods

Method 1: Docker Compose (Single Server)

# Pull latest image
docker pull veza/backend-api:latest

# Stop current deployment
docker-compose -f docker-compose.production.yml down

# Backup database
./scripts/backup-database.sh

# Update environment variables
nano .env.production

# Start new deployment
docker-compose -f docker-compose.production.yml up -d

# Run migrations
docker-compose -f docker-compose.production.yml exec backend-api ./veza-api migrate up

# Verify health
curl https://api.veza.com/healthz

# Check logs
docker-compose -f docker-compose.production.yml logs -f backend-api

Method 2: Docker (Standalone)

# Build production image
docker build -f Dockerfile.production -t veza/backend-api:latest .

# Tag for registry
docker tag veza/backend-api:latest registry.veza.com/veza/backend-api:v1.2.0

# Push to registry
docker push registry.veza.com/veza/backend-api:v1.2.0

# Run container
docker run -d \
  --name veza-backend-api \
  --restart unless-stopped \
  -p 8080:8080 \
  --env-file .env.production \
  -v /var/lib/veza/uploads:/app/uploads \
  veza/backend-api:latest

# Check logs
docker logs -f veza-backend-api

Method 3: Systemd Service

Create /etc/systemd/system/veza-backend-api.service:

[Unit]
Description=Veza Backend API
After=network.target postgresql.service redis.service

[Service]
Type=simple
User=veza
Group=veza
WorkingDirectory=/opt/veza/backend-api
EnvironmentFile=/etc/veza/backend-api.env
ExecStart=/opt/veza/backend-api/bin/veza-backend-api
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start:

# Reload systemd
sudo systemctl daemon-reload

# Enable service
sudo systemctl enable veza-backend-api

# Start service
sudo systemctl start veza-backend-api

# Check status
sudo systemctl status veza-backend-api

# View logs
sudo journalctl -u veza-backend-api -f

Docker Deployment

Building Docker Images

Development Image

docker build -t veza/backend-api:dev -f Dockerfile .

Production Image

docker build -t veza/backend-api:latest -f Dockerfile.production .

Docker Compose Production

Use docker-compose.production.yml:

# Start services
docker-compose -f docker-compose.production.yml up -d

# View logs
docker-compose -f docker-compose.production.yml logs -f

# Scale backend (if needed)
docker-compose -f docker-compose.production.yml up -d --scale backend-api=3

# Stop services
docker-compose -f docker-compose.production.yml down

# Stop and remove volumes
docker-compose -f docker-compose.production.yml down -v

Docker Health Checks

The production Dockerfile includes health checks:

HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

Check container health:

docker ps  # Shows health status
docker inspect veza-backend-api | grep Health

Ansible Deployment

Prerequisites

# Install Ansible
pip install ansible

# Install required collections
ansible-galaxy collection install community.general community.docker

Full Deployment

cd ansible

# Run full deployment
./deploy-veza.sh

# Or with custom domain
./deploy-veza.sh -d api.veza.com -e admin@veza.com

Step-by-Step Deployment

# 1. Bootstrap target host
ansible-playbook -i inventory/prod/hosts.yml playbooks/00-bootstrap-remote.yml

# 2. Install Incus and OVN
ansible-playbook -i inventory/prod/hosts.yml playbooks/10-incus-ovn.yml

# 3. Create containers
ansible-playbook -i inventory/prod/hosts.yml playbooks/20-incus-containers.yml

# 4. Configure HAProxy and SSL
ansible-playbook -i inventory/prod/hosts.yml playbooks/30-haproxy-acme.yml \
  -e domain=api.veza.com -e acme_email=admin@veza.com

# 5. Deploy applications
ansible-playbook -i inventory/prod/hosts.yml playbooks/40-veza-apps.yml

# 6. Run smoke tests
ansible-playbook -i inventory/prod/hosts.yml playbooks/50-smoke-tests.yml

Ansible Inventory

Create ansible/inventory/prod/hosts.yml:

all:
  children:
    production:
      hosts:
        api-server:
          ansible_host: 192.168.1.100
          ansible_user: deploy
          ansible_ssh_private_key_file: ~/.ssh/deploy_key

Kubernetes Deployment

Prerequisites

• Kubernetes cluster 1.24+
• kubectl configured
• Helm 3.8+ (optional)

Deploy with kubectl

# Create namespace
kubectl create namespace veza

# Create secrets
kubectl create secret generic veza-backend-secrets \
  --from-env-file=.env.production \
  -n veza

# Deploy application
kubectl apply -f k8s/deployment.yaml

# Deploy service
kubectl apply -f k8s/service.yaml

# Check status
kubectl get pods -n veza
kubectl logs -f deployment/veza-backend-api -n veza

Kubernetes Deployment Example

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: veza-backend-api
  namespace: veza
spec:
  replicas: 3
  selector:
    matchLabels:
      app: veza-backend-api
  template:
    metadata:
      labels:
        app: veza-backend-api
    spec:
      containers:
      - name: backend-api
        image: veza/backend-api:latest
        ports:
        - containerPort: 8080
        envFrom:
        - secretRef:
            name: veza-backend-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

Deploy with Helm

# Install Helm chart
helm install veza-backend ./helm/veza-backend \
  --namespace veza \
  --set image.tag=v1.2.0 \
  --set env.databaseUrl=postgres://... \
  --set env.jwtSecret=...

# Upgrade
helm upgrade veza-backend ./helm/veza-backend \
  --namespace veza \
  --set image.tag=v1.2.1

# Rollback
helm rollback veza-backend 1 --namespace veza

Database Migrations

Running Migrations

With Docker

# Run migrations
docker-compose exec backend-api ./veza-api migrate up

# Rollback last migration
docker-compose exec backend-api ./veza-api migrate down

# Check migration status
docker-compose exec backend-api ./veza-api migrate status

Manual Migration

# Install migrate tool
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

# Run migrations
migrate -path ./migrations -database "$DATABASE_URL" up

# Rollback
migrate -path ./migrations -database "$DATABASE_URL" down 1

# Check status
migrate -path ./migrations -database "$DATABASE_URL" version

Migration Best Practices

1. Always backup before migrations in production
2. Test migrations in staging first
3. Run migrations during low-traffic periods
4. Monitor application after migration
5. Have rollback plan ready

# Backup before migration
pg_dump -h db.veza.internal -U veza_user veza_prod > backup_$(date +%Y%m%d_%H%M%S).sql

# Run migration
migrate -path ./migrations -database "$DATABASE_URL" up

# Verify
psql -h db.veza.internal -U veza_user -d veza_prod -c "SELECT version FROM schema_migrations;"

PostgreSQL Read Replica Setup

For high-traffic deployments, you can offload read queries to a PostgreSQL read replica. The API routes read-only operations (e.g. GET /tracks, GET /playlists, GET /search) to the replica when configured.

1. Configure PostgreSQL Streaming Replication

Set up a read replica using PostgreSQL streaming replication. The replica must use the same schema as the primary (migrations run only on the primary).

2. Environment Variables

# Primary (write) connection - required
DATABASE_URL=postgres://user:password@primary-host:5432/veza_prod?sslmode=require

# Read replica - optional
DATABASE_READ_URL=postgres://user:password@read-replica-host:5432/veza_prod?sslmode=require

# Replica connection pool (optional, defaults shown)
DB_READ_MAX_OPEN_CONNS=25
DB_READ_MAX_IDLE_CONNS=6

3. Behavior

• When DATABASE_READ_URL is set, the API establishes a separate connection pool to the replica.
• Read-only handlers use the replica; write operations use the primary.
• If the replica connection fails at startup, the API logs a warning and uses the primary for all operations.
• Replication lag: expect a few seconds of delay; avoid time-sensitive reads on the replica.

Health Checks and Monitoring

Health Check Endpoints

# Basic health check
curl http://localhost:8080/healthz

# Detailed health check
curl http://localhost:8080/health

# Readiness check
curl http://localhost:8080/ready

Health Check Response

{
  "status": "healthy",
  "timestamp": "2025-01-01T00:00:00Z",
  "checks": {
    "database": "healthy",
    "redis": "healthy",
    "disk": "healthy"
  }
}

Monitoring Setup

Prometheus Metrics

The API exposes Prometheus metrics at /metrics:

# Scrape metrics
curl http://localhost:8080/metrics

Prometheus Configuration

scrape_configs:
  - job_name: 'veza-backend-api'
    scrape_interval: 15s
    static_configs:
      - targets: ['backend-api:8080']
    metrics_path: '/metrics'

Grafana Dashboard

Import the provided Grafana dashboard from ops/prometheus/grafana-dashboard.json.

Logging

Log Levels

• DEBUG: Detailed information for debugging
• INFO: General informational messages
• WARN: Warning messages
• ERROR: Error messages

Log Formats

• json: Structured JSON logs (production)
• text: Human-readable text logs (development)

Viewing Logs

# Docker logs
docker logs -f veza-backend-api

# Docker Compose logs
docker-compose logs -f backend-api

# Systemd logs
journalctl -u veza-backend-api -f

# Kubernetes logs
kubectl logs -f deployment/veza-backend-api -n veza

Rollback Procedures

Docker Rollback

# Stop current deployment
docker-compose -f docker-compose.production.yml down

# Tag previous image
docker tag veza/backend-api:previous veza/backend-api:latest

# Start previous version
docker-compose -f docker-compose.production.yml up -d

# Verify
curl https://api.veza.com/healthz

Database Migration Rollback

# Rollback last migration
migrate -path ./migrations -database "$DATABASE_URL" down 1

# Rollback to specific version
migrate -path ./migrations -database "$DATABASE_URL" force <version>

Kubernetes Rollback

# Rollback deployment
kubectl rollout undo deployment/veza-backend-api -n veza

# Rollback to specific revision
kubectl rollout undo deployment/veza-backend-api -n veza --to-revision=2

# Check rollout history
kubectl rollout history deployment/veza-backend-api -n veza

Ansible Rollback

# Revert to previous playbook version
git checkout <previous-commit>
ansible-playbook -i inventory/prod/hosts.yml playbooks/40-veza-apps.yml

Troubleshooting

Common Issues

Application Won't Start

# Check logs
docker logs veza-backend-api
journalctl -u veza-backend-api

# Verify environment variables
docker exec veza-backend-api env | grep -E 'DATABASE|JWT|APP_ENV'

# Check database connectivity
docker exec veza-backend-api ./veza-api migrate status

Database Connection Issues

# Test database connection
psql "$DATABASE_URL" -c "SELECT 1;"

# Check database is running
docker ps | grep postgres
systemctl status postgresql

# Verify network connectivity
docker exec veza-backend-api ping postgres

High Memory Usage

# Check memory usage
docker stats veza-backend-api

# Adjust database connection pool
export DB_MAX_OPEN_CONNS=25
export DB_MAX_IDLE_CONNS=5

# Restart with new settings
docker-compose restart backend-api

Slow Response Times

# Check application metrics
curl http://localhost:8080/metrics | grep http_request_duration

# Check database query times
# Enable slow query log in PostgreSQL

# Check Redis connectivity
redis-cli -h redis.veza.internal ping

Debug Mode

Enable debug logging:

export LOG_LEVEL=DEBUG
export LOG_FORMAT=text

Performance Tuning

Database Connection Pool

DB_MAX_OPEN_CONNS=50
DB_MAX_IDLE_CONNS=10
DB_CONN_MAX_LIFETIME=5m

Application Resources

# Kubernetes resources
resources:
  requests:
    memory: "1Gi"
    cpu: "1000m"
  limits:
    memory: "2Gi"
    cpu: "2000m"

Maintenance

Regular Maintenance Tasks

Daily

• Monitor health checks
• Review error logs
• Check disk space
• Verify backups

Weekly

• Review performance metrics
• Check for security updates
• Review database size
• Clean up old logs

Monthly

• Update dependencies
• Review and optimize database
• Review and update documentation
• Security audit

Backup Procedures

Database Backup

# Automated backup script
#!/bin/bash
BACKUP_DIR="/var/backups/veza"
DATE=$(date +%Y%m%d_%H%M%S)
pg_dump "$DATABASE_URL" > "$BACKUP_DIR/veza_db_$DATE.sql"
gzip "$BACKUP_DIR/veza_db_$DATE.sql"

# Keep only last 30 days
find "$BACKUP_DIR" -name "veza_db_*.sql.gz" -mtime +30 -delete

Application Backup

# Backup uploads directory
tar -czf /var/backups/veza/uploads_$(date +%Y%m%d).tar.gz /var/lib/veza/uploads

# Backup configuration
cp .env.production /var/backups/veza/env_$(date +%Y%m%d).env

Update Procedures

Minor Updates

# Pull latest image
docker pull veza/backend-api:latest

# Rolling update
docker-compose -f docker-compose.production.yml up -d --no-deps backend-api

# Verify
curl https://api.veza.com/healthz

Major Updates

1. Backup database
2. Test in staging
3. Schedule maintenance window
4. Deploy new version
5. Run migrations
6. Verify functionality
7. Monitor for issues

Security Updates

# Update base image
docker pull golang:1.24-alpine

# Rebuild with security updates
docker build --no-cache -f Dockerfile.production -t veza/backend-api:latest .

# Scan for vulnerabilities
docker scan veza/backend-api:latest

Additional Resources

• API Documentation: See docs/API_DOCUMENTATION.md
• Development Guide: See README.md
• Ansible Deployment: See ansible/DEPLOYMENT_GUIDE.md
• Kubernetes Manifests: See k8s/ directory
• Monitoring Setup: See ops/prometheus/ directory

Support

For deployment issues:

• Email: ops@veza.com
• Documentation: https://docs.veza.com
• Issues: https://github.com/veza/veza-backend-api/issues