veza/veza-backend-api/docs/DEPLOYMENT_GUIDE.md

# Veza Backend API - Deployment Guide

## Table of Contents

1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Environment Configuration](#environment-configuration)
4. [Local Development Deployment](#local-development-deployment)
5. [Staging Deployment](#staging-deployment)
6. [Production Deployment](#production-deployment)
7. [Docker Deployment](#docker-deployment)
8. [Ansible Deployment](#ansible-deployment)
9. [Kubernetes Deployment](#kubernetes-deployment)
10. [Database Migrations](#database-migrations)
11. [Health Checks and Monitoring](#health-checks-and-monitoring)
12. [Rollback Procedures](#rollback-procedures)
13. [Troubleshooting](#troubleshooting)
14. [Maintenance](#maintenance)

## Overview

This guide covers deployment procedures for the Veza Backend API across all environments:
- **Local Development**: Docker Compose for local testing
- **Staging**: Pre-production environment for testing
- **Production**: Production deployment with high availability

The API supports multiple deployment methods:
- Docker and Docker Compose
- Ansible automation
- Kubernetes orchestration
- Manual deployment

## Prerequisites

### System Requirements

**Minimum Requirements:**
- CPU: 2 cores
- RAM: 4GB
- Storage: 20GB
- Network: 100Mbps

**Recommended for Production:**
- CPU: 4+ cores
- RAM: 8GB+
- Storage: 100GB+ (SSD recommended)
- Network: 1Gbps

### Software Requirements

- **Operating System**: Linux (Debian 12+, Ubuntu 22.04+, RHEL 8+)
- **Container Runtime**: Docker 20.10+ or containerd 1.6+
- **Orchestration**: Kubernetes 1.24+ (optional)
- **Database**: PostgreSQL 12+
- **Cache**: Redis 6+ (optional but recommended)
- **Reverse Proxy**: Nginx 1.20+ or HAProxy 2.4+ (for production)

### Development Tools

- Go 1.23+
- Git
- Make
- Docker and Docker Compose

## Environment Configuration

### Environment Variables

The API uses environment variables for configuration. Create a `.env` file or set environment variables:

#### Required Variables

```bash
# Database
DATABASE_URL=postgres://user:password@host:5432/dbname?sslmode=disable
# Optional: Read replica for scaling read-heavy workloads (same format as DATABASE_URL)
# DATABASE_READ_URL=postgres://user:password@read-replica-host:5432/dbname?sslmode=disable

# Security
JWT_SECRET=<32+ character secret>
JWT_EXPIRY=24h

# Application
APP_ENV=production  # development|staging|production
API_PORT=8080
```

#### Production Required Variables

```bash
# CORS (Required in production)
CORS_ALLOWED_ORIGINS=https://app.veza.com,https://www.veza.com

# Database
DB_MAX_RETRIES=5
DB_RETRY_INTERVAL=5s
DB_MAX_OPEN_CONNS=25
DB_MAX_IDLE_CONNS=5
DB_CONN_MAX_LIFETIME=5m

# Redis (Optional but recommended)
REDIS_URL=redis://:password@host:6379
REDIS_PASSWORD=your_redis_password

# Logging
LOG_LEVEL=INFO  # DEBUG|INFO|WARN|ERROR
LOG_FORMAT=json  # json|text
```

#### Optional Variables

```bash
# ClamAV (Antivirus scanning)
ENABLE_CLAMAV=true
CLAMAV_REQUIRED=true
CLAMAV_HOST=localhost
CLAMAV_PORT=3310

# Email
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=user@example.com
SMTP_PASSWORD=password
SMTP_FROM=noreply@veza.com

# Storage
UPLOAD_DIR=/var/lib/veza/uploads
MAX_UPLOAD_SIZE=500MB
ALLOWED_AUDIO_FORMATS=mp3,wav,flac,ogg

# Rate Limiting
RATE_LIMIT_ENABLED=true
RATE_LIMIT_REDIS_URL=redis://:password@host:6379

# Monitoring
PROMETHEUS_ENABLED=true
PROMETHEUS_PORT=9090

# Chat Integration
CHAT_SERVER_URL=http://chat-server:8081
CHAT_JWT_SECRET=<32+ character secret>

# Stream Server Integration
STREAM_SERVER_URL=http://stream-server:8082

# Webhooks
WEBHOOK_SECRET=<32+ character secret>
WEBHOOK_TIMEOUT=30s
```

### Configuration Files

#### Production Config Example

Create `config/production.env`:

```bash
# Production Configuration
APP_ENV=production
LOG_LEVEL=INFO
LOG_FORMAT=json

# Database
DATABASE_URL=postgres://veza_user:secure_password@db.veza.internal:5432/veza_prod?sslmode=require
DB_MAX_OPEN_CONNS=50
DB_MAX_IDLE_CONNS=10

# Security
JWT_SECRET=<generate-strong-secret-32-chars-min>
JWT_EXPIRY=24h
CORS_ALLOWED_ORIGINS=https://app.veza.com,https://www.veza.com

# Redis
REDIS_URL=redis://:redis_password@redis.veza.internal:6379/0

# Storage
UPLOAD_DIR=/var/lib/veza/uploads
MAX_UPLOAD_SIZE=500MB

# Monitoring
PROMETHEUS_ENABLED=true
```

## Local Development Deployment

### Quick Start with Docker Compose

```bash
# Clone repository
git clone <repository-url>
cd veza

# Copy environment file
cp veza-backend-api/.env.example veza-backend-api/.env

# Edit .env with your local settings
nano veza-backend-api/.env

# Start all services
docker-compose up -d

# Check logs
docker-compose logs -f backend-api

# Stop services
docker-compose down
```

### Manual Local Setup

```bash
# Install dependencies
cd veza-backend-api
go mod download

# Run database migrations
make migrate-up

# Build application
make build

# Run application
./bin/veza-backend-api

# Or run with hot reload (development)
make dev
```

### Local Development with Hot Reload

```bash
# Install air for hot reload
go install github.com/cosmtrek/air@latest

# Run with hot reload
air

# Or use make
make dev
```

## Staging Deployment

### Staging Environment Setup

Staging environment should mirror production as closely as possible:

```bash
# Set environment
export APP_ENV=staging

# Use staging database
export DATABASE_URL=postgres://user:pass@staging-db:5432/veza_staging

# Use staging Redis
export REDIS_URL=redis://:pass@staging-redis:6379

# Deploy with Docker Compose
docker-compose -f docker-compose.staging.yml up -d

# Run migrations
docker-compose -f docker-compose.staging.yml exec backend-api ./veza-api migrate up

# Check health
curl https://staging-api.veza.com/healthz
```

### Staging Docker Compose

Create `docker-compose.staging.yml`:

```yaml
version: '3.8'

services:
  backend-api:
    image: veza/backend-api:staging
    environment:
      APP_ENV: staging
      DATABASE_URL: ${DATABASE_URL}
      JWT_SECRET: ${JWT_SECRET}
      CORS_ALLOWED_ORIGINS: https://staging.veza.com
      LOG_LEVEL: DEBUG
    ports:
      - "8080:8080"
    restart: unless-stopped
```

## Production Deployment

### Pre-Deployment Checklist

- [ ] All environment variables configured
- [ ] Database backups created
- [ ] SSL certificates valid
- [ ] DNS records configured
- [ ] Firewall rules configured
- [ ] Monitoring alerts configured
- [ ] Rollback plan prepared
- [ ] Team notified of deployment

### Production Deployment Methods

#### Method 1: Docker Compose (Single Server)

```bash
# Pull latest image
docker pull veza/backend-api:latest

# Stop current deployment
docker-compose -f docker-compose.production.yml down

# Backup database
./scripts/backup-database.sh

# Update environment variables
nano .env.production

# Start new deployment
docker-compose -f docker-compose.production.yml up -d

# Run migrations
docker-compose -f docker-compose.production.yml exec backend-api ./veza-api migrate up

# Verify health
curl https://api.veza.com/healthz

# Check logs
docker-compose -f docker-compose.production.yml logs -f backend-api
```

#### Method 2: Docker (Standalone)

```bash
# Build production image
docker build -f Dockerfile.production -t veza/backend-api:latest .

# Tag for registry
docker tag veza/backend-api:latest registry.veza.com/veza/backend-api:v1.2.0

# Push to registry
docker push registry.veza.com/veza/backend-api:v1.2.0

# Run container
docker run -d \
  --name veza-backend-api \
  --restart unless-stopped \
  -p 8080:8080 \
  --env-file .env.production \
  -v /var/lib/veza/uploads:/app/uploads \
  veza/backend-api:latest

# Check logs
docker logs -f veza-backend-api
```

#### Method 3: Systemd Service

Create `/etc/systemd/system/veza-backend-api.service`:

```ini
[Unit]
Description=Veza Backend API
After=network.target postgresql.service redis.service

[Service]
Type=simple
User=veza
Group=veza
WorkingDirectory=/opt/veza/backend-api
EnvironmentFile=/etc/veza/backend-api.env
ExecStart=/opt/veza/backend-api/bin/veza-backend-api
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
# Reload systemd
sudo systemctl daemon-reload

# Enable service
sudo systemctl enable veza-backend-api

# Start service
sudo systemctl start veza-backend-api

# Check status
sudo systemctl status veza-backend-api

# View logs
sudo journalctl -u veza-backend-api -f
```

## Docker Deployment

### Building Docker Images

#### Development Image

```bash
docker build -t veza/backend-api:dev -f Dockerfile .
```

#### Production Image

```bash
docker build -t veza/backend-api:latest -f Dockerfile.production .
```

### Docker Compose Production

Use `docker-compose.production.yml`:

```bash
# Start services
docker-compose -f docker-compose.production.yml up -d

# View logs
docker-compose -f docker-compose.production.yml logs -f

# Scale backend (if needed)
docker-compose -f docker-compose.production.yml up -d --scale backend-api=3

# Stop services
docker-compose -f docker-compose.production.yml down

# Stop and remove volumes
docker-compose -f docker-compose.production.yml down -v
```

### Docker Health Checks

The production Dockerfile includes health checks:

```dockerfile
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
```

Check container health:

```bash
docker ps  # Shows health status
docker inspect veza-backend-api | grep Health
```

## Ansible Deployment

### Prerequisites

```bash
# Install Ansible
pip install ansible

# Install required collections
ansible-galaxy collection install community.general community.docker
```

### Full Deployment

```bash
cd ansible

# Run full deployment
./deploy-veza.sh

# Or with custom domain
./deploy-veza.sh -d api.veza.com -e admin@veza.com
```

### Step-by-Step Deployment

```bash
# 1. Bootstrap target host
ansible-playbook -i inventory/prod/hosts.yml playbooks/00-bootstrap-remote.yml

# 2. Install Incus and OVN
ansible-playbook -i inventory/prod/hosts.yml playbooks/10-incus-ovn.yml

# 3. Create containers
ansible-playbook -i inventory/prod/hosts.yml playbooks/20-incus-containers.yml

# 4. Configure HAProxy and SSL
ansible-playbook -i inventory/prod/hosts.yml playbooks/30-haproxy-acme.yml \
  -e domain=api.veza.com -e acme_email=admin@veza.com

# 5. Deploy applications
ansible-playbook -i inventory/prod/hosts.yml playbooks/40-veza-apps.yml

# 6. Run smoke tests
ansible-playbook -i inventory/prod/hosts.yml playbooks/50-smoke-tests.yml
```

### Ansible Inventory

Create `ansible/inventory/prod/hosts.yml`:

```yaml
all:
  children:
    production:
      hosts:
        api-server:
          ansible_host: 192.168.1.100
          ansible_user: deploy
          ansible_ssh_private_key_file: ~/.ssh/deploy_key
```

## Kubernetes Deployment

### Prerequisites

- Kubernetes cluster 1.24+
- kubectl configured
- Helm 3.8+ (optional)

### Deploy with kubectl

```bash
# Create namespace
kubectl create namespace veza

# Create secrets
kubectl create secret generic veza-backend-secrets \
  --from-env-file=.env.production \
  -n veza

# Deploy application
kubectl apply -f k8s/deployment.yaml

# Deploy service
kubectl apply -f k8s/service.yaml

# Check status
kubectl get pods -n veza
kubectl logs -f deployment/veza-backend-api -n veza
```

### Kubernetes Deployment Example

Create `k8s/deployment.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: veza-backend-api
  namespace: veza
spec:
  replicas: 3
  selector:
    matchLabels:
      app: veza-backend-api
  template:
    metadata:
      labels:
        app: veza-backend-api
    spec:
      containers:
      - name: backend-api
        image: veza/backend-api:latest
        ports:
        - containerPort: 8080
        envFrom:
        - secretRef:
            name: veza-backend-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
```

### Deploy with Helm

```bash
# Install Helm chart
helm install veza-backend ./helm/veza-backend \
  --namespace veza \
  --set image.tag=v1.2.0 \
  --set env.databaseUrl=postgres://... \
  --set env.jwtSecret=...

# Upgrade
helm upgrade veza-backend ./helm/veza-backend \
  --namespace veza \
  --set image.tag=v1.2.1

# Rollback
helm rollback veza-backend 1 --namespace veza
```

## Database Migrations

### Running Migrations

#### With Docker

```bash
# Run migrations
docker-compose exec backend-api ./veza-api migrate up

# Rollback last migration
docker-compose exec backend-api ./veza-api migrate down

# Check migration status
docker-compose exec backend-api ./veza-api migrate status
```

#### Manual Migration

```bash
# Install migrate tool
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

# Run migrations
migrate -path ./migrations -database "$DATABASE_URL" up

# Rollback
migrate -path ./migrations -database "$DATABASE_URL" down 1

# Check status
migrate -path ./migrations -database "$DATABASE_URL" version
```

### Migration Best Practices

1. **Always backup before migrations in production**
2. **Test migrations in staging first**
3. **Run migrations during low-traffic periods**
4. **Monitor application after migration**
5. **Have rollback plan ready**

```bash
# Backup before migration
pg_dump -h db.veza.internal -U veza_user veza_prod > backup_$(date +%Y%m%d_%H%M%S).sql

# Run migration
migrate -path ./migrations -database "$DATABASE_URL" up

# Verify
psql -h db.veza.internal -U veza_user -d veza_prod -c "SELECT version FROM schema_migrations;"
```

### PostgreSQL Read Replica Setup

For high-traffic deployments, you can offload read queries to a PostgreSQL read replica. The API routes read-only operations (e.g. `GET /tracks`, `GET /playlists`, `GET /search`) to the replica when configured.

#### 1. Configure PostgreSQL Streaming Replication

Set up a read replica using PostgreSQL streaming replication. The replica must use the same schema as the primary (migrations run only on the primary).

#### 2. Environment Variables

```bash
# Primary (write) connection - required
DATABASE_URL=postgres://user:password@primary-host:5432/veza_prod?sslmode=require

# Read replica - optional
DATABASE_READ_URL=postgres://user:password@read-replica-host:5432/veza_prod?sslmode=require

# Replica connection pool (optional, defaults shown)
DB_READ_MAX_OPEN_CONNS=25
DB_READ_MAX_IDLE_CONNS=6
```

#### 3. Behavior

- When `DATABASE_READ_URL` is set, the API establishes a separate connection pool to the replica.
- Read-only handlers use the replica; write operations use the primary.
- If the replica connection fails at startup, the API logs a warning and uses the primary for all operations.
- Replication lag: expect a few seconds of delay; avoid time-sensitive reads on the replica.

## Health Checks and Monitoring

### Health Check Endpoints

```bash
# Basic health check
curl http://localhost:8080/healthz

# Detailed health check
curl http://localhost:8080/health

# Readiness check
curl http://localhost:8080/ready
```

### Health Check Response

```json
{
  "status": "healthy",
  "timestamp": "2025-01-01T00:00:00Z",
  "checks": {
    "database": "healthy",
    "redis": "healthy",
    "disk": "healthy"
  }
}
```

### Monitoring Setup

#### Prometheus Metrics

The API exposes Prometheus metrics at `/metrics`:

```bash
# Scrape metrics
curl http://localhost:8080/metrics
```

#### Prometheus Configuration

```yaml
scrape_configs:
  - job_name: 'veza-backend-api'
    scrape_interval: 15s
    static_configs:
      - targets: ['backend-api:8080']
    metrics_path: '/metrics'
```

#### Grafana Dashboard

Import the provided Grafana dashboard from `ops/prometheus/grafana-dashboard.json`.

### Logging

#### Log Levels

- `DEBUG`: Detailed information for debugging
- `INFO`: General informational messages
- `WARN`: Warning messages
- `ERROR`: Error messages

#### Log Formats

- `json`: Structured JSON logs (production)
- `text`: Human-readable text logs (development)

#### Viewing Logs

```bash
# Docker logs
docker logs -f veza-backend-api

# Docker Compose logs
docker-compose logs -f backend-api

# Systemd logs
journalctl -u veza-backend-api -f

# Kubernetes logs
kubectl logs -f deployment/veza-backend-api -n veza
```

## Rollback Procedures

### Docker Rollback

```bash
# Stop current deployment
docker-compose -f docker-compose.production.yml down

# Tag previous image
docker tag veza/backend-api:previous veza/backend-api:latest

# Start previous version
docker-compose -f docker-compose.production.yml up -d

# Verify
curl https://api.veza.com/healthz
```

### Database Migration Rollback

```bash
# Rollback last migration
migrate -path ./migrations -database "$DATABASE_URL" down 1

# Rollback to specific version
migrate -path ./migrations -database "$DATABASE_URL" force <version>
```

### Kubernetes Rollback

```bash
# Rollback deployment
kubectl rollout undo deployment/veza-backend-api -n veza

# Rollback to specific revision
kubectl rollout undo deployment/veza-backend-api -n veza --to-revision=2

# Check rollout history
kubectl rollout history deployment/veza-backend-api -n veza
```

### Ansible Rollback

```bash
# Revert to previous playbook version
git checkout <previous-commit>
ansible-playbook -i inventory/prod/hosts.yml playbooks/40-veza-apps.yml
```

## Troubleshooting

### Common Issues

#### Application Won't Start

```bash
# Check logs
docker logs veza-backend-api
journalctl -u veza-backend-api

# Verify environment variables
docker exec veza-backend-api env | grep -E 'DATABASE|JWT|APP_ENV'

# Check database connectivity
docker exec veza-backend-api ./veza-api migrate status
```

#### Database Connection Issues

```bash
# Test database connection
psql "$DATABASE_URL" -c "SELECT 1;"

# Check database is running
docker ps | grep postgres
systemctl status postgresql

# Verify network connectivity
docker exec veza-backend-api ping postgres
```

#### High Memory Usage

```bash
# Check memory usage
docker stats veza-backend-api

# Adjust database connection pool
export DB_MAX_OPEN_CONNS=25
export DB_MAX_IDLE_CONNS=5

# Restart with new settings
docker-compose restart backend-api
```

#### Slow Response Times

```bash
# Check application metrics
curl http://localhost:8080/metrics | grep http_request_duration

# Check database query times
# Enable slow query log in PostgreSQL

# Check Redis connectivity
redis-cli -h redis.veza.internal ping
```

### Debug Mode

Enable debug logging:

```bash
export LOG_LEVEL=DEBUG
export LOG_FORMAT=text
```

### Performance Tuning

#### Database Connection Pool

```bash
DB_MAX_OPEN_CONNS=50
DB_MAX_IDLE_CONNS=10
DB_CONN_MAX_LIFETIME=5m
```

#### Application Resources

```yaml
# Kubernetes resources
resources:
  requests:
    memory: "1Gi"
    cpu: "1000m"
  limits:
    memory: "2Gi"
    cpu: "2000m"
```

## Maintenance

### Regular Maintenance Tasks

#### Daily

- Monitor health checks
- Review error logs
- Check disk space
- Verify backups

#### Weekly

- Review performance metrics
- Check for security updates
- Review database size
- Clean up old logs

#### Monthly

- Update dependencies
- Review and optimize database
- Review and update documentation
- Security audit

### Backup Procedures

#### Database Backup

```bash
# Automated backup script
#!/bin/bash
BACKUP_DIR="/var/backups/veza"
DATE=$(date +%Y%m%d_%H%M%S)
pg_dump "$DATABASE_URL" > "$BACKUP_DIR/veza_db_$DATE.sql"
gzip "$BACKUP_DIR/veza_db_$DATE.sql"

# Keep only last 30 days
find "$BACKUP_DIR" -name "veza_db_*.sql.gz" -mtime +30 -delete
```

#### Application Backup

```bash
# Backup uploads directory
tar -czf /var/backups/veza/uploads_$(date +%Y%m%d).tar.gz /var/lib/veza/uploads

# Backup configuration
cp .env.production /var/backups/veza/env_$(date +%Y%m%d).env
```

### Update Procedures

#### Minor Updates

```bash
# Pull latest image
docker pull veza/backend-api:latest

# Rolling update
docker-compose -f docker-compose.production.yml up -d --no-deps backend-api

# Verify
curl https://api.veza.com/healthz
```

#### Major Updates

1. Backup database
2. Test in staging
3. Schedule maintenance window
4. Deploy new version
5. Run migrations
6. Verify functionality
7. Monitor for issues

### Security Updates

```bash
# Update base image
docker pull golang:1.23-alpine

# Rebuild with security updates
docker build --no-cache -f Dockerfile.production -t veza/backend-api:latest .

# Scan for vulnerabilities
docker scan veza/backend-api:latest
```

## Additional Resources

- **API Documentation**: See `docs/API_DOCUMENTATION.md`
- **Development Guide**: See `README.md`
- **Ansible Deployment**: See `ansible/DEPLOYMENT_GUIDE.md`
- **Kubernetes Manifests**: See `k8s/` directory
- **Monitoring Setup**: See `ops/prometheus/` directory

## Support

For deployment issues:
- **Email**: ops@veza.com
- **Documentation**: https://docs.veza.com
- **Issues**: https://github.com/veza/veza-backend-api/issues