# Secrets Management for Veza Platform This directory contains configurations and documentation for managing secrets in the Veza platform using Kubernetes Secrets, External Secrets Operator, and optionally HashiCorp Vault. ## Overview Veza uses a multi-layered approach to secrets management: 1. **Kubernetes Secrets** (Basic): For simple, static secrets 2. **External Secrets Operator** (Recommended): For dynamic secrets from external providers 3. **HashiCorp Vault** (Advanced): For enterprise-grade secrets management with rotation ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ External Providers │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │ │ │ Vault │ │ AWS │ │ GCP │ │ Azure │ │ │ │ │ │ Secrets │ │ Secrets │ │ KeyVault│ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬────┘ │ └───────┼─────────────┼─────────────┼─────────────┼──────┘ │ │ │ │ └─────────────┴─────────────┴─────────────┘ │ ┌─────────────▼─────────────┐ │ External Secrets Operator │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ │ Kubernetes Secrets │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ │ Veza Pods │ │ (Backend, Frontend, etc.) │ └───────────────────────────┘ ``` ## Quick Start ### Option 1: Basic Kubernetes Secrets (Development) ```bash # Create secrets manually kubectl create secret generic veza-secrets \ --from-literal=database-url='postgresql://user:pass@host:5432/db' \ --from-literal=jwt-secret='your-secret-key-min-32-chars' \ --from-literal=redis-url='redis://host:6379' \ -n veza-development # Or from file kubectl create secret generic veza-secrets \ --from-env-file=secrets.env \ -n veza-development ``` ### Option 2: External Secrets Operator (Recommended for Production) ```bash # 1. Install External Secrets Operator kubectl apply -f k8s/secrets/external-secrets-operator.yaml # 2. Configure secret store (e.g., Vault) kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml # 3. Create ExternalSecret resources kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml ``` ## Secret Stores ### HashiCorp Vault Vault provides enterprise-grade secrets management with: - Automatic secret rotation - Audit logging - Fine-grained access control - Dynamic secrets **Setup:** ```bash # 1. Install Vault (if not already installed) helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault -n vault-system --create-namespace # 2. Configure Vault store for External Secrets kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml # 3. Create secrets in Vault vault kv put secret/veza/production \ database-url="postgresql://..." \ jwt-secret="..." \ redis-url="redis://..." # 4. Create ExternalSecret kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml ``` ### AWS Secrets Manager ```bash # 1. Configure AWS store kubectl apply -f k8s/secrets/secret-stores/aws-store.yaml # 2. Create secrets in AWS Secrets Manager aws secretsmanager create-secret \ --name veza/production/database-url \ --secret-string "postgresql://..." # 3. Create ExternalSecret kubectl apply -f k8s/secrets/external-secrets/veza-secrets-aws.yaml ``` ### Google Cloud Secret Manager ```bash # 1. Configure GCP store kubectl apply -f k8s/secrets/secret-stores/gcp-store.yaml # 2. Create secrets in GCP gcloud secrets create veza-database-url --data-file=- <<< "postgresql://..." # 3. Create ExternalSecret kubectl apply -f k8s/secrets/external-secrets/veza-secrets-gcp.yaml ``` ## Secret Structure ### Required Secrets All Veza services require the following secrets: | Secret Key | Description | Example | | -------------- | --------------------------------- | ------------------------------------------------------- | | `database-url` | PostgreSQL connection string | `postgresql://user:pass@host:5432/veza?sslmode=require` | | `redis-url` | Redis connection string | `redis://host:6379/0` | | `jwt-secret` | JWT signing secret (min 32 chars) | `your-super-secret-jwt-key-min-32-chars-long` | ### Backend API Additional Secrets | Secret Key | Description | Example | | ----------------------- | ----------------------------- | ------------- | | `stripe-api-key` | Stripe API key for payments | `sk_live_...` | | `stripe-webhook-secret` | Stripe webhook signing secret | `whsec_...` | | `smtp-password` | SMTP password for emails | `password` | | `s3-access-key` | AWS S3 access key | `AKIA...` | | `s3-secret-key` | AWS S3 secret key | `...` | ### Stream Server Additional Secrets | Secret Key | Description | Example | | ---------------------- | --------------------------------------- | ------------------- | | `stream-server-secret` | Secret for stream server authentication | `stream-secret-key` | ## Environment-Specific Secrets Secrets are organized by environment: ``` secret/ ├── veza/ │ ├── development/ │ │ ├── database-url │ │ ├── jwt-secret │ │ └── redis-url │ ├── staging/ │ │ ├── database-url │ │ ├── jwt-secret │ │ └── redis-url │ └── production/ │ ├── database-url │ ├── jwt-secret │ └── redis-url ``` ## Secret Rotation ### Automatic Rotation with Vault Vault can automatically rotate secrets: ```bash # Enable automatic rotation for database credentials vault write database/config/veza \ plugin_name=postgresql-database-plugin \ allowed_roles="veza-role" \ connection_url="postgresql://{{username}}:{{password}}@postgres:5432/veza" \ username="vault" \ password="vault-password" \ rotation_period="24h" ``` ### Manual Rotation ```bash # 1. Update secret in source (Vault, AWS, etc.) vault kv put secret/veza/production/jwt-secret value="new-secret-key" # 2. External Secrets Operator will automatically sync # Or force sync: kubectl annotate externalsecret veza-secrets \ force-sync=$(date +%s) \ -n veza-production # 3. Restart pods to pick up new secrets kubectl rollout restart deployment/veza-backend-api -n veza-production ``` ### Rotation CronJob A CronJob is provided to rotate secrets periodically: ```bash kubectl apply -f k8s/secrets/secrets-rotation.yaml ``` ## Security Best Practices 1. **Never commit secrets to Git** - Use `.gitignore` for secret files - Use `secrets.yaml.example` as template 2. **Use separate secrets per environment** - Different secrets for dev, staging, production - Use namespaces to isolate environments 3. **Rotate secrets regularly** - JWT secrets: Every 90 days - Database passwords: Every 180 days - API keys: As per provider recommendations 4. **Limit access with RBAC** ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-reader namespace: veza-production rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] ``` 5. **Enable audit logging** - Log all secret access - Monitor for unauthorized access 6. **Use encryption at rest** - Enable encryption for etcd (Kubernetes secrets) - Use encrypted volumes for Vault 7. **Principle of least privilege** - Only grant access to secrets that are needed - Use service accounts with minimal permissions ## Troubleshooting ### Secret not syncing ```bash # Check ExternalSecret status kubectl describe externalsecret veza-secrets -n veza-production # Check External Secrets Operator logs kubectl logs -n external-secrets-system deployment/external-secrets # Check SecretStore connection kubectl describe secretstore vault-store -n veza-production ``` ### Secret not found in pod ```bash # Verify secret exists kubectl get secret veza-secrets -n veza-production # Check pod environment variables kubectl exec -it deployment/veza-backend-api -n veza-production -- env | grep -i secret # Verify secret is mounted kubectl describe pod -l app=veza-backend-api -n veza-production | grep -A 5 "Mounts:" ``` ### Permission denied ```bash # Check RBAC permissions kubectl auth can-i get secrets --namespace=veza-production # Check service account kubectl get serviceaccount -n veza-production kubectl describe serviceaccount veza-backend-api -n veza-production ``` ## Migration Guide ### From Kubernetes Secrets to External Secrets 1. **Install External Secrets Operator** ```bash kubectl apply -f k8s/secrets/external-secrets-operator.yaml ``` 2. **Configure secret store** ```bash kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml ``` 3. **Create ExternalSecret resources** ```bash kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml ``` 4. **Verify secrets are synced** ```bash kubectl get externalsecret -n veza-production kubectl get secret veza-secrets -n veza-production ``` 5. **Update deployments** (if needed, External Secrets creates the same secret name) 6. **Remove old manual secrets** (optional, after verification) ## References - [External Secrets Operator Documentation](https://external-secrets.io/) - [HashiCorp Vault Documentation](https://www.vaultproject.io/docs) - [Kubernetes Secrets Documentation](https://kubernetes.io/docs/concepts/configuration/secret/) - [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) - [Google Cloud Secret Manager](https://cloud.google.com/secret-manager)