veza/k8s/secrets/README.md

Secrets Management for Veza Platform

This directory contains configurations and documentation for managing secrets in the Veza platform using Kubernetes Secrets, External Secrets Operator, and optionally HashiCorp Vault.

Overview

Veza uses a multi-layered approach to secrets management:

1. Kubernetes Secrets (Basic): For simple, static secrets
2. External Secrets Operator (Recommended): For dynamic secrets from external providers
3. HashiCorp Vault (Advanced): For enterprise-grade secrets management with rotation

Architecture

┌─────────────────────────────────────────────────────────┐
│                    External Providers                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────┐ │
│  │   Vault  │  │   AWS    │  │  GCP     │  │  Azure │ │
│  │          │  │ Secrets  │  │ Secrets  │  │ KeyVault│ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘ │
└───────┼─────────────┼─────────────┼─────────────┼──────┘
        │             │             │             │
        └─────────────┴─────────────┴─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │  External Secrets Operator │
        └─────────────┬─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │    Kubernetes Secrets      │
        └─────────────┬─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │      Veza Pods            │
        │  (Backend, Frontend, etc.) │
        └───────────────────────────┘

Quick Start

Option 1: Basic Kubernetes Secrets (Development)

# Create secrets manually
kubectl create secret generic veza-secrets \
  --from-literal=database-url='postgresql://user:pass@host:5432/db' \
  --from-literal=jwt-secret='your-secret-key-min-32-chars' \
  --from-literal=redis-url='redis://host:6379' \
  -n veza-development

# Or from file
kubectl create secret generic veza-secrets \
  --from-env-file=secrets.env \
  -n veza-development

Option 2: External Secrets Operator (Recommended for Production)

# 1. Install External Secrets Operator
kubectl apply -f k8s/secrets/external-secrets-operator.yaml

# 2. Configure secret store (e.g., Vault)
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml

# 3. Create ExternalSecret resources
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml

Secret Stores

HashiCorp Vault

Vault provides enterprise-grade secrets management with:

• Automatic secret rotation
• Audit logging
• Fine-grained access control
• Dynamic secrets

Setup:

# 1. Install Vault (if not already installed)
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -n vault-system --create-namespace

# 2. Configure Vault store for External Secrets
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml

# 3. Create secrets in Vault
vault kv put secret/veza/production \
  database-url="postgresql://..." \
  jwt-secret="..." \
  redis-url="redis://..."

# 4. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml

AWS Secrets Manager

# 1. Configure AWS store
kubectl apply -f k8s/secrets/secret-stores/aws-store.yaml

# 2. Create secrets in AWS Secrets Manager
aws secretsmanager create-secret \
  --name veza/production/database-url \
  --secret-string "postgresql://..."

# 3. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-aws.yaml

Google Cloud Secret Manager

# 1. Configure GCP store
kubectl apply -f k8s/secrets/secret-stores/gcp-store.yaml

# 2. Create secrets in GCP
gcloud secrets create veza-database-url --data-file=- <<< "postgresql://..."

# 3. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-gcp.yaml

Secret Structure

Required Secrets

All Veza services require the following secrets:

Secret Key	Description	Example
database-url	PostgreSQL connection string	postgresql://user:pass@host:5432/veza?sslmode=require
redis-url	Redis connection string	redis://host:6379/0
jwt-secret	JWT signing secret (min 32 chars)	your-super-secret-jwt-key-min-32-chars-long

Backend API Additional Secrets

Secret Key	Description	Example
stripe-api-key	Stripe API key for payments	sk_live_...
stripe-webhook-secret	Stripe webhook signing secret	whsec_...
smtp-password	SMTP password for emails	password
s3-access-key	AWS S3 access key	AKIA...
s3-secret-key	AWS S3 secret key	...

Stream Server Additional Secrets

Secret Key	Description	Example
stream-server-secret	Secret for stream server authentication	stream-secret-key

Environment-Specific Secrets

Secrets are organized by environment:

secret/
├── veza/
│   ├── development/
│   │   ├── database-url
│   │   ├── jwt-secret
│   │   └── redis-url
│   ├── staging/
│   │   ├── database-url
│   │   ├── jwt-secret
│   │   └── redis-url
│   └── production/
│       ├── database-url
│       ├── jwt-secret
│       └── redis-url

Secret Rotation

Automatic Rotation with Vault

Vault can automatically rotate secrets:

# Enable automatic rotation for database credentials
vault write database/config/veza \
  plugin_name=postgresql-database-plugin \
  allowed_roles="veza-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/veza" \
  username="vault" \
  password="vault-password" \
  rotation_period="24h"

Manual Rotation

# 1. Update secret in source (Vault, AWS, etc.)
vault kv put secret/veza/production/jwt-secret value="new-secret-key"

# 2. External Secrets Operator will automatically sync
# Or force sync:
kubectl annotate externalsecret veza-secrets \
  force-sync=$(date +%s) \
  -n veza-production

# 3. Restart pods to pick up new secrets
kubectl rollout restart deployment/veza-backend-api -n veza-production

Rotation CronJob

A CronJob is provided to rotate secrets periodically:

kubectl apply -f k8s/secrets/secrets-rotation.yaml

Security Best Practices

1. Never commit secrets to Git

   • Use .gitignore for secret files
   • Use secrets.yaml.example as template

2. Use separate secrets per environment

   • Different secrets for dev, staging, production
   • Use namespaces to isolate environments

3. Rotate secrets regularly

   • JWT secrets: Every 90 days
   • Database passwords: Every 180 days
   • API keys: As per provider recommendations

4. Limit access with RBAC

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
       name: secrets-reader
       namespace: veza-production
   rules:
       - apiGroups: [""]
         resources: ["secrets"]
         verbs: ["get", "list"]
   

5. Enable audit logging

   • Log all secret access
   • Monitor for unauthorized access

6. Use encryption at rest

   • Enable encryption for etcd (Kubernetes secrets)
   • Use encrypted volumes for Vault

7. Principle of least privilege

   • Only grant access to secrets that are needed
   • Use service accounts with minimal permissions

Troubleshooting

Secret not syncing

# Check ExternalSecret status
kubectl describe externalsecret veza-secrets -n veza-production

# Check External Secrets Operator logs
kubectl logs -n external-secrets-system deployment/external-secrets

# Check SecretStore connection
kubectl describe secretstore vault-store -n veza-production

Secret not found in pod

# Verify secret exists
kubectl get secret veza-secrets -n veza-production

# Check pod environment variables
kubectl exec -it deployment/veza-backend-api -n veza-production -- env | grep -i secret

# Verify secret is mounted
kubectl describe pod -l app=veza-backend-api -n veza-production | grep -A 5 "Mounts:"

Permission denied

# Check RBAC permissions
kubectl auth can-i get secrets --namespace=veza-production

# Check service account
kubectl get serviceaccount -n veza-production
kubectl describe serviceaccount veza-backend-api -n veza-production

Migration Guide

From Kubernetes Secrets to External Secrets

1. Install External Secrets Operator

   kubectl apply -f k8s/secrets/external-secrets-operator.yaml
   

2. Configure secret store

   kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml
   

3. Create ExternalSecret resources

   kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
   

4. Verify secrets are synced

   kubectl get externalsecret -n veza-production
   kubectl get secret veza-secrets -n veza-production
   

5. Update deployments (if needed, External Secrets creates the same secret name)

6. Remove old manual secrets (optional, after verification)

References

• External Secrets Operator Documentation
• HashiCorp Vault Documentation
• Kubernetes Secrets Documentation
• AWS Secrets Manager
• Google Cloud Secret Manager