341 lines
10 KiB
Markdown
341 lines
10 KiB
Markdown
|
# Secrets Management for Veza Platform
|
||
|
|
|
||
|
|
This directory contains configurations and documentation for managing secrets in the Veza platform using Kubernetes Secrets, External Secrets Operator, and optionally HashiCorp Vault.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
Veza uses a multi-layered approach to secrets management:
|
||
|
|
|
||
|
|
1. **Kubernetes Secrets** (Basic): For simple, static secrets
|
||
|
|
2. **External Secrets Operator** (Recommended): For dynamic secrets from external providers
|
||
|
|
3. **HashiCorp Vault** (Advanced): For enterprise-grade secrets management with rotation
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────┐
|
||
|
|
│ External Providers │
|
||
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │
|
||
|
|
│ │ Vault │ │ AWS │ │ GCP │ │ Azure │ │
|
||
|
|
│ │ │ │ Secrets │ │ Secrets │ │ KeyVault│ │
|
||
|
|
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬────┘ │
|
||
|
|
└───────┼─────────────┼─────────────┼─────────────┼──────┘
|
||
|
|
│ │ │ │
|
||
|
|
└─────────────┴─────────────┴─────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼─────────────┐
|
||
|
|
│ External Secrets Operator │
|
||
|
|
└─────────────┬─────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼─────────────┐
|
||
|
|
│ Kubernetes Secrets │
|
||
|
|
└─────────────┬─────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼─────────────┐
|
||
|
|
│ Veza Pods │
|
||
|
|
│ (Backend, Frontend, etc.) │
|
||
|
|
└───────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
### Option 1: Basic Kubernetes Secrets (Development)
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Create secrets manually
|
||
|
|
kubectl create secret generic veza-secrets \
|
||
|
|
--from-literal=database-url='postgresql://user:pass@host:5432/db' \
|
||
|
|
--from-literal=jwt-secret='your-secret-key-min-32-chars' \
|
||
|
|
--from-literal=redis-url='redis://host:6379' \
|
||
|
|
-n veza-development
|
||
|
|
|
||
|
|
# Or from file
|
||
|
|
kubectl create secret generic veza-secrets \
|
||
|
|
--from-env-file=secrets.env \
|
||
|
|
-n veza-development
|
||
|
|
```
|
||
|
|
|
||
|
|
### Option 2: External Secrets Operator (Recommended for Production)
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Install External Secrets Operator
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets-operator.yaml
|
||
|
|
|
||
|
|
# 2. Configure secret store (e.g., Vault)
|
||
|
|
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml
|
||
|
|
|
||
|
|
# 3. Create ExternalSecret resources
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
## Secret Stores
|
||
|
|
|
||
|
|
### HashiCorp Vault
|
||
|
|
|
||
|
|
Vault provides enterprise-grade secrets management with:
|
||
|
|
- Automatic secret rotation
|
||
|
|
- Audit logging
|
||
|
|
- Fine-grained access control
|
||
|
|
- Dynamic secrets
|
||
|
|
|
||
|
|
**Setup:**
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Install Vault (if not already installed)
|
||
|
|
helm repo add hashicorp https://helm.releases.hashicorp.com
|
||
|
|
helm install vault hashicorp/vault -n vault-system --create-namespace
|
||
|
|
|
||
|
|
# 2. Configure Vault store for External Secrets
|
||
|
|
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml
|
||
|
|
|
||
|
|
# 3. Create secrets in Vault
|
||
|
|
vault kv put secret/veza/production \
|
||
|
|
database-url="postgresql://..." \
|
||
|
|
jwt-secret="..." \
|
||
|
|
redis-url="redis://..."
|
||
|
|
|
||
|
|
# 4. Create ExternalSecret
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
### AWS Secrets Manager
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Configure AWS store
|
||
|
|
kubectl apply -f k8s/secrets/secret-stores/aws-store.yaml
|
||
|
|
|
||
|
|
# 2. Create secrets in AWS Secrets Manager
|
||
|
|
aws secretsmanager create-secret \
|
||
|
|
--name veza/production/database-url \
|
||
|
|
--secret-string "postgresql://..."
|
||
|
|
|
||
|
|
# 3. Create ExternalSecret
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-aws.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
### Google Cloud Secret Manager
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Configure GCP store
|
||
|
|
kubectl apply -f k8s/secrets/secret-stores/gcp-store.yaml
|
||
|
|
|
||
|
|
# 2. Create secrets in GCP
|
||
|
|
gcloud secrets create veza-database-url --data-file=- <<< "postgresql://..."
|
||
|
|
|
||
|
|
# 3. Create ExternalSecret
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-gcp.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
## Secret Structure
|
||
|
|
|
||
|
|
### Required Secrets
|
||
|
|
|
||
|
|
All Veza services require the following secrets:
|
||
|
|
|
||
|
|
| Secret Key | Description | Example |
|
||
|
|
|------------|-------------|---------|
|
||
|
|
| `database-url` | PostgreSQL connection string | `postgresql://user:pass@host:5432/veza?sslmode=require` |
|
||
|
|
| `redis-url` | Redis connection string | `redis://host:6379/0` |
|
||
|
|
| `jwt-secret` | JWT signing secret (min 32 chars) | `your-super-secret-jwt-key-min-32-chars-long` |
|
||
|
|
|
||
|
|
### Backend API Additional Secrets
|
||
|
|
|
||
|
|
| Secret Key | Description | Example |
|
||
|
|
|------------|-------------|---------|
|
||
|
|
| `stripe-api-key` | Stripe API key for payments | `sk_live_...` |
|
||
|
|
| `stripe-webhook-secret` | Stripe webhook signing secret | `whsec_...` |
|
||
|
|
| `smtp-password` | SMTP password for emails | `password` |
|
||
|
|
| `s3-access-key` | AWS S3 access key | `AKIA...` |
|
||
|
|
| `s3-secret-key` | AWS S3 secret key | `...` |
|
||
|
|
|
||
|
|
### Chat Server Additional Secrets
|
||
|
|
|
||
|
|
| Secret Key | Description | Example |
|
||
|
|
|------------|-------------|---------|
|
||
|
|
| `chat-server-secret` | Secret for chat server authentication | `chat-secret-key` |
|
||
|
|
|
||
|
|
### Stream Server Additional Secrets
|
||
|
|
|
||
|
|
| Secret Key | Description | Example |
|
||
|
|
|------------|-------------|---------|
|
||
|
|
| `stream-server-secret` | Secret for stream server authentication | `stream-secret-key` |
|
||
|
|
|
||
|
|
## Environment-Specific Secrets
|
||
|
|
|
||
|
|
Secrets are organized by environment:
|
||
|
|
|
||
|
|
```
|
||
|
|
secret/
|
||
|
|
├── veza/
|
||
|
|
│ ├── development/
|
||
|
|
│ │ ├── database-url
|
||
|
|
│ │ ├── jwt-secret
|
||
|
|
│ │ └── redis-url
|
||
|
|
│ ├── staging/
|
||
|
|
│ │ ├── database-url
|
||
|
|
│ │ ├── jwt-secret
|
||
|
|
│ │ └── redis-url
|
||
|
|
│ └── production/
|
||
|
|
│ ├── database-url
|
||
|
|
│ ├── jwt-secret
|
||
|
|
│ └── redis-url
|
||
|
|
```
|
||
|
|
|
||
|
|
## Secret Rotation
|
||
|
|
|
||
|
|
### Automatic Rotation with Vault
|
||
|
|
|
||
|
|
Vault can automatically rotate secrets:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Enable automatic rotation for database credentials
|
||
|
|
vault write database/config/veza \
|
||
|
|
plugin_name=postgresql-database-plugin \
|
||
|
|
allowed_roles="veza-role" \
|
||
|
|
connection_url="postgresql://{{username}}:{{password}}@postgres:5432/veza" \
|
||
|
|
username="vault" \
|
||
|
|
password="vault-password" \
|
||
|
|
rotation_period="24h"
|
||
|
|
```
|
||
|
|
|
||
|
|
### Manual Rotation
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Update secret in source (Vault, AWS, etc.)
|
||
|
|
vault kv put secret/veza/production/jwt-secret value="new-secret-key"
|
||
|
|
|
||
|
|
# 2. External Secrets Operator will automatically sync
|
||
|
|
# Or force sync:
|
||
|
|
kubectl annotate externalsecret veza-secrets \
|
||
|
|
force-sync=$(date +%s) \
|
||
|
|
-n veza-production
|
||
|
|
|
||
|
|
# 3. Restart pods to pick up new secrets
|
||
|
|
kubectl rollout restart deployment/veza-backend-api -n veza-production
|
||
|
|
```
|
||
|
|
|
||
|
|
### Rotation CronJob
|
||
|
|
|
||
|
|
A CronJob is provided to rotate secrets periodically:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
kubectl apply -f k8s/secrets/secrets-rotation.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
## Security Best Practices
|
||
|
|
|
||
|
|
1. **Never commit secrets to Git**
|
||
|
|
- Use `.gitignore` for secret files
|
||
|
|
- Use `secrets.yaml.example` as template
|
||
|
|
|
||
|
|
2. **Use separate secrets per environment**
|
||
|
|
- Different secrets for dev, staging, production
|
||
|
|
- Use namespaces to isolate environments
|
||
|
|
|
||
|
|
3. **Rotate secrets regularly**
|
||
|
|
- JWT secrets: Every 90 days
|
||
|
|
- Database passwords: Every 180 days
|
||
|
|
- API keys: As per provider recommendations
|
||
|
|
|
||
|
|
4. **Limit access with RBAC**
|
||
|
|
```yaml
|
||
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
||
|
|
kind: Role
|
||
|
|
metadata:
|
||
|
|
name: secrets-reader
|
||
|
|
namespace: veza-production
|
||
|
|
rules:
|
||
|
|
- apiGroups: [""]
|
||
|
|
resources: ["secrets"]
|
||
|
|
verbs: ["get", "list"]
|
||
|
|
```
|
||
|
|
|
||
|
|
5. **Enable audit logging**
|
||
|
|
- Log all secret access
|
||
|
|
- Monitor for unauthorized access
|
||
|
|
|
||
|
|
6. **Use encryption at rest**
|
||
|
|
- Enable encryption for etcd (Kubernetes secrets)
|
||
|
|
- Use encrypted volumes for Vault
|
||
|
|
|
||
|
|
7. **Principle of least privilege**
|
||
|
|
- Only grant access to secrets that are needed
|
||
|
|
- Use service accounts with minimal permissions
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
### Secret not syncing
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check ExternalSecret status
|
||
|
|
kubectl describe externalsecret veza-secrets -n veza-production
|
||
|
|
|
||
|
|
# Check External Secrets Operator logs
|
||
|
|
kubectl logs -n external-secrets-system deployment/external-secrets
|
||
|
|
|
||
|
|
# Check SecretStore connection
|
||
|
|
kubectl describe secretstore vault-store -n veza-production
|
||
|
|
```
|
||
|
|
|
||
|
|
### Secret not found in pod
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Verify secret exists
|
||
|
|
kubectl get secret veza-secrets -n veza-production
|
||
|
|
|
||
|
|
# Check pod environment variables
|
||
|
|
kubectl exec -it deployment/veza-backend-api -n veza-production -- env | grep -i secret
|
||
|
|
|
||
|
|
# Verify secret is mounted
|
||
|
|
kubectl describe pod -l app=veza-backend-api -n veza-production | grep -A 5 "Mounts:"
|
||
|
|
```
|
||
|
|
|
||
|
|
### Permission denied
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check RBAC permissions
|
||
|
|
kubectl auth can-i get secrets --namespace=veza-production
|
||
|
|
|
||
|
|
# Check service account
|
||
|
|
kubectl get serviceaccount -n veza-production
|
||
|
|
kubectl describe serviceaccount veza-backend-api -n veza-production
|
||
|
|
```
|
||
|
|
|
||
|
|
## Migration Guide
|
||
|
|
|
||
|
|
### From Kubernetes Secrets to External Secrets
|
||
|
|
|
||
|
|
1. **Install External Secrets Operator**
|
||
|
|
```bash
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets-operator.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Configure secret store**
|
||
|
|
```bash
|
||
|
|
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Create ExternalSecret resources**
|
||
|
|
```bash
|
||
|
|
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
4. **Verify secrets are synced**
|
||
|
|
```bash
|
||
|
|
kubectl get externalsecret -n veza-production
|
||
|
|
kubectl get secret veza-secrets -n veza-production
|
||
|
|
```
|
||
|
|
|
||
|
|
5. **Update deployments** (if needed, External Secrets creates the same secret name)
|
||
|
|
|
||
|
|
6. **Remove old manual secrets** (optional, after verification)
|
||
|
|
|
||
|
|
## References
|
||
|
|
|
||
|
|
- [External Secrets Operator Documentation](https://external-secrets.io/)
|
||
|
|
- [HashiCorp Vault Documentation](https://www.vaultproject.io/docs)
|
||
|
|
- [Kubernetes Secrets Documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||
|
|
- [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/)
|
||
|
|
- [Google Cloud Secret Manager](https://cloud.google.com/secret-manager)
|
||
|
|
|