veza/k8s/secrets/README.md

# Secrets Management for Veza Platform

This directory contains configurations and documentation for managing secrets in the Veza platform using Kubernetes Secrets, External Secrets Operator, and optionally HashiCorp Vault.

## Overview

Veza uses a multi-layered approach to secrets management:

1. **Kubernetes Secrets** (Basic): For simple, static secrets
2. **External Secrets Operator** (Recommended): For dynamic secrets from external providers
3. **HashiCorp Vault** (Advanced): For enterprise-grade secrets management with rotation

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    External Providers                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────┐ │
│  │   Vault  │  │   AWS    │  │  GCP     │  │  Azure │ │
│  │          │  │ Secrets  │  │ Secrets  │  │ KeyVault│ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘ │
└───────┼─────────────┼─────────────┼─────────────┼──────┘
        │             │             │             │
        └─────────────┴─────────────┴─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │  External Secrets Operator │
        └─────────────┬─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │    Kubernetes Secrets      │
        └─────────────┬─────────────┘
                      │
        ┌─────────────▼─────────────┐
        │      Veza Pods            │
        │  (Backend, Frontend, etc.) │
        └───────────────────────────┘
```

## Quick Start

### Option 1: Basic Kubernetes Secrets (Development)

```bash
# Create secrets manually
kubectl create secret generic veza-secrets \
  --from-literal=database-url='postgresql://user:pass@host:5432/db' \
  --from-literal=jwt-secret='your-secret-key-min-32-chars' \
  --from-literal=redis-url='redis://host:6379' \
  -n veza-development

# Or from file
kubectl create secret generic veza-secrets \
  --from-env-file=secrets.env \
  -n veza-development
```

### Option 2: External Secrets Operator (Recommended for Production)

```bash
# 1. Install External Secrets Operator
kubectl apply -f k8s/secrets/external-secrets-operator.yaml

# 2. Configure secret store (e.g., Vault)
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml

# 3. Create ExternalSecret resources
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
```

## Secret Stores

### HashiCorp Vault

Vault provides enterprise-grade secrets management with:

- Automatic secret rotation
- Audit logging
- Fine-grained access control
- Dynamic secrets

**Setup:**

```bash
# 1. Install Vault (if not already installed)
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -n vault-system --create-namespace

# 2. Configure Vault store for External Secrets
kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml

# 3. Create secrets in Vault
vault kv put secret/veza/production \
  database-url="postgresql://..." \
  jwt-secret="..." \
  redis-url="redis://..."

# 4. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
```

### AWS Secrets Manager

```bash
# 1. Configure AWS store
kubectl apply -f k8s/secrets/secret-stores/aws-store.yaml

# 2. Create secrets in AWS Secrets Manager
aws secretsmanager create-secret \
  --name veza/production/database-url \
  --secret-string "postgresql://..."

# 3. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-aws.yaml
```

### Google Cloud Secret Manager

```bash
# 1. Configure GCP store
kubectl apply -f k8s/secrets/secret-stores/gcp-store.yaml

# 2. Create secrets in GCP
gcloud secrets create veza-database-url --data-file=- <<< "postgresql://..."

# 3. Create ExternalSecret
kubectl apply -f k8s/secrets/external-secrets/veza-secrets-gcp.yaml
```

## Secret Structure

### Required Secrets

All Veza services require the following secrets:

| Secret Key     | Description                       | Example                                                 |
| -------------- | --------------------------------- | ------------------------------------------------------- |
| `database-url` | PostgreSQL connection string      | `postgresql://user:pass@host:5432/veza?sslmode=require` |
| `redis-url`    | Redis connection string           | `redis://host:6379/0`                                   |
| `jwt-secret`   | JWT signing secret (min 32 chars) | `your-super-secret-jwt-key-min-32-chars-long`           |

### Backend API Additional Secrets

| Secret Key              | Description                   | Example       |
| ----------------------- | ----------------------------- | ------------- |
| `stripe-api-key`        | Stripe API key for payments   | `sk_live_...` |
| `stripe-webhook-secret` | Stripe webhook signing secret | `whsec_...`   |
| `smtp-password`         | SMTP password for emails      | `password`    |
| `s3-access-key`         | AWS S3 access key             | `AKIA...`     |
| `s3-secret-key`         | AWS S3 secret key             | `...`         |

<!-- Chat Server section removed: merged into backend-api at commit 05d02386d (v0.502). -->
<!-- Chat now uses the same JWT secret as backend-api. -->

### Stream Server Additional Secrets

| Secret Key             | Description                             | Example             |
| ---------------------- | --------------------------------------- | ------------------- |
| `stream-server-secret` | Secret for stream server authentication | `stream-secret-key` |

## Environment-Specific Secrets

Secrets are organized by environment:

```
secret/
├── veza/
│   ├── development/
│   │   ├── database-url
│   │   ├── jwt-secret
│   │   └── redis-url
│   ├── staging/
│   │   ├── database-url
│   │   ├── jwt-secret
│   │   └── redis-url
│   └── production/
│       ├── database-url
│       ├── jwt-secret
│       └── redis-url
```

## Secret Rotation

### Automatic Rotation with Vault

Vault can automatically rotate secrets:

```bash
# Enable automatic rotation for database credentials
vault write database/config/veza \
  plugin_name=postgresql-database-plugin \
  allowed_roles="veza-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/veza" \
  username="vault" \
  password="vault-password" \
  rotation_period="24h"
```

### Manual Rotation

```bash
# 1. Update secret in source (Vault, AWS, etc.)
vault kv put secret/veza/production/jwt-secret value="new-secret-key"

# 2. External Secrets Operator will automatically sync
# Or force sync:
kubectl annotate externalsecret veza-secrets \
  force-sync=$(date +%s) \
  -n veza-production

# 3. Restart pods to pick up new secrets
kubectl rollout restart deployment/veza-backend-api -n veza-production
```

### Rotation CronJob

A CronJob is provided to rotate secrets periodically:

```bash
kubectl apply -f k8s/secrets/secrets-rotation.yaml
```

## Security Best Practices

1. **Never commit secrets to Git**
    - Use `.gitignore` for secret files
    - Use `secrets.yaml.example` as template

2. **Use separate secrets per environment**
    - Different secrets for dev, staging, production
    - Use namespaces to isolate environments

3. **Rotate secrets regularly**
    - JWT secrets: Every 90 days
    - Database passwords: Every 180 days
    - API keys: As per provider recommendations

4. **Limit access with RBAC**

    ```yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
        name: secrets-reader
        namespace: veza-production
    rules:
        - apiGroups: [""]
          resources: ["secrets"]
          verbs: ["get", "list"]
    ```

5. **Enable audit logging**
    - Log all secret access
    - Monitor for unauthorized access

6. **Use encryption at rest**
    - Enable encryption for etcd (Kubernetes secrets)
    - Use encrypted volumes for Vault

7. **Principle of least privilege**
    - Only grant access to secrets that are needed
    - Use service accounts with minimal permissions

## Troubleshooting

### Secret not syncing

```bash
# Check ExternalSecret status
kubectl describe externalsecret veza-secrets -n veza-production

# Check External Secrets Operator logs
kubectl logs -n external-secrets-system deployment/external-secrets

# Check SecretStore connection
kubectl describe secretstore vault-store -n veza-production
```

### Secret not found in pod

```bash
# Verify secret exists
kubectl get secret veza-secrets -n veza-production

# Check pod environment variables
kubectl exec -it deployment/veza-backend-api -n veza-production -- env | grep -i secret

# Verify secret is mounted
kubectl describe pod -l app=veza-backend-api -n veza-production | grep -A 5 "Mounts:"
```

### Permission denied

```bash
# Check RBAC permissions
kubectl auth can-i get secrets --namespace=veza-production

# Check service account
kubectl get serviceaccount -n veza-production
kubectl describe serviceaccount veza-backend-api -n veza-production
```

## Migration Guide

### From Kubernetes Secrets to External Secrets

1. **Install External Secrets Operator**

    ```bash
    kubectl apply -f k8s/secrets/external-secrets-operator.yaml
    ```

2. **Configure secret store**

    ```bash
    kubectl apply -f k8s/secrets/secret-stores/vault-store.yaml
    ```

3. **Create ExternalSecret resources**

    ```bash
    kubectl apply -f k8s/secrets/external-secrets/veza-secrets.yaml
    ```

4. **Verify secrets are synced**

    ```bash
    kubectl get externalsecret -n veza-production
    kubectl get secret veza-secrets -n veza-production
    ```

5. **Update deployments** (if needed, External Secrets creates the same secret name)

6. **Remove old manual secrets** (optional, after verification)

## References

- [External Secrets Operator Documentation](https://external-secrets.io/)
- [HashiCorp Vault Documentation](https://www.vaultproject.io/docs)
- [Kubernetes Secrets Documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
- [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/)
- [Google Cloud Secret Manager](https://cloud.google.com/secret-manager)