veza/k8s/certificates

SSL/TLS Certificate Management

This directory contains Kubernetes manifests for managing SSL/TLS certificates using cert-manager and Let's Encrypt.

Components

cert-manager

• Purpose: Automated certificate management
• Namespace: cert-manager
• Installation: Via official cert-manager release

ClusterIssuers

• letsencrypt-prod: Production Let's Encrypt issuer
• letsencrypt-staging: Staging Let's Encrypt issuer (for testing)

Prerequisites

1. Kubernetes cluster with ingress controller (nginx-ingress recommended)
2. cert-manager installed in the cluster
3. DNS records pointing to your ingress controller
4. Ingress controller with HTTP-01 challenge support

Installation

1. Install cert-manager

# Install cert-manager CRDs
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.crds.yaml

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Or using Helm
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

2. Verify cert-manager Installation

# Check cert-manager pods
kubectl get pods -n cert-manager

# Check ClusterIssuers
kubectl get clusterissuers

3. Configure Let's Encrypt Issuers

Important: Update the email address in letsencrypt-issuer.yaml:

email: ops@veza.com  # Change to your email

Then apply:

kubectl apply -f k8s/certificates/letsencrypt-issuer.yaml

4. Verify ClusterIssuers

# Check ClusterIssuers status
kubectl get clusterissuers

# Describe to see details
kubectl describe clusterissuer letsencrypt-prod

Usage

Automatic Certificate via Ingress

The easiest way is to use Ingress annotations. The ingress is already configured in k8s/ingress.yaml:

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - app.veza.com
    - api.veza.com
    secretName: veza-tls

When you apply the ingress, cert-manager will automatically:

1. Create a Certificate resource
2. Request certificate from Let's Encrypt
3. Store it in the specified secret
4. Renew it automatically before expiration

Manual Certificate Request

You can also create a Certificate resource manually:

kubectl apply -f k8s/certificates/certificate-example.yaml

Verification

Check Certificate Status

# List certificates
kubectl get certificates -n veza-production

# Describe certificate to see status
kubectl describe certificate veza-tls -n veza-production

# Check certificate events
kubectl get events -n veza-production --field-selector involvedObject.name=veza-tls

Check Certificate Secret

# List TLS secrets
kubectl get secrets -n veza-production | grep tls

# View certificate details (base64 encoded)
kubectl get secret veza-tls -n veza-production -o yaml

Test Certificate

# Port forward to ingress
kubectl port-forward service/ingress-nginx-controller 8443:443 -n ingress-nginx

# Test with curl
curl -v https://localhost:8443 -k --resolve app.veza.com:8443:127.0.0.1

# Or check certificate expiration
echo | openssl s_client -servername app.veza.com -connect app.veza.com:443 2>/dev/null | openssl x509 -noout -dates

Staging vs Production

Use Staging for Testing

When testing certificate issuance, use the staging issuer first:

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-staging

Staging certificates:

• Don't count against rate limits
• Are not trusted by browsers (expected)
• Useful for testing the setup

Switch to Production

Once staging works, switch to production:

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-prod

Troubleshooting

Certificate Not Issued

1. Check cert-manager logs:

   kubectl logs -n cert-manager deployment/cert-manager
   kubectl logs -n cert-manager deployment/cert-manager-webhook
   kubectl logs -n cert-manager deployment/cert-manager-cainjector
   

2. Check Certificate status:

   kubectl describe certificate veza-tls -n veza-production
   

3. Check CertificateRequest:

   kubectl get certificaterequests -n veza-production
   kubectl describe certificaterequest <name> -n veza-production
   

4. Check Challenge:

   kubectl get challenges -n veza-production
   kubectl describe challenge <name> -n veza-production
   

Common Issues

Issue: Certificate pending

• Cause: DNS not pointing to ingress, or ingress controller not accessible
• Solution: Verify DNS records and ingress controller accessibility

Issue: Rate limit exceeded

• Cause: Too many certificate requests to Let's Encrypt
• Solution: Use staging issuer for testing, wait for rate limit reset

Issue: HTTP-01 challenge failing

• Cause: Ingress not accessible on port 80, or wrong ingress class
• Solution: Verify ingress controller is accessible and annotation matches

Issue: Certificate secret not created

• Cause: Certificate request failed
• Solution: Check certificate status and cert-manager logs

Debug Commands

# Check all cert-manager resources
kubectl get certificates,certificaterequests,challenges -n veza-production

# Check ClusterIssuer status
kubectl describe clusterissuer letsencrypt-prod

# Check ingress annotations
kubectl get ingress veza-ingress -n veza-production -o yaml

# Check cert-manager pods
kubectl get pods -n cert-manager

# View cert-manager controller logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100

Certificate Renewal

cert-manager automatically renews certificates before expiration (default: 30 days before expiry).

Manual Renewal

# Delete the certificate to force renewal
kubectl delete certificate veza-tls -n veza-production

# cert-manager will automatically recreate it

Check Expiration

# View certificate expiration
kubectl get certificate veza-tls -n veza-production -o jsonpath='{.status.notAfter}'

# Or decode and view
kubectl get secret veza-tls -n veza-production -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

Best Practices

1. Use staging first: Test with staging issuer before production
2. Monitor certificates: Set up alerts for certificate expiration
3. Backup secrets: Backup certificate secrets for disaster recovery
4. Multiple domains: Use wildcard certificates for multiple subdomains
5. DNS-01 for wildcards: Use DNS-01 challenge for wildcard certificates

Additional Resources

• cert-manager Documentation
• Let's Encrypt Documentation
• cert-manager GitHub