veza/k8s/monitoring/README.md

Monitoring and Logging Setup

This directory contains Kubernetes manifests for monitoring and logging infrastructure.

Components

Prometheus

• Purpose: Metrics collection and alerting
• Port: 9090
• Storage: 50Gi PVC
• Retention: 30 days

Grafana

• Purpose: Metrics visualization and dashboards
• Port: 3000
• Storage: 10Gi PVC
• Default User: admin (password from secret)

Loki

• Purpose: Log aggregation
• Port: 3100
• Storage: 50Gi PVC
• Retention: 30 days

Promtail

• Purpose: Log collection agent (DaemonSet)
• Port: 9080
• Collects: Pod logs from all nodes

Deployment

1. Deploy Prometheus

kubectl apply -f k8s/monitoring/prometheus-configmap.yaml
kubectl apply -f k8s/monitoring/prometheus-deployment.yaml

2. Deploy Grafana

kubectl apply -f k8s/monitoring/grafana-deployment.yaml

Note: Make sure to set grafana-password in veza-secrets:

kubectl create secret generic veza-secrets \
  --from-literal=grafana-password=your-secure-password \
  -n veza-production \
  --dry-run=client -o yaml | kubectl apply -f -

3. Deploy Loki

kubectl apply -f k8s/monitoring/loki-deployment.yaml

4. Deploy Promtail

kubectl apply -f k8s/monitoring/promtail-deployment.yaml

Access

Prometheus

kubectl port-forward service/prometheus 9090:9090 -n veza-production
# Access at http://localhost:9090

Grafana

kubectl port-forward service/grafana 3000:3000 -n veza-production
# Access at http://localhost:3000
# Default credentials: admin / (from secret)

Loki

kubectl port-forward service/loki 3100:3100 -n veza-production
# Access at http://localhost:3100

Integration with Services

All services should expose metrics at /metrics endpoint. Prometheus will automatically discover and scrape them using Kubernetes service discovery.

Adding Metrics to Services

1. Backend API (Go): Already has Prometheus metrics via internal/metrics/prometheus.go
2. Chat Server (Rust): Already has Prometheus metrics
3. Stream Server (Rust): Already has Prometheus metrics

Viewing Logs in Grafana

1. Add Loki as a data source in Grafana:

   • URL: http://loki:3100
   • Access: Server (default)

2. Use LogQL queries:

   {namespace="veza-production", app="veza-backend-api"}
   

Dashboards

Grafana will automatically provision dashboards from ConfigMaps. To add custom dashboards:

1. Create a ConfigMap with dashboard JSON
2. Mount it in Grafana deployment
3. Grafana will auto-discover and load it

Alerts

Prometheus alerting rules can be added via ConfigMap. Create rules files and mount them in Prometheus deployment.

Troubleshooting

Check Prometheus Targets

kubectl port-forward service/prometheus 9090:9090 -n veza-production
# Visit http://localhost:9090/targets

Check Promtail Logs

kubectl logs -f daemonset/promtail -n veza-production

Check Loki Logs

kubectl logs -f deployment/loki -n veza-production

Verify Service Discovery

kubectl get pods -n veza-production -l app=veza-backend-api
kubectl get pods -n veza-production -l app=veza-chat-server