default (Monitoring)

Overview

• Namespace: default
• Purpose: Grafana Alloy monitoring stack
• Age: 299 days (monitoring components)
• Status: Active

Workloads

Deployments

Name	Replicas	Status	Age	Purpose
grafana-k8s-monitoring-alloy-events	1/1	Ready	299d	Kubernetes event collection

StatefulSets

Name	Replicas	Status	Age	Purpose
grafana-k8s-monitoring-alloy	1/1	Ready	299d	Metrics collection and processing

DaemonSets

Name	Desired	Current	Ready	Age	Purpose
grafana-k8s-monitoring-alloy-logs	2	2	2	299d	Log collection from all nodes

Services

Name	Type	Cluster IP	Ports	Purpose
grafana-k8s-monitoring-alloy	ClusterIP	10.29.122.147	Multiple (OTLP, Jaeger, Zipkin)	Metrics endpoint
grafana-k8s-monitoring-alloy-cluster	Headless	None	Multiple	Cluster communication
grafana-k8s-monitoring-alloy-events	ClusterIP	10.29.99.159	12345	Event collection
grafana-k8s-monitoring-alloy-logs	ClusterIP	10.29.137.221	12345	Log aggregation
grafana-k8s-monitoring-grafana-agent	ClusterIP	10.29.6.174	Multiple	Grafana agent

Port Details

The monitoring stack exposes multiple protocols:

• 12345: Alloy API
• 4317/4318: OpenTelemetry Protocol (OTLP) - gRPC/HTTP
• 9999: Internal metrics
• 14250: Jaeger gRPC
• 6831/6832: Jaeger thrift
• 14268: Jaeger HTTP
• 9411: Zipkin

Architecture

Cluster Resources
    ↓
grafana-k8s-monitoring-alloy-logs (DaemonSet on each node)
    ↓ (logs)
grafana-k8s-monitoring-alloy-events (Deployment)
    ↓ (events)
grafana-k8s-monitoring-alloy (StatefulSet)
    ↓ (metrics, logs, traces)
Grafana Cloud / Prometheus / Loki

Monitoring Coverage

1. Logs: DaemonSet on all nodes collects container logs
2. Events: Deployment watches Kubernetes events
3. Metrics: StatefulSet collects cluster and application metrics
4. Traces: Support for OpenTelemetry, Jaeger, Zipkin

Management

View all monitoring resources:

kubectl get all -n default | grep grafana

Check DaemonSet coverage:

kubectl get daemonset grafana-k8s-monitoring-alloy-logs -n default -o wide

View logs:

# Event collector
kubectl logs -f deployment/grafana-k8s-monitoring-alloy-events -n default

# Metrics collector
kubectl logs -f statefulset/grafana-k8s-monitoring-alloy -n default

# Log collectors (on all nodes)
kubectl logs -f daemonset/grafana-k8s-monitoring-alloy-logs -n default

Restart components:

# Restart event collector
kubectl rollout restart deployment/grafana-k8s-monitoring-alloy-events -n default

# Restart metrics collector
kubectl rollout restart statefulset/grafana-k8s-monitoring-alloy -n default

# Restart log collectors
kubectl rollout restart daemonset/grafana-k8s-monitoring-alloy-logs -n default

Configuration

View ConfigMaps:

kubectl get configmaps -n default | grep grafana
kubectl describe configmap <configmap-name> -n default

View Secrets:

kubectl get secrets -n default | grep grafana

Monitoring

Resource usage:

# All monitoring pods
kubectl top pods -n default | grep grafana

# Specific components
kubectl top pod -l app.kubernetes.io/name=alloy -n default

Events:

kubectl get events -n default --sort-by='.lastTimestamp' | grep grafana

Grafana Alloy

Grafana Alloy is the next-generation distribution of Grafana Agent, providing:

• Unified Observability: Metrics, logs, and traces
• OpenTelemetry Support: Native OTLP support
• Efficient Collection: Low overhead data collection
• Flexible Pipeline: Transform and route telemetry data

Data Flow

1. DaemonSet (alloy-logs): Runs on each node, collects logs from all pods
2. Deployment (alloy-events): Watches Kubernetes events
3. StatefulSet (alloy): Central collector, processes and forwards to Grafana Cloud/Prometheus

Integration

To send application metrics/logs/traces:

Prometheus Metrics

# Application exposes metrics on /metrics
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

OpenTelemetry

# Send traces/metrics to:
# gRPC: grafana-k8s-monitoring-alloy.default.svc.cluster.local:4317
# HTTP: grafana-k8s-monitoring-alloy.default.svc.cluster.local:4318

Logs

Logs are automatically collected from stdout/stderr of all pods by the DaemonSet.

Recommendations

1. Resource Monitoring:

   • Monitor the monitoring stack itself for resource usage
   • DaemonSet runs on all nodes - ensure adequate resources

2. Data Retention:

   • Configure appropriate retention policies
   • Monitor storage usage for log volumes

3. Alerting:

   • Set up alerts for monitoring stack failures
   • Alert when DaemonSet pods are not running on all nodes

4. Performance:

   • Monitor cardinality of metrics
   • Optimize log collection filters if needed

5. Security:

   • Secure Grafana Cloud API keys in secrets
   • Review RBAC permissions for monitoring components

Troubleshooting

Logs not appearing:

# Check DaemonSet status
kubectl get daemonset grafana-k8s-monitoring-alloy-logs -n default

# Check logs from DaemonSet
kubectl logs -f daemonset/grafana-k8s-monitoring-alloy-logs -n default

# Verify pod is running on each node
kubectl get pods -n default -o wide | grep alloy-logs

Metrics not being collected:

# Check StatefulSet logs
kubectl logs -f statefulset/grafana-k8s-monitoring-alloy -n default

# Verify scrape targets configuration
kubectl describe configmap -n default | grep -A 20 "scrape_configs"

High resource usage:

# Check resource consumption
kubectl top pods -n default | grep grafana

# Review and adjust resource limits
kubectl get deployment/statefulset/daemonset -n default -o yaml

Documentation

• Grafana Alloy Documentation
• Grafana Agent (predecessor)
• OpenTelemetry