apisix

Overview

• Namespace: apisix
• Purpose: API Gateway - PRODUCTION
• Age: ~216+ days (since May 2024)
• Status: Active - API Gateway and ingress controller
• Workloads: 3 deployments (all active)
• Environment: PRODUCTION - Main API gateway for cluster

Architecture

Apache APISIX API gateway with dashboard and ingress controller:

• Main Gateway: APISIX API gateway (3 replicas) - Good HA
• Dashboard: APISIX dashboard (1 replica) - UI for configuration
• Ingress Controller: APISIX ingress controller (1 replica)
• Backend: etcd key-value store for configuration

Auto-Scaling Configuration

No Auto-Scaling Configured:

• No HorizontalPodAutoscalers (HPAs)
• No KEDA scaled objects
• Fixed replica counts

Workload Categories

Main Gateway (1 deployment with good HA)

Name	Replicas	Status	Purpose
apisix	3/3	Running	API Gateway (3 replicas - Good HA)

Dashboard (1 deployment)

Name	Replicas	Status	Purpose
apisix-dashboard	1/1	Running	APISIX management dashboard

Ingress Controller (1 deployment)

Name	Replicas	Status	Purpose
apisix-ingress-controller	1/1	Running	Kubernetes ingress controller

Services

Name	Type	Cluster IP	Ports	External IP	Purpose
apisix-gateway	LoadBalancer	10.8.19.252	80, 443, 2222	34.87.114.121	API Gateway (Public)
apisix-admin	ClusterIP	10.8.22.39	9180	-	Admin API
apisix-dashboard	ClusterIP	10.8.24.10	80	-	Dashboard UI
apisix-ingress-controller	ClusterIP	10.8.19.111	80	-	Ingress controller
apisix-prometheus-metrics	ClusterIP	10.8.26.252	9091	-	Prometheus metrics

Access & Management

View all resources:

kubectl get all -n apisix

Check gateway:

# View gateway pods (3 replicas)
kubectl get pods -n apisix | grep "apisix-" | grep -v dashboard

# View logs from all replicas
kubectl logs -f deployment/apisix -n apisix

# Check metrics
kubectl logs -f deployment/apisix -n apisix | grep -i "metric\|request"

Access dashboard:

# Port forward to dashboard
kubectl port-forward -n apisix service/apisix-dashboard 8080:80
# Access http://localhost:8080

Access admin API:

# Port forward to admin API
kubectl port-forward -n apisix service/apisix-admin 9180:9180
# Access http://localhost:9180/apisix/admin

Restart services:

# Restart gateway
kubectl rollout restart deployment/apisix -n apisix

# Restart dashboard
kubectl rollout restart deployment/apisix-dashboard -n apisix

# Restart ingress controller
kubectl rollout restart deployment/apisix-ingress-controller -n apisix

Monitoring

Resource usage:

kubectl top pods -n apisix --sort-by=memory
kubectl top pods -n apisix --sort-by=cpu

Gateway metrics:

# Check metrics endpoint
kubectl port-forward -n apisix service/apisix-prometheus-metrics 9091:9091
# Access http://localhost:9091/metrics

Events:

kubectl get events -n apisix --sort-by='.lastTimestamp' | head -20

Data Flow

External Client Request
    ↓
Load Balancer (34.87.114.121)
    ↓
APISIX Gateway (3 replicas - Good HA)
    ├─ Port 80 (HTTP)
    ├─ Port 443 (HTTPS/TLS)
    └─ Port 2222 (Admin port)
    ↓
Route to Backend Services
    ↓
APISIX Admin API (Configuration)
    ↓
etcd (Configuration storage)

APISIX Gateway Workflow

1. API Gateway (Good HA)

• 3 replicas provide good redundancy
• HTTP/HTTPS traffic routing
• Rate limiting and throttling
• Authentication and authorization
• API versioning support
• Plugin architecture

2. Dashboard

• Web UI for APISIX management
• Route configuration
• Service management
• Plugin management
• User-friendly administration

3. Ingress Controller

• Kubernetes ingress resource support
• Automatic route configuration
• Service discovery
• Ingress rule synchronization

4. Admin API

• RESTful API for configuration
• Programmatic route management
• Plugin management
• Statistics and monitoring

Production Considerations

High Availability

Good Gateway Configuration:

• Main Gateway: 3 replicas (good HA)
• LoadBalancer service with public IP
• Very mature (~216+ days)
• Critical for all API traffic

Single Points of Failure:

• Dashboard: 1 replica (no HA, admin tool)
• Ingress Controller: 1 replica (no HA)
• etcd backend: Critical dependency (documented separately)

Recommendations

1. Maintain Current Gateway HA:

   • Gateway: 3 replicas (excellent)
   • Well-established in production
   • Stable operation

2. Dashboard/Ingress Controller:

   • Dashboard: 1 replica (acceptable for admin tool)
   • Ingress Controller: 1 replica (consider 2+ for HA)
   • Less critical than main gateway

3. Consider Auto-Scaling:

   • Gateway: Add HPA (3-10 replicas based on traffic)
   • Scale based on request rate and latency
   • LoadBalancer already distributes traffic

4. etcd Dependency:

   • Verify etcd namespace is healthy
   • etcd stores all APISIX configuration
   • Critical for gateway operation

5. Monitoring Priorities:

   • Request rate and latency
   • Error rates (4xx, 5xx)
   • Gateway pod health
   • etcd connectivity
   • Plugin execution time

Troubleshooting

Gateway issues:

# Check all 3 gateway pods
kubectl get pods -n apisix | grep "^apisix-"

# Check logs from all replicas
kubectl logs deployment/apisix -n apisix --all-containers=true --tail=100

# Check for errors
kubectl logs deployment/apisix -n apisix --tail=100 | grep -i "error\|fail"

# Test gateway health
kubectl port-forward -n apisix service/apisix-gateway 8080:80
curl http://localhost:8080/apisix/status

Admin API issues:

# Check admin API connectivity
kubectl port-forward -n apisix service/apisix-admin 9180:9180
curl http://localhost:9180/apisix/admin/routes

# Check for API errors
kubectl logs deployment/apisix -n apisix --tail=100 | grep -i "admin\|api"

Dashboard issues:

# Check dashboard pod
kubectl get pods -n apisix | grep dashboard

# Check dashboard logs
kubectl logs -f deployment/apisix-dashboard -n apisix

# Check for dashboard errors
kubectl logs deployment/apisix-dashboard -n apisix --tail=100 | grep -i "error\|fail"

etcd connectivity:

# Check etcd from gateway
kubectl exec -it deployment/apisix -n apisix -- curl -v http://apisix-etcd.etcd:2379/health

# Check etcd service
kubectl get svc -n etcd apisix-etcd

Performance Metrics

Current Scale

• Gateway: 3 replicas - Good HA
• Dashboard: 1 replica (admin tool)
• Ingress Controller: 1 replica
• Total Active Pods: 5 pods

Stability

• Gateway Age: ~216+ days (very mature)
• Gateway Replicas: Fixed 3 replicas (good HA)
• Public IP: 34.87.114.121 (stable external access)
• Critical Service: Handles all ingress traffic

Architecture Notes

• Apache APISIX: Modern, cloud-native API gateway
• Plugin Architecture: Extensible with custom plugins
• etcd Backend: Configuration distributed and highly available
• LoadBalancer Service: Provides public IP for external access
• Kubernetes Native: Supports ingress controller pattern