Skip to main content

spc--booking-qs

Overview​

  • Namespace: spc--booking-qs
  • Purpose: Sapoche Booking Queue Service Backend - PRODUCTION
  • Age: ~2 years 361 days (since October 2022)
  • Status: Active - Booking queue management
  • Workloads: 2 deployments (both at 3 replicas) - Excellent HA
  • Environment: PRODUCTION - Booking queue processing

Architecture​

Booking queue system with excellent high availability:

  • Main Application: REST API backend (3 replicas) - Excellent HA
  • Event Consumer: Booking topic consumer (3 replicas) - Excellent HA

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed 3 replicas for both deployments

Workload Categories​

Main Application (1 deployment with excellent HA)​

NameReplicasStatusPurpose
spc--booking-qs--be--app--prod3/3RunningBooking queue API - Excellent HA

Event Consumer (1 deployment with excellent HA)​

NameReplicasStatusPurpose
spc-booking-topic--consumer3/3RunningBooking event consumer - Excellent HA

Services​

NameTypeCluster IPPortsNodePortPurpose
spc--booking-qs--be--app--prodNodePort10.8.27.1358031798Booking queue API

Access & Management​

View all resources:​

kubectl get all -n spc--booking-qs

Check main application:​

# View app pods (3 replicas)
kubectl get pods -n spc--booking-qs | grep "app--prod"

# View logs from all replicas
kubectl logs -f deployment/spc--booking-qs--be--app--prod -n spc--booking-qs

Check consumer:​

# View consumer pods (3 replicas)
kubectl get pods -n spc--booking-qs | grep "consumer"

# View logs from all replicas
kubectl logs -f deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs

Restart services:​

# Restart main app
kubectl rollout restart deployment/spc--booking-qs--be--app--prod -n spc--booking-qs

# Restart consumer
kubectl rollout restart deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs

# Restart all
kubectl rollout restart deployment --all -n spc--booking-qs

Monitoring​

Resource usage:​

kubectl top pods -n spc--booking-qs --sort-by=memory
kubectl top pods -n spc--booking-qs --sort-by=cpu

Events:​

kubectl get events -n spc--booking-qs --sort-by='.lastTimestamp' | head -20

Data Flow​

Booking Queue Request
    ↓
spc--booking-qs--be--app--prod (NodePort 31798)
    ↓
Booking Queue API (3 replicas - Excellent HA)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Booking Topic Consumer (3 replicas - Excellent HA)
    ↓
Process booking queue events
    ↓
Update booking status and queue

Booking Queue Workflow​

1. Booking Queue API (Excellent HA)​

  • 3 replicas provide excellent redundancy
  • Booking queue management
  • Queue status queries
  • Booking slot management
  • Queue position tracking
  • Booking availability

2. Booking Topic Consumer (Excellent HA)​

  • spc-booking-topic--consumer processes booking events with 3 replicas
  • Booking confirmation events
  • Queue updates
  • Slot allocation
  • Booking status changes
  • Real-time queue processing

Production Considerations​

High Availability​

EXCELLENT Configuration:

  • Main API: 3 replicas - Excellent HA
  • Consumer: 3 replicas - Excellent HA
  • Very mature namespace (~3 years)
  • Stable operation (66 days since update)
  • Both deployments with redundancy

Recommendations​

  1. Maintain Current Excellent HA (Already Perfect):

    • Main API: 3 replicas (excellent)
    • Consumer: 3 replicas (excellent)
    • Well-established (~3 years in production)
    • Stable updates (66 days ago)

  2. Consider Auto-Scaling (Optional Enhancement):

    • Main API: Add HPA (3-10 replicas based on traffic)
    • Consumer: Consider KEDA for queue-based scaling (3-10 replicas)
    • Current fixed 3 replicas works well but auto-scaling could optimize

  3. Recent Activity:

    • Stable: Updated 66 days ago
    • Both deployments updated together
    • Consistent and mature operation

  4. Monitoring Priorities:

    • Queue processing lag
    • Booking event consumer lag
    • API response times
    • Queue depth and wait times
    • Booking success rates

Troubleshooting​

Main API issues:​

# Check all 3 API pods
kubectl get pods -n spc--booking-qs | grep "app--prod"

# Check logs from all replicas
kubectl logs deployment/spc--booking-qs--be--app--prod -n spc--booking-qs --all-containers=true --tail=100

# Test API endpoint
kubectl port-forward -n spc--booking-qs service/spc--booking-qs--be--app--prod 8080:80

Consumer issues:​

# Check all 3 consumer pods
kubectl get pods -n spc--booking-qs | grep "consumer"

# Check logs from all replicas
kubectl logs -f deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs

# Check for consumer errors
kubectl logs deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs --tail=100 | grep -i "error\|booking\|fail"

# Restart consumer
kubectl rollout restart deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs

Queue processing issues:​

# Check consumer lag
kubectl logs deployment/spc--spc-booking-topic--consumer--prod -n spc--booking-qs --tail=100 | grep -i "lag\|queue\|pending"

# Check all pods status
kubectl get pods -n spc--booking-qs

# Restart both deployments
kubectl rollout restart deployment --all -n spc--booking-qs

Performance Metrics​

Current Scale​

  • Main API: 3 replicas - Excellent HA
  • Consumer: 3 replicas - Excellent HA
  • Total Active Pods: 6 pods

Stability​

  • Namespace Age: ~3 years (very mature)
  • Recent Update: 66 days ago (stable)
  • All Deployments: Fixed 3 replicas (excellent HA)
  • Configuration: Perfect for production booking queue