Skip to main content

rfd--order-history

Overview​

  • Namespace: rfd--order-history
  • Purpose: Referring Doctor Order History Backend - PRODUCTION
  • Age: ~2 years 335 days (since December 2022)
  • Status: Active - Doctor order tracking and history
  • Workloads: 5 deployments (all active)
  • Environment: PRODUCTION - Order management and tracking

Architecture​

Order history system handling order creation, updates, and account synchronization:

  • Main Application: REST API backend (7 replicas) - Excellent HA with HPA
  • Event Consumers: Order creation, updates, account sync (3 deployments)
  • Worker: Background job processing (2 replicas)

Auto-Scaling Configuration​

Excellent Auto-Scaling Setup:

  • 1 Standard HPA: Main API (1-10 replicas, CPU + Memory based)
  • 2 KEDA HPAs: Queue-based consumer scaling
    • consumer-update-order (1-10 replicas)
    • consumer-cs-account-update-order (1-3 replicas)

Workload Categories​

Main Application (1 deployment with HPA)​

NameReplicasHPA StatusPurpose
rfd--order-history--be--app--prod7/7HPA Active (1-10, CPU+Memory)Main order history API

Event Consumers (3 deployments - 2 with KEDA)​

NameReplicasKEDA StatusPurpose
consumer-update-order1/1KEDA (1-10, queue depth: 3/20)Order update processing
consumer-cs-account-update-order1/1KEDA (1-3, queue depth: 0/20)Account order updates
consumer-create-booking1/1No scalingBooking creation from orders

Workers (1 deployment)​

NameReplicasStatusPurpose
wrk--default2/2RunningDefault worker queue (HA)

HPAs & KEDA Details​

Standard HPA - Main API​

Name: rfd--order-history--be--app--prod
Current: 7 replicas
Min: 1 replica
Max: 10 replicas
Metrics: CPU (7%) + Memory (191298998857m/200Mi)
Status:  Scaling actively based on load

KEDA HPA #1 - Order Updates​

Name: keda-hpa-rfd--order-history--be--consumer-update-order--prod
Current: 1 replica
Min: 1 replica
Max: 10 replicas
Queue Depth: 3/20 (avg)
Status:  Active, ready to scale

KEDA HPA #2 - Account Updates​

Name: keda-hpa-rfd--order-history--be--consumer-cs-account-update-order--prod
Current: 1 replica
Min: 1 replica
Max: 3 replicas
Queue Depth: 0/20 (avg)
Status:  Active, no current load

Services​

NameTypeCluster IPPortsNodePortPurpose
rfd--order-history--be--app--prodNodePort10.8.31.508032616Main order history API

Access & Management​

View all resources:​

kubectl get all -n rfd--order-history

Check HPAs:​

# All HPAs
kubectl get hpa -n rfd--order-history

# Main API HPA
kubectl describe hpa rfd--order-history--be--app--prod -n rfd--order-history

# KEDA HPAs
kubectl get scaledobjects -n rfd--order-history

Check main application:​

# View app pods (7 replicas with HPA)
kubectl get pods -n rfd--order-history | grep "app--prod"

# View logs from all replicas
kubectl logs -f deployment/rfd--order-history--be--app--prod -n rfd--order-history

Check consumers:​

# All consumers
kubectl get pods -n rfd--order-history | grep consumer

# Order update consumer (KEDA scaled)
kubectl logs -f deployment/rfd--order-history--be--consumer-update-order--prod -n rfd--order-history

# Account update consumer (KEDA scaled)
kubectl logs -f deployment/rfd--order-history--be--consumer-cs-account-update-order--prod -n rfd--order-history

Restart services:​

# Restart main app (HPA will maintain replicas)
kubectl rollout restart deployment/rfd--order-history--be--app--prod -n rfd--order-history

# Restart all consumers
kubectl get deployments -n rfd--order-history | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n rfd--order-history

Monitoring​

Resource usage:​

kubectl top pods -n rfd--order-history --sort-by=memory
kubectl top pods -n rfd--order-history --sort-by=cpu

HPA Status:​

# Watch HPA scaling
kubectl get hpa -n rfd--order-history -w

# Detailed HPA info
kubectl describe hpa -n rfd--order-history

Events:​

kubectl get events -n rfd--order-history --sort-by='.lastTimestamp' | head -20

Data Flow​

Doctor Order Request
    ↓
rfd--order-history--be--app--prod (NodePort 32616)
    ↓
Main Order History API (7 replicas - HPA scaling 1-10)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Consumers Process Events (KEDA auto-scaling)
    ├─ Order Updates → consumer-update-order (KEDA: 1-10 replicas)
    ├─ Account Updates → consumer-cs-account-update-order (KEDA: 1-3 replicas)
    └─ Booking Creation → consumer-create-booking
    ↓
Workers Process Background Jobs (2 replicas)
    ↓
Order history, tracking, doctor notifications

Order History Workflow​

1. Order History API (Excellent Scaling)​

  • 7 replicas currently active (HPA managing)
  • HPA scales 1-10 based on CPU + Memory
  • Doctor order queries
  • Historical order views
  • Order status tracking
  • Report generation

2. Order Update Processing (KEDA Scaled)​

  • consumer-update-order with KEDA auto-scaling
  • Scales 1-10 based on queue depth
  • Currently at 1 replica (low load)
  • Ready to scale up during peak times

3. Account Update Processing (KEDA Scaled)​

  • consumer-cs-account-update-order with KEDA auto-scaling
  • Scales 1-3 based on queue depth
  • Currently at 1 replica (no load)
  • Customer service account order updates

4. Booking Creation​

  • consumer-create-booking handles booking from orders
  • Fixed at 1 replica (no auto-scaling)
  • Consider adding KEDA if volume increases

5. Background Workers (HA)​

  • 2 replicas for redundancy
  • General background processing
  • Order notifications

Production Considerations​

High Availability​

Excellent Configuration:

  • Main API: 7 replicas with HPA (1-10) - excellent HA and scaling
  • Workers: 2 replicas for redundancy
  • KEDA auto-scaling on 2 critical consumers
  • Very mature namespace (~2 years)

Auto-Scaling Strengths:

  • Standard HPA for API (CPU + Memory based)
  • KEDA for queue-based consumer scaling
  • Proven setup (running 207 days)

Minor Recommendations:

  • consumer-create-booking: No auto-scaling (consider KEDA)

Recommendations​

  1. Maintain Current Setup (Already Excellent):

    • HPA for main API: Working well (7/10 replicas)
    • KEDA for consumers: Active and ready
    • Workers at 2 replicas: Good redundancy

  2. Consider KEDA for Booking Consumer:

    • consumer-create-booking currently fixed at 1
    • Add KEDA if booking volume increases
    • Monitor queue depth

  3. HPA Tuning (Optional):

    • Current max: 10 replicas
    • Consider increasing if needed during peak times
    • Monitor CPU/Memory thresholds

  4. Monitoring Priorities:

    • HPA scaling events (API currently at 7/10)
    • KEDA queue depths
    • Consumer lag (especially KEDA-scaled consumers)
    • Worker queue depth

  5. Recent Activity:

    • Very active: Pods restarted 6-16 days ago
    • Regular updates and maintenance
    • HPA been active for 207 days

Troubleshooting​

HPA issues:​

# Check HPA status
kubectl get hpa -n rfd--order-history

# HPA events
kubectl describe hpa rfd--order-history--be--app--prod -n rfd--order-history | grep -A 20 "Events:"

# Check if HPA is scaling correctly
kubectl get hpa rfd--order-history--be--app--prod -n rfd--order-history -w

KEDA scaling issues:​

# Check KEDA scaled objects
kubectl get scaledobjects -n rfd--order-history

# Detailed KEDA status
kubectl describe scaledobjects -n rfd--order-history

# Check KEDA triggers
kubectl get scaledobjects -n rfd--order-history -o yaml

Main API issues:​

# Check all 7 API pods
kubectl get pods -n rfd--order-history | grep "app--prod"

# Check logs from scaled replicas
kubectl logs deployment/rfd--order-history--be--app--prod -n rfd--order-history --all-containers=true --tail=100

# Test API endpoint
kubectl port-forward -n rfd--order-history service/rfd--order-history--be--app--prod 8080:80

Consumer scaling issues:​

# Check KEDA-scaled consumers
kubectl get pods -n rfd--order-history | grep consumer

# Check order update consumer (KEDA 1-10)
kubectl logs deployment/rfd--order-history--be--consumer-update-order--prod -n rfd--order-history --tail=100

# Check account update consumer (KEDA 1-3)
kubectl logs deployment/rfd--order-history--be--consumer-cs-account-update-order--prod -n rfd--order-history --tail=100

# Force KEDA scaling (check queue depth)
kubectl describe scaledobject -n rfd--order-history

Performance Metrics​

Current Scale​

  • Main API: 7/10 replicas (HPA active, CPU+Memory based)
  • Consumer Update Order: 1/10 replicas (KEDA active, queue: 3/20)
  • Consumer Account Update: 1/3 replicas (KEDA active, queue: 0/20)
  • Consumer Booking: 1 replica (no scaling)
  • Workers: 2 replicas (HA)
  • Total Active Pods: ~12 pods

Stability​

  • Namespace Age: ~2 years (very mature)
  • HPA Age: 207 days (proven scaling setup)
  • Recent Updates: 6-16 days ago (active maintenance)
  • Auto-Scaling: Excellent (Standard HPA + KEDA)