spc--delivery--be

Overview

• Namespace: spc--delivery--be
• Purpose: Sapoche Delivery Management Backend - PRODUCTION
• Age: 232 days (~7-8 months, since February 2025)
• Status: Active - Delivery and logistics management
• Workloads: 3 deployments (all active)
• Environment: PRODUCTION - Sample delivery and logistics

Architecture

Delivery management system handling sample pickup and delivery logistics:

• Main Application: REST API backend (3 replicas) - High Availability
• Scheduler: Cron jobs for scheduled tasks (1 deployment)
• Observability: OpenTelemetry collector for tracing

Auto-Scaling Configuration

No Auto-Scaling Configured:

• No HorizontalPodAutoscalers (HPAs)
• No KEDA scaled objects
• Fixed replica counts (Main app: 3 replicas)

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
spc--delivery--be--app--prod	3/3	Running	Main delivery API (HA configured)

Scheduler (1 deployment)

Name	Replicas	Status	Purpose
cron--prod	1/1	Running	Scheduled cron jobs

Observability (1 deployment)

Name	Replicas	Status	Purpose
spc--delivery--be-otel-collector	1/1	Running	OpenTelemetry trace collector

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
spc--delivery--be--app--prod	NodePort	10.8.20.114	80	31757	Main delivery API
spc--delivery--be-otel-collector	ClusterIP	10.8.27.102	4317, 4318	-	OTLP trace collection

Access & Management

View all resources:

kubectl get all -n spc--delivery--be

Check main application:

# View app pods (3 replicas)
kubectl get pods -n spc--delivery--be | grep "app--prod"

# View logs from all replicas
kubectl logs -f deployment/spc--delivery--be--app--prod -n spc--delivery--be

# Check specific replica
kubectl logs -f deployment/spc--delivery--be--app--prod -n spc--delivery--be --all-containers=true

Check cron jobs:

# View cron pod
kubectl get pods -n spc--delivery--be | grep cron

# Cron logs
kubectl logs -f deployment/spc--delivery--be--cron--prod -n spc--delivery--be

Restart services:

# Restart main app (all 3 replicas)
kubectl rollout restart deployment/spc--delivery--be--app--prod -n spc--delivery--be

# Restart cron
kubectl rollout restart deployment/spc--delivery--be--cron--prod -n spc--delivery--be

Monitoring

Resource usage:

kubectl top pods -n spc--delivery--be --sort-by=memory
kubectl top pods -n spc--delivery--be --sort-by=cpu

Events:

kubectl get events -n spc--delivery--be --sort-by='.lastTimestamp' | head -20

Traces (OpenTelemetry):

# Check OTEL collector
kubectl logs -f deployment/spc--delivery--be-otel-collector -n spc--delivery--be

# Port forward to access OTEL endpoints
kubectl port-forward -n spc--delivery--be deployment/spc--delivery--be-otel-collector 4317:4317

Data Flow

Delivery Request
    ↓
spc--delivery--be--app--prod (NodePort 31757)
    ↓
Main Delivery API (3 replicas - HA)
    ↓
Database (external)
    ↓
Cron Jobs → Scheduled Tasks
    ↓
Sample pickup scheduling, delivery routing, logistics updates

OpenTelemetry Tracing

Application → OTEL Collector (4317/4318) → Backend (Grafana/Jaeger)

Delivery Management Workflow

1. Delivery API (High Availability)

• 3 replicas for redundancy and load distribution
• Sample pickup scheduling
• Delivery route optimization
• Driver assignment
• Delivery status tracking
• Real-time updates

2. Scheduled Tasks

• Cron jobs for periodic delivery tasks
• Scheduled route optimization
• Delivery reminders
• Status synchronization
• Report generation

3. OpenTelemetry Tracing

• Distributed tracing for API requests
• Performance monitoring
• Request flow analysis
• Integration with Grafana/Jaeger

Production Considerations

High Availability

Well Configured:

• Main API: 3 replicas for redundancy
• Good HA setup for delivery API

x Single Points of Failure:

• Cron job: 1 replica
• OTEL collector: 1 replica

Recommendations

1. Auto-Scaling:

   • Currently fixed at 3 replicas (good baseline)
   • Add HPA to scale during peak delivery hours
   • Target: 3-10 replicas based on load
   • Consider KEDA if event-driven scaling needed

2. Cron Resilience:

   • Currently 1 replica
   • Consider 2 replicas if cron jobs are critical
   • Or ensure cron jobs are idempotent

3. OTEL Collector:

   • Currently 1 replica
   • Consider 2+ replicas for reliability
   • Monitor trace collection lag

4. Monitoring Priorities:

   • API response times (3 replicas handling load)
   • Delivery assignment success rates
   • Route optimization performance
   • Cron job execution
   • OTEL collector health

5. Recent Updates:

   • Namespace is relatively new (232 days)
   • Recent redeployment 20 days ago
   • Monitor stability and performance

Troubleshooting

Main API issues:

# Check all 3 API pods
kubectl get pods -n spc--delivery--be | grep "app--prod"

# Check logs from all replicas
kubectl logs deployment/spc--delivery--be--app--prod -n spc--delivery--be --all-containers=true --tail=100

# Check specific pod
POD_NAME=$(kubectl get pods -n spc--delivery--be | grep "app--prod" | head -1 | awk '{print $1}')
kubectl logs $POD_NAME -n spc--delivery--be --tail=100

# Test API endpoint
kubectl port-forward -n spc--delivery--be service/spc--delivery--be--app--prod 8080:80
# Access http://localhost:8080

Load distribution issues:

# Check resource usage across replicas
kubectl top pods -n spc--delivery--be | grep app--prod

# Check logs from each replica
for pod in $(kubectl get pods -n spc--delivery--be | grep "app--prod" | awk '{print $1}'); do
  echo "=== $pod ==="
  kubectl logs $pod -n spc--delivery--be --tail=10
done

# Restart all replicas
kubectl rollout restart deployment/spc--delivery--be--app--prod -n spc--delivery--be

Cron job failures:

# Check cron pod
kubectl get pods -n spc--delivery--be | grep cron

# Check cron logs
kubectl logs -f deployment/spc--delivery--be--cron--prod -n spc--delivery--be

# Check for errors
kubectl logs deployment/spc--delivery--be--cron--prod -n spc--delivery--be --tail=100 | grep -i "error\|fail"

# Restart cron
kubectl rollout restart deployment/spc--delivery--be--cron--prod -n spc--delivery--be

Tracing issues:

# Check OTEL collector
kubectl logs -f deployment/spc--delivery--be-otel-collector -n spc--delivery--be

# Check collector metrics
kubectl port-forward -n spc--delivery--be deployment/spc--delivery--be-otel-collector 8888:8888
# Access http://localhost:8888/metrics

# Restart OTEL collector
kubectl rollout restart deployment/spc--delivery--be-otel-collector -n spc--delivery--be

Performance Metrics

Current Scale

• Main API: 3 replicas (good HA)
• Cron: 1 replica
• OTEL Collector: 1 replica
• Total Active Pods: ~5 pods

Stability

• Namespace Age: 232 days (~7-8 months) - relatively new
• Recent Updates: 20 days ago
• HA Configuration: 3 replicas for main API (good)