Skip to main content

spc--purchase-order--be

Overview​

  • Namespace: spc--purchase-order--be
  • Purpose: Sapoche Purchase Order Management Backend - PRODUCTION
  • Age: 195 days (~6.5 months, since April 2025)
  • Status: Active - Purchase order processing
  • Workloads: 5 deployments (all active)
  • Environment: PRODUCTION - Inventory and procurement management

Architecture​

Purchase order management system handling order creation, bulk uploads, and conversions:

  • Main Application: REST API backend (1 replica)
  • Event Consumers: Bulk upload processing, order conversion (2 deployments)
  • Worker: Batch job publisher (1 deployment)
  • Observability: OpenTelemetry collector for tracing

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed replica counts (all at 1 replica)

Workload Categories​

Main Application (1 deployment)​

NameReplicasStatusPurpose
spc--purchase-order--be--app--prod1/1RunningMain purchase order API

Event Consumers (2 deployments)​

NameReplicasStatusPurpose
consumer-bulk-upload-purchase-order1/1RunningBulk purchase order uploads
consumer-convert-purchase-order1/1RunningPurchase order conversions

Workers (1 deployment)​

NameReplicasStatusPurpose
wrk--batch-publisher1/1RunningBatch job publisher

Observability (1 deployment)​

NameReplicasStatusPurpose
spc--purchase-order--be-otel-collector1/1RunningOpenTelemetry trace collector

Services​

NameTypeCluster IPPortsNodePortPurpose
spc--purchase-order--be--app--prodNodePort10.8.19.1328030195Main purchase order API
spc--purchase-order--be-otel-collectorClusterIP10.8.23.2014317, 4318-OTLP trace collection

Access & Management​

View all resources:​

kubectl get all -n spc--purchase-order--be

Check main application:​

kubectl get pods -n spc--purchase-order--be | grep "app--prod"
kubectl logs -f deployment/spc--purchase-order--be--app--prod -n spc--purchase-order--be

Check consumers:​

# All consumers
kubectl get pods -n spc--purchase-order--be | grep consumer

# Bulk upload consumer
kubectl logs -f deployment/spc--purchase-order--be--consumer-bulk-upload-purchase-order--prod -n spc--purchase-order--be

# Convert consumer
kubectl logs -f deployment/spc--purchase-order--be--consumer-convert-purchase-order--prod -n spc--purchase-order--be

Check batch publisher:​

kubectl get pods -n spc--purchase-order--be | grep batch-publisher
kubectl logs -f deployment/spc--purchase-order--be--wrk--batch-publisher--prod -n spc--purchase-order--be

Restart services:​

# Restart main app
kubectl rollout restart deployment/spc--purchase-order--be--app--prod -n spc--purchase-order--be

# Restart specific consumer
kubectl rollout restart deployment/spc--purchase-order--be--consumer-bulk-upload-purchase-order--prod -n spc--purchase-order--be

# Restart all consumers
kubectl get deployments -n spc--purchase-order--be | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n spc--purchase-order--be

Monitoring​

Resource usage:​

kubectl top pods -n spc--purchase-order--be --sort-by=memory
kubectl top pods -n spc--purchase-order--be --sort-by=cpu

Events:​

kubectl get events -n spc--purchase-order--be --sort-by='.lastTimestamp' | head -20

Traces (OpenTelemetry):​

# Check OTEL collector
kubectl logs -f deployment/spc--purchase-order--be-otel-collector -n spc--purchase-order--be

# Port forward to access OTEL endpoints
kubectl port-forward -n spc--purchase-order--be deployment/spc--purchase-order--be-otel-collector 4317:4317

Data Flow​

Purchase Order Request
    ↓
spc--purchase-order--be--app--prod (NodePort 30195)
    ↓
Main Purchase Order API
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Consumers Process Events
    ├─ Bulk Upload → consumer-bulk-upload-purchase-order
    └─ Conversions → consumer-convert-purchase-order
    ↓
Batch Publisher Creates Background Jobs
    ↓
Purchase order processing, inventory updates

OpenTelemetry Tracing​

Application → OTEL Collector (4317/4318) → Backend (Grafana/Jaeger)

Purchase Order Workflow​

1. Purchase Order API​

  • Create purchase orders
  • Manage supplier orders
  • Track order status
  • Inventory management integration
  • Procurement workflows

2. Bulk Upload Processing​

  • Bulk purchase order uploads from Excel/CSV
  • consumer-bulk-upload-purchase-order processes upload events
  • Validation and data transformation
  • Batch order creation

3. Order Conversion​

  • Convert between order types
  • consumer-convert-purchase-order processes conversion events
  • Order format transformations
  • System integration conversions

4. Batch Publisher​

  • Creates background jobs for async processing
  • Handles large-scale order operations
  • Scheduled processing tasks

5. OpenTelemetry Tracing​

  • Distributed tracing for API requests
  • Performance monitoring
  • Request flow analysis

Production Considerations​

High Availability​

x Single Points of Failure:

  • Main API: 1 replica (no HA)
  • All consumers: 1 replica each
  • Batch publisher: 1 replica
  • OTEL collector: 1 replica

Recommendations​

  1. Main API Resilience:

    • Currently 1 replica (single point of failure)
    • Increase to 2+ replicas or add HPA
    • Critical for purchase order operations

  2. Add Auto-Scaling:

    • Consider HPA for main API
    • Add KEDA for consumers based on queue depth:
      • consumer-bulk-upload-purchase-order (bulk operations)
      • consumer-convert-purchase-order

  3. Consumer Resilience:

    • All consumers at 1 replica
    • Consider 2 replicas for:
      • consumer-bulk-upload-purchase-order (critical for bulk ops)
      • consumer-convert-purchase-order

  4. Recent Updates:

    • Very recent deployment: 20 hours ago
    • Active development/updates
    • Monitor stability after recent changes

  5. Monitoring Priorities:

    • API response times
    • Bulk upload processing success rates
    • Order conversion success rates
    • Consumer lag monitoring

Troubleshooting​

Main API issues:​

# Check API pod
kubectl get pods -n spc--purchase-order--be | grep "app--prod"

# Check logs
kubectl logs -f deployment/spc--purchase-order--be--app--prod -n spc--purchase-order--be --tail=100

# Test API endpoint
kubectl port-forward -n spc--purchase-order--be service/spc--purchase-order--be--app--prod 8080:80
# Access http://localhost:8080

Bulk upload issues:​

# Check bulk upload consumer
kubectl logs -f deployment/spc--purchase-order--be--consumer-bulk-upload-purchase-order--prod -n spc--purchase-order--be

# Check for upload errors
kubectl logs deployment/spc--purchase-order--be--consumer-bulk-upload-purchase-order--prod -n spc--purchase-order--be --tail=100 | grep -i "error\|fail\|upload"

# Restart consumer
kubectl rollout restart deployment/spc--purchase-order--be--consumer-bulk-upload-purchase-order--prod -n spc--purchase-order--be

Order conversion issues:​

# Check convert consumer
kubectl logs -f deployment/spc--purchase-order--be--consumer-convert-purchase-order--prod -n spc--purchase-order--be

# Check for conversion errors
kubectl logs deployment/spc--purchase-order--be--consumer-convert-purchase-order--prod -n spc--purchase-order--be --tail=100 | grep -i "error\|convert\|fail"

# Restart consumer
kubectl rollout restart deployment/spc--purchase-order--be--consumer-convert-purchase-order--prod -n spc--purchase-order--be

Batch publisher issues:​

# Check batch publisher
kubectl logs -f deployment/spc--purchase-order--be--wrk--batch-publisher--prod -n spc--purchase-order--be

# Check for job publishing errors
kubectl logs deployment/spc--purchase-order--be--wrk--batch-publisher--prod -n spc--purchase-order--be --tail=100 | grep -i "error\|batch\|publish"

# Restart batch publisher
kubectl rollout restart deployment/spc--purchase-order--be--wrk--batch-publisher--prod -n spc--purchase-order--be

Tracing issues:​

# Check OTEL collector
kubectl logs -f deployment/spc--purchase-order--be-otel-collector -n spc--purchase-order--be

# Check collector metrics
kubectl port-forward -n spc--purchase-order--be deployment/spc--purchase-order--be-otel-collector 8888:8888
# Access http://localhost:8888/metrics

Performance Metrics​

Current Scale​

  • Main API: 1 replica (no HA)
  • Consumers: 2 consumers at 1 replica each
  • Worker: 1 batch publisher
  • OTEL Collector: 1 replica
  • Total Active Pods: ~5 pods

Stability​

  • Namespace Age: 195 days (~6.5 months) - relatively new
  • Recent Updates: 20 hours ago (very active development)
  • All Single Replicas: No redundancy configured