spc--pos

Overview

• Namespace: spc--pos
• Purpose: Sapoche Point of Sale (POS) System - PRODUCTION
• Age: 682 days (~23 months)
• Status: Active - Medical services POS and billing system
• Workloads: 18 deployments (15 active, 2 scaled to 0, 1 legacy nginx)
• Environment: PRODUCTION - Critical billing and order management

Architecture

The POS system handles medical service ordering, billing, and bulk upload operations:

• Main Application: REST API backend (2 replicas with HPA)
• Event Consumers: Process orders, rate cards, and patient data (11 deployments)
• Batch Publisher: Async job publishing (1 deployment)
• Workers: Background processing (2 deployments, both scaled to 0)
• Supporting Services: Nginx proxy, cron jobs, OpenTelemetry collector

Auto-Scaling Configuration

HorizontalPodAutoscaler (1 HPA)

HPA Name	Target	Min	Max	Current	Metrics	Type
spc--pos--be--app--prod	Main app	2	15	2	CPU: 37%/85%, Mem: 103MB/160Mi	Standard HPA

Scaling Summary:

• Main API auto-scales based on CPU/memory load
• Currently at minimum replicas (low load)
• No KEDA-based autoscaling for consumers

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
spc--pos--be--app--prod	2/2	Running + HPA	Main POS API (auto-scales 2-15)

The main application handles:

• Service ordering and billing
• Rate card management
• Order validation and processing
• Integration with payment systems
• RESTful API for frontend and integrations

Frontend (1 deployment)

Name	Replicas	Status	Purpose
sapoche-micro-pos--nginx--prod	1/1	Running	Nginx proxy for frontend (legacy)

Event Consumers (11 deployments)

Process POS-related events from message queues:

Name	Replicas	Status	Purpose
consumer-auto-change-answer-na	1/1	Running	Auto-change answer to N/A
consumer-customer	1/1	Running	Customer data sync
consumer-default	1/1	Running	Default event consumer
consumer-lis-patient	1/1	Running	LIS patient data sync
consumer-order	1/1	Running	General order processing
consumer-orders-bulk-upload	3/3	Running	Bulk order uploads (3 replicas for high volume)
consumer-orders-bulk-upload-svip1	1/1	Running	SVIP tier 1 bulk uploads
consumer-orders-bulk-upload-svip2	1/1	Running	SVIP tier 2 bulk uploads
consumer-orders-bulk-upload-vip1	1/1	Running	VIP tier 1 bulk uploads
consumer-pos-order	1/1	Running	POS order processing
consumer-rate-card-bulk-upload	1/1	Running	Rate card bulk uploads

Workers (3 deployments)

Background job processing:

Name	Replicas	Status	Purpose
wrk--batch-publisher	1/1	Running	Batch job publisher
wrk--read-hl7-sftp	0/0	x Scaled to 0	HL7 SFTP reader (inactive)
wrk--sync-patient	0/0	x Scaled to 0	Patient sync worker (inactive)

Supporting Services (2 deployments)

Name	Replicas	Status	Purpose
spc--pos--be--cron--prod	1/1	Running	Scheduled cron jobs
spc-pos-otel-collector	1/1	Running	OpenTelemetry trace collector

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
spc--pos--be--app--prod	NodePort	10.8.21.22	80	32095	Main POS API
sapoche-micro-pos--nginx--prod	NodePort	10.8.23.0	80	30059	Nginx proxy (legacy)
spc-pos-otel-collector	ClusterIP	10.8.18.192	4317, 4318	-	OTLP trace collection

Access & Management

View all resources:

kubectl get all -n spc--pos

Check main application:

# View app pods
kubectl get pods -n spc--pos | grep "app--prod"

# Check HPA status
kubectl describe hpa spc--pos--be--app--prod -n spc--pos

# View logs
kubectl logs -f deployment/spc--pos--be--app--prod -n spc--pos

Check consumers:

# All consumers
kubectl get pods -n spc--pos | grep consumer

# Bulk upload consumers (high volume)
kubectl get pods -n spc--pos | grep orders-bulk-upload

# Consumer logs
kubectl logs -f deployment/spc--pos--be--consumer-orders-bulk-upload--prod -n spc--pos

Check workers:

# All workers
kubectl get pods -n spc--pos | grep "wrk--"

# Batch publisher logs
kubectl logs -f deployment/spc--pos--be--wrk--batch-publisher--prod -n spc--pos

Scaling:

# View current HPA
kubectl get hpa -n spc--pos

# Manual scale (HPA will override)
kubectl scale deployment spc--pos--be--app--prod -n spc--pos --replicas=5

Restart services:

# Restart main app
kubectl rollout restart deployment/spc--pos--be--app--prod -n spc--pos

# Restart specific consumer
kubectl rollout restart deployment/spc--pos--be--consumer-orders-bulk-upload--prod -n spc--pos

# Restart all consumers
kubectl get deployments -n spc--pos | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n spc--pos

Monitoring

Resource usage:

kubectl top pods -n spc--pos --sort-by=memory
kubectl top pods -n spc--pos --sort-by=cpu

HPA metrics:

# HPA status
kubectl get hpa -n spc--pos

# Detailed HPA status
kubectl describe hpa spc--pos--be--app--prod -n spc--pos

Deployment status:

# Active deployments
kubectl get deployments -n spc--pos | grep -v "0/0"

# Scaled to 0
kubectl get deployments -n spc--pos | grep "0/0"

Events:

kubectl get events -n spc--pos --sort-by='.lastTimestamp' | head -20

Traces (OpenTelemetry):

# Check OTEL collector
kubectl logs -f deployment/spc-pos-otel-collector -n spc--pos

# Port forward to access OTEL endpoints
kubectl port-forward -n spc--pos deployment/spc-pos-otel-collector 4317:4317

Data Flow

External Requests (via APISIX/Traefik)
    ↓
spc--pos--be--app--prod (NodePort 32095)
    ↓
Main POS API (2-15 replicas via HPA)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Consumers Process Events
    ↓
Batch Publisher Creates Background Jobs
    ↓
Order fulfillment, billing, notifications

OpenTelemetry Tracing

Application → OTEL Collector (4317/4318) → Backend (Grafana/Jaeger)

POS Workflow

1. Order Creation

• Orders created via API or bulk upload
• consumer-order / consumer-pos-order process orders
• Rate cards applied via consumer-rate-card-bulk-upload

2. Bulk Upload Processing

• High Volume: consumer-orders-bulk-upload (3 replicas)
• VIP Tiers: Separate consumers for SVIP1, SVIP2, VIP1
• Excel/CSV file processing for mass order creation

3. Patient Data Sync

• Patient information from LIS
• consumer-lis-patient handles patient data
• Customer data sync via consumer-customer

4. Validation & Processing

• Order validation and verification
• Rate card application
• Auto-answer changes via consumer-auto-change-answer-na

5. Background Jobs

• Batch publisher creates async jobs
• Job processing for non-time-critical tasks
• Scheduled tasks via cron

6. Legacy Integration

• HL7 SFTP integration (scaled to 0 - potentially deprecated)
• Patient sync worker (scaled to 0 - potentially deprecated)

Production Considerations

High Availability

Well Configured:

• Main API: 2 replicas with HPA (scales to 15)
• Bulk upload consumer: 3 replicas (high volume)
• Separate consumers for VIP tiers

x Single Points of Failure:

• Most consumers: 1 replica
• Batch publisher: 1 replica
• Cron job: 1 replica
• OTEL collector: 1 replica
• Nginx proxy: 1 replica (legacy)

Auto-Scaling Configuration

Workload	Type	Current	Min	Max	Scaling Metric
Main API	Standard HPA	2	2	15	CPU 85%, Mem 160Mi

Current State: System at low load with single HPA at minimum replicas

Recommendations

1. Main API Scaling:

   • Currently at 2 replicas (minimum)
   • CPU at 37% - significant headroom
   • Consider if max of 15 is sufficient for peak loads

2. Consumer Resilience:

   • All consumers at 1 replica (except bulk upload)
   • Critical consumers should have 2 replicas:
     • consumer-order
     • consumer-pos-order
     • consumer-customer

3. Bulk Upload Capacity:

   • Main bulk upload consumer: 3 replicas (good)
   • VIP tier consumers: 1 replica each
   • Monitor if VIP consumers need scaling during peak times

4. Resource Cleanup:

   • 2 workers scaled to 0 - review and potentially remove:
     • wrk--read-hl7-sftp (315 days old)
     • wrk--sync-patient (106 days old)
   • Legacy nginx proxy - review if still needed

5. Observability:

   • OTEL collector at 1 replica
   • Consider 2+ replicas for reliability
   • Monitor trace collection lag

6. KEDA Integration:

   • No KEDA-based autoscaling currently
   • Consider KEDA for bulk upload consumers
   • Queue-based scaling would be more efficient than manual scaling

7. Monitoring Priorities:

   • API response times
   • Bulk upload success rates
   • Order processing latency
   • Rate card application accuracy
   • Consumer lag monitoring

Troubleshooting

Main API issues:

# Check API pods
kubectl get pods -n spc--pos | grep "app--prod"

# Check HPA status
kubectl describe hpa spc--pos--be--app--prod -n spc--pos

# Check logs
kubectl logs -f deployment/spc--pos--be--app--prod -n spc--pos --tail=100

# Test API endpoint
kubectl port-forward -n spc--pos service/spc--pos--be--app--prod 8080:80
# Access http://localhost:8080

Consumer not processing:

# Check consumer status
kubectl get pods -n spc--pos | grep consumer-orders-bulk-upload

# Check logs
kubectl logs -f deployment/spc--pos--be--consumer-orders-bulk-upload--prod -n spc--pos

# Check for errors
kubectl logs deployment/spc--pos--be--consumer-orders-bulk-upload--prod -n spc--pos --tail=100 | grep -i error

# Restart consumer
kubectl rollout restart deployment/spc--pos--be--consumer-orders-bulk-upload--prod -n spc--pos

HPA not scaling:

# Check HPA events
kubectl describe hpa spc--pos--be--app--prod -n spc--pos

# Check metrics server
kubectl top nodes
kubectl top pods -n spc--pos

# Check resource requests/limits
kubectl get deployment spc--pos--be--app--prod -n spc--pos -o yaml | grep -A 10 resources

Bulk upload delays:

# Check all bulk upload consumers
kubectl get pods -n spc--pos | grep orders-bulk-upload

# Check logs for each
for consumer in orders-bulk-upload orders-bulk-upload-svip1 orders-bulk-upload-svip2 orders-bulk-upload-vip1; do
  echo "=== consumer-$consumer ==="
  kubectl logs deployment/spc--pos--be--consumer-$consumer--prod -n spc--pos --tail=20
done

# Check if scaling is needed
kubectl top pods -n spc--pos | grep orders-bulk-upload

Order processing issues:

# Check order consumers
kubectl logs -f deployment/spc--pos--be--consumer-order--prod -n spc--pos
kubectl logs -f deployment/spc--pos--be--consumer-pos-order--prod -n spc--pos

# Check batch publisher
kubectl logs -f deployment/spc--pos--be--wrk--batch-publisher--prod -n spc--pos

# Restart both
kubectl rollout restart deployment/spc--pos--be--consumer-order--prod -n spc--pos
kubectl rollout restart deployment/spc--pos--be--consumer-pos-order--prod -n spc--pos

Rate card issues:

# Check rate card consumer
kubectl logs -f deployment/spc--pos--be--consumer-rate-card-bulk-upload--prod -n spc--pos

# Check for validation errors
kubectl logs deployment/spc--pos--be--consumer-rate-card-bulk-upload--prod -n spc--pos --tail=100 | grep -i "rate card"

# Restart consumer
kubectl rollout restart deployment/spc--pos--be--consumer-rate-card-bulk-upload--prod -n spc--pos

Tracing issues:

# Check OTEL collector
kubectl logs -f deployment/spc-pos-otel-collector -n spc--pos

# Check collector metrics
kubectl port-forward -n spc--pos deployment/spc-pos-otel-collector 8888:8888
# Access http://localhost:8888/metrics

Scaled-to-0 investigation:

# Check HL7 SFTP worker
kubectl get deployment spc--pos--be--wrk--read-hl7-sftp--prod -n spc--pos -o yaml

# Check patient sync worker
kubectl get deployment spc--pos--be--wrk--sync-patient--prod -n spc--pos -o yaml

# Scale up if needed for testing
kubectl scale deployment spc--pos--be--wrk--read-hl7-sftp--prod -n spc--pos --replicas=1
kubectl scale deployment spc--pos--be--wrk--sync-patient--prod -n spc--pos --replicas=1

Performance Metrics

Current Scale (Production Load)

• Main API: 2 replicas (low load, can scale to 15)
• Consumers:
  • High volume: 3 replicas (orders-bulk-upload)
  • VIP tiers: 1 replica each
  • Standard: 1 replica
• Workers: 1 batch publisher active, 2 scaled to 0
• Total Active Pods: ~20-25 pods in namespace

Scaling Behavior

• Main API HPA: Scales based on CPU (85% threshold) and memory (160Mi)
• No KEDA: All consumers at fixed replica counts
• Bulk Upload: Manually scaled to 3 replicas for high volume

Integration Points

External Systems

1. Laboratory Information System (LIS):

   • Patient data synchronization
   • Test order integration
   • consumer-lis-patient handles LIS data

2. Customer Management:

   • Customer data sync
   • consumer-customer processes customer events

3. Bulk Upload Systems:

   • Excel/CSV file processing
   • Mass order creation
   • VIP/SVIP tier handling

4. Payment Systems:

   • Rate card application
   • Billing integration
   • Order pricing

5. Legacy HL7 Integration:

   • HL7 SFTP reader (currently inactive)
   • Patient sync worker (currently inactive)

Important Notes

x PRODUCTION ENVIRONMENT:

• This is a CRITICAL PRODUCTION system handling medical service orders and billing
• Downtime directly impacts revenue and patient service delivery
• Changes must be tested in staging first
• Coordinate with finance and operations teams
• Monitor bulk upload processing carefully
• Have immediate rollback plan ready

x Cleanup Opportunity:

• 2 workers scaled to 0 for extended periods
• Review HL7 SFTP and patient sync workers
• Remove if permanently deprecated

Compliance: Financial and medical data subject to regulatory requirements (HIPAA, PCI-DSS if handling payments)

System Purpose

The POS (Point of Sale) system serves as:

1. Order Management: Medical service ordering and fulfillment
2. Billing Engine: Rate card application and pricing
3. Bulk Operations: High-volume order processing
4. Patient Integration: Connects with LIS for patient data
5. VIP Handling: Separate processing for VIP/SVIP customers

Key Role: Central billing and order management platform for medical services with emphasis on bulk upload capabilities for high-volume operations.