sapoche

Overview

• Namespace: sapoche
• Purpose: Main Sapoche application and microservices (PRODUCTION)
• Age: ~4 years
• Status: Active - Largest application in production1 cluster
• Workloads: 39 deployments (36 active, 3 scaled to 0)
• Environment: PRODUCTION - Critical medical platform

Architecture

Sapoche is the core medical platform with multiple components handling patient care, medical imaging, and clinic operations:

• Consumers: Event consumers for various integrations (14 deployments)
• Workers: Background job processors (17 deployments)
• Backend Services: PHP-based application servers (8 replicas with HPA)
• Supporting Services: Redis, DICOM viewers, design system, frontend

Auto-Scaling Configuration

HorizontalPodAutoscalers (5 HPAs)

HPA Name	Target	Min	Max	Current	Metrics
sp-bnkd-php	sp-bnkd-php deployment	1	20	4-8	CPU: 46%/85%, Mem: 109MB/160Mi
keda-hpa-consumer-lis-testresult	consumer-lis-testresult	1	4	1	Queue depth: 0/20
keda-hpa-consumer-spc-checkup-orders	consumer-spc-checkup-orders	1	8	1	Queue depth: 0/20
keda-hpa-consumer-spc-pdf-generate	consumer-spc-pdf-generate	1	15	2	Queue depth: 0/20
keda-hpa-consumer-spc-pdf-generate-examination	consumer-spc-pdf-generate-examination	1	3	1	Queue depth: 0/20

KEDA Integration: 4 HPAs use KEDA (Kubernetes Event Driven Autoscaling) for queue-based autoscaling

Workload Categories

Event Consumers (14 deployments)

Consume events from message queues for various integrations:

Name	Replicas	Status	Purpose
consumer-dicom	1/1	Running	DICOM medical imaging
consumer-ecg	1/1	Running	ECG data processing
consumer-employee	1/1	Running	Employee data sync
consumer-lis-testresult	1/1	Running + HPA	Lab test results (auto-scales 1-4)
consumer-pat-dtc-homekit-register	0/0	x Scaled to 0	Patient homekit registration
consumer-pat-test-result-update	1/1	Running	Patient test result updates
consumer-revenue	0/0	x Scaled to 0	Revenue processing
consumer-rfd-push-notification	1/1	Running	RFD notifications
consumer-spc-aborted-booking	1/1	Running	Aborted booking handling
consumer-spc-checkup-orders	1/1	Running + HPA	Checkup orders (auto-scales 1-8)
consumer-spc-pdf-generate	2/2	Running + HPA	PDF generation (auto-scales 1-15)
consumer-spc-pdf-generate-examination	1/1	Running + HPA	Examination PDFs (auto-scales 1-3)
consumer-spc-pos-orders	3/3	Running	POS order processing
consumer-spc-sync-test-master-data	1/1	Running	Test master data sync

Background Workers (17 deployments)

Process async jobs, generate PDFs, send notifications:

Name	Replicas	Status	Purpose
worker-booking	4/4	Running	Booking operations
worker-sapoche-batch-publisher	1/1	Running	Batch job publisher
worker-sapoche-checkup-audit-history	1/1	Running	Checkup audit logging
worker-sapoche-default	10/10	Running	Default worker queue (10 replicas!)
worker-sapoche-event-tracking	1/1	Running	Event tracking
worker-sapoche-examination-audit-history	1/1	Running	Examination audit logging
worker-sapoche-export-csv	1/1	Running	CSV export generation
worker-sapoche-imaging-report	3/3	Running	Medical imaging reports
worker-sapoche-notifications	10/10	Running	General notifications (10 replicas!)
worker-sapoche-notifications-refdoc	1/1	Running	Reference doctor notifications
worker-sapoche-pdf	2/2	Running	PDF generation (general)
worker-sapoche-pdf-en	1/1	Running	PDF generation (English)
worker-sapoche-pdf-svip-regen	1/1	Running	SVIP PDF regeneration
worker-sapoche-pdf-svip1	1/1	Running	SVIP tier 1 PDFs
worker-sapoche-pdf-svip2	1/1	Running	SVIP tier 2 PDFs
worker-sapoche-pdf-vip1	1/1	Running	VIP tier 1 PDFs
ds-sapoche-customer-consumer	3/3	Running	Data streaming customer consumer

Backend Application Services (5 deployments)

Main application servers:

Name	Replicas	Status	Purpose
sp-bnkd-php	8/8	Running + HPA	PHP application backend (auto-scales 1-20)
sp-bknd-nginx	3/3	Running	Nginx frontend for PHP
redis-master	1/1	Running	Redis cache for application
sapoche-be--scheduler--prod	1/1	Running	Production task scheduler
orthanc-read-api	4/4	Running	Orthanc DICOM API (with sidecar)

Frontend Services (3 deployments)

Name	Replicas	Status	Purpose
sapoche-frontend-web	2/2	Running	Main frontend web application
diag-design-system-nginx	0/0	x Scaled to 0	Design system/component library
diag-dicom-viewer-nginx	1/1	Running	DICOM medical image viewer

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
sapoche-bknd-php	NodePort	10.8.27.48	9000, 80	32766, 30178	PHP application
bknd-services-nginx	NodePort	10.8.21.221	80	32702	Nginx proxy
redis-sp	ClusterIP	10.8.29.128	6379	-	Redis cache
sapoche-frontend-web	NodePort	10.8.23.96	80	31620	Frontend web app
diag-design-system-nginx	NodePort	10.8.20.168	80	32262	Design system
diag-dicom-viewer-nginx	NodePort	10.8.29.67	80	30103	DICOM viewer
orthanc-read-api-service	NodePort	10.8.25.196	80	32104	Orthanc API

Access & Management

View all resources:

kubectl get all -n sapoche

Check specific workload type:

# All consumers
kubectl get pods -n sapoche | grep consumer

# All workers
kubectl get pods -n sapoche | grep worker

# Backend services
kubectl get pods -n sapoche | grep -E "sp-bnkd|redis|scheduler"

# Frontend
kubectl get pods -n sapoche | grep -E "frontend|nginx"

View HPAs:

# All HPAs
kubectl get hpa -n sapoche

# Detailed HPA status
kubectl describe hpa sp-bnkd-php -n sapoche
kubectl describe hpa keda-hpa-consumer-spc-pdf-generate -n sapoche

View logs:

# Consumer logs
kubectl logs -f deployment/consumer-lis-testresult -n sapoche

# Worker logs
kubectl logs -f deployment/worker-sapoche-pdf -n sapoche

# Backend logs (with HPA)
kubectl logs -f deployment/sp-bnkd-php -n sapoche

# Frontend logs
kubectl logs -f deployment/sapoche-frontend-web -n sapoche

Scale workloads:

# Manual scaling (HPA will override if configured)
kubectl scale deployment worker-sapoche-pdf -n sapoche --replicas=4

# View current scaling
kubectl get hpa -n sapoche -w

Restart services:

# Restart backend (HPA-managed)
kubectl rollout restart deployment sp-bnkd-php -n sapoche

# Restart specific consumer
kubectl rollout restart deployment consumer-lis-testresult -n sapoche

# Restart frontend
kubectl rollout restart deployment sapoche-frontend-web -n sapoche

Monitoring

Resource usage:

kubectl top pods -n sapoche --sort-by=memory
kubectl top pods -n sapoche --sort-by=cpu

Check deployment status:

kubectl get deployments -n sapoche

View HPA metrics:

kubectl get hpa -n sapoche
kubectl describe hpa sp-bnkd-php -n sapoche

View events:

kubectl get events -n sapoche --sort-by='.lastTimestamp' | head -20

Check KEDA scaled objects:

kubectl get scaledobjects -n sapoche
kubectl describe scaledobjects -n sapoche

Data Flow

External Requests
    ↓
APISIX Gateway (apisix namespace)
    ↓
bknd-services-nginx (NodePort 32702)
    ↓
sp-bknd-nginx (load balancer, 3 replicas)
    ↓
sp-bnkd-php (PHP application, 4-20 replicas via HPA)
    ↓
redis-sp (cache)
    ↓
External Databases / Storage

Events → Consumers (with KEDA HPAs) → Process → Workers (10+ replicas) → Background Jobs

Special Components

PHP Backend with HPA

• Deployment: sp-bnkd-php
• Current: 4-8 replicas
• HPA Range: 1-20 replicas
• Scaling Triggers: CPU 85%, Memory 160Mi
• Purpose: Main application logic

Orthanc DICOM Server

• Medical imaging server (PACS)
• 4 replicas for high availability
• Read-only API exposed on NodePort 32104
• Stores and retrieves DICOM images (X-rays, CT scans, MRI, etc.)

DICOM Viewer

• Web-based medical image viewer
• Exposed on NodePort 30103
• Displays DICOM images from Orthanc

Worker Scaling

• worker-sapoche-default: 10 replicas (high-volume queue)
• worker-sapoche-notifications: 10 replicas (high notification load)
• worker-booking: 4 replicas (critical booking operations)

KEDA-Based Auto-Scaling

4 consumers use KEDA for queue-depth based scaling:

• Monitors queue depth in message broker
• Scales from 0 to max based on pending messages
• More efficient than CPU/memory-based scaling for event consumers

Production Considerations

High Availability

Well Configured:

• PHP backend: 4-8 replicas with HPA (scales to 20)
• Notifications: 10 replicas
• Default workers: 10 replicas
• Orthanc DICOM: 4 replicas
• Frontend: 2 replicas

x Single Points of Failure:

• Redis: 1 replica (consider Redis Sentinel/Cluster)
• Scheduler: 1 replica
• Most consumers: 1 replica (but have KEDA for scaling)

Auto-Scaling Status

Workload	Type	Current	Min	Max	Status
PHP Backend	CPU/Memory	4-8	1	20	Active
PDF Generate	Queue	2	1	15	KEDA Active
Checkup Orders	Queue	1	1	8	KEDA Active
LIS Test Result	Queue	1	1	4	KEDA Active
PDF Examination	Queue	1	1	3	KEDA Active

Recommendations

1. Redis High Availability:

   • x Currently 1 replica - single point of failure
   • Consider Redis Sentinel (3 nodes) or Redis Cluster
   • Critical for caching and session management

2. Consumer Resilience:

   • Most consumers at 1 replica
   • KEDA HPAs configured but currently at minimum
   • Consider baseline of 2 replicas for critical consumers

3. Monitoring Priorities:

   • Monitor HPA scaling events
   • Track queue depths (KEDA metrics)
   • Monitor PDF generation times
   • Alert on Redis failures
   • Track Orthanc storage capacity

4. Resource Optimization:

   • Review scaled-to-0 deployments:
     • consumer-pat-dtc-homekit-register
     • consumer-revenue
     • diag-design-system-nginx
   • Remove if permanently unused

5. Scaling Thresholds:

   • PHP Backend HPA target: CPU 85% (consider lowering to 70% for faster response)
   • KEDA queue thresholds: 20 messages (review if appropriate)

Troubleshooting

Consumer not processing:

# Check consumer logs
kubectl logs -f deployment/consumer-lis-testresult -n sapoche

# Check if KEDA scaler is working
kubectl describe scaledobject consumer-lis-testresult -n sapoche

# Check queue connection
kubectl exec -it deployment/consumer-lis-testresult -n sapoche -- env | grep QUEUE

# Restart consumer
kubectl rollout restart deployment/consumer-lis-testresult -n sapoche

Worker backlog:

# Check worker logs
kubectl logs deployment/worker-sapoche-pdf -n sapoche --tail=100

# Scale up workers temporarily
kubectl scale deployment/worker-sapoche-pdf -n sapoche --replicas=5

# Check Redis queue depth
kubectl exec -it deployment/redis-master -n sapoche -- redis-cli LLEN queue_name

PHP application issues:

# Check PHP pods
kubectl get pods -n sapoche | grep sp-bnkd-php

# Check HPA status
kubectl describe hpa sp-bnkd-php -n sapoche

# Check PHP logs
kubectl logs -f deployment/sp-bnkd-php -n sapoche

# Check Nginx logs
kubectl logs -f deployment/sp-bknd-nginx -n sapoche

# Execute into PHP pod
kubectl exec -it deployment/sp-bnkd-php -n sapoche -- bash

HPA not scaling:

# Check HPA events
kubectl describe hpa sp-bnkd-php -n sapoche

# Check metrics server
kubectl top nodes
kubectl top pods -n sapoche

# Check KEDA operator
kubectl get pods -n keda
kubectl logs -f -n keda deployment/keda-operator

DICOM issues:

# Check Orthanc API (4 replicas)
kubectl logs -f deployment/orthanc-read-api -n sapoche

# Test Orthanc API
kubectl port-forward -n sapoche service/orthanc-read-api-service 8042:80
# Access http://localhost:8042

# Check storage capacity
kubectl exec -it deployment/orthanc-read-api -n sapoche -- df -h

Frontend issues:

# Check frontend pods
kubectl get pods -n sapoche | grep sapoche-frontend-web

# Check frontend logs
kubectl logs -f deployment/sapoche-frontend-web -n sapoche

# Test frontend service
kubectl port-forward -n sapoche service/sapoche-frontend-web 8080:80
# Access http://localhost:8080

Performance Metrics

Current Scale (Production Load)

• PHP Backend: 4-8 replicas (auto-scaling based on load)
• Workers:
  • Default queue: 10 replicas (high volume)
  • Notifications: 10 replicas (high volume)
  • Booking: 4 replicas (critical path)
  • Others: 1-3 replicas
• Consumers: Mostly 1 replica with KEDA for burst scaling
• Frontend: 2 replicas

Total Pod Count

~65-75 pods running in sapoche namespace (varies with HPA/KEDA scaling)

Important Notes

x PRODUCTION ENVIRONMENT:

• This is a CRITICAL PRODUCTION namespace
• Serves medical platform - exercise extreme caution
• Changes should be tested in staging first
• Monitor HPA scaling during changes
• Have rollback plan ready