sapoche

Overview

• Namespace: sapoche
• Purpose: Main Sapoche application and microservices
• Age: ~2 years
• Status: Active - Largest application in cluster
• Workloads: 38 deployments (36 active, 2 scaled to 0)

Architecture

Sapoche is the core medical platform with multiple components:

• Consumers: Event consumers for various integrations
• Workers: Background job processors
• Backend Services: PHP-based application servers
• Supporting Services: Redis, DICOM viewers, design system

Workload Categories

Event Consumers (13 deployments)

Consume events from message queues for various integrations:

Name	Replicas	Purpose
consumer-dicom	1/1	DICOM medical imaging
consumer-ecg	1/1	ECG data processing
consumer-employee	1/1	Employee data sync
consumer-lis-testresult	1/1	Lab test results
consumer-pat-dtc-homekit-register	1/1	Patient homekit registration
consumer-pat-test-result-update	1/1	Patient test result updates
consumer-revenue	0/0	Revenue processing (scaled to 0)
consumer-rfd-push-notification	1/1	RFD notifications
consumer-spc-aborted-booking	1/1	Aborted booking handling
consumer-spc-checkup-orders	1/1	Checkup order processing
consumer-spc-pdf-generate	1/1	PDF generation
consumer-spc-pdf-generate-examination	1/1	Examination PDF generation
consumer-spc-pos-orders	1/1	POS order processing
consumer-spc-sync-test-master-data	1/1	Test master data sync

Background Workers (16 deployments)

Process async jobs, generate PDFs, send notifications:

Name	Replicas	Purpose
worker-booking	1/1	Booking operations
worker-sapoche-batch-publisher	1/1	Batch job publisher
worker-sapoche-batch-publisher-notification	0/0	Batch notifications (scaled to 0)
worker-sapoche-checkup-audit-history	1/1	Checkup audit logging
worker-sapoche-default	1/1	Default worker queue
worker-sapoche-event-tracking	1/1	Event tracking
worker-sapoche-examination-audit-history	1/1	Examination audit logging
worker-sapoche-export-csv	1/1	CSV export generation
worker-sapoche-imaging-report	1/1	Medical imaging reports
worker-sapoche-notifications	1/1	General notifications
worker-sapoche-notifications-refdoc	1/1	Reference doctor notifications
worker-sapoche-pdf	1/1	PDF generation (general)
worker-sapoche-pdf-en	1/1	PDF generation (English)
worker-sapoche-pdf-svip-regen	1/1	SVIP PDF regeneration
worker-sapoche-pdf-svip1	1/1	SVIP tier 1 PDFs
worker-sapoche-pdf-svip2	1/1	SVIP tier 2 PDFs
worker-sapoche-pdf-vip1	1/1	VIP tier 1 PDFs

Backend Application Services (5 deployments)

Main application servers:

Name	Replicas	Purpose
st-bnkd-php	3/3	PHP application backend (3 replicas)
st-bnkd-php-nginx	3/3	Nginx frontend for PHP (3 replicas)
redis-master-st-sp	1/1	Redis cache for application
sapoche-be--scheduler--stag	1/1	Task scheduler
orthanc-read-api	1/1	Orthanc DICOM API (with sidecar)

Frontend Services (2 deployments)

Name	Replicas	Purpose
diag-design-system-nginx	1/1	Design system/component library
diag-dicom-viewer-nginx	1/1	DICOM medical image viewer

Services

Name	Type	Cluster IP	Ports	Purpose
st-bnkd-php	ClusterIP	10.27.183.187	9000, 80	PHP application
st-sapoche-nginx	ClusterIP	10.27.41.39	80	Nginx proxy
st-redis-sp	ClusterIP	10.27.62.167	6379	Redis cache
diag-design-system-nginx	NodePort	10.27.254.171	80:30788	Design system
diag-dicom-viewer-nginx	NodePort	10.27.16.173	80:30120	DICOM viewer
orthanc-read-api-service	NodePort	10.27.18.84	80:30635	Orthanc API

Access & Management

View all resources:

kubectl get all -n sapoche

Check specific workload type:

# All consumers
kubectl get pods -n sapoche | grep consumer

# All workers
kubectl get pods -n sapoche | grep worker

# Backend services
kubectl get pods -n sapoche | grep -E "st-bnkd|redis|scheduler"

View logs:

# Consumer logs
kubectl logs -f deployment/consumer-lis-testresult -n sapoche

# Worker logs
kubectl logs -f deployment/worker-sapoche-pdf -n sapoche

# Backend logs
kubectl logs -f deployment/st-bnkd-php -n sapoche

Scale workloads:

# Scale backend replicas
kubectl scale deployment st-bnkd-php -n sapoche --replicas=5

# Scale workers
kubectl scale deployment worker-sapoche-pdf -n sapoche --replicas=2

Restart services:

# Restart all backend pods
kubectl rollout restart deployment st-bnkd-php -n sapoche

# Restart specific consumer
kubectl rollout restart deployment consumer-lis-testresult -n sapoche

Monitoring

Resource usage:

kubectl top pods -n sapoche --sort-by=memory
kubectl top pods -n sapoche --sort-by=cpu

Check deployment status:

kubectl get deployments -n sapoche

View events:

kubectl get events -n sapoche --sort-by='.lastTimestamp' | head -20

Data Flow

External Requests
    ↓
APISIX Gateway (other namespace)
    ↓
st-sapoche-nginx (10.27.41.39:80)
    ↓
st-bnkd-php-nginx (load balancer, 3 replicas)
    ↓
st-bnkd-php (PHP application, 3 replicas)
    ↓
st-redis-sp (cache)
    ↓
External Databases / Storage

Events → Consumers → Process → Workers → Background Jobs → Notifications/PDFs

Special Components

Orthanc DICOM Server

• Medical imaging server (PACS)
• Read-only API exposed on NodePort 30635
• Stores and retrieves DICOM images (X-rays, CT scans, MRI, etc.)

DICOM Viewer

• Web-based medical image viewer
• Exposed on NodePort 30120
• Displays DICOM images from Orthanc

Design System

• Shared UI component library
• Exposed on NodePort 30788
• Used across multiple applications

Recommendations

1. High Availability:

   • Backend: 3 replicas (good)
   • x Most consumers/workers: 1 replica (consider 2+ for critical ones)
   • x Redis: 1 replica (consider Redis Sentinel or cluster mode)

2. Scaling:

   • Backend auto-scales well with manual intervention
   • Consider HPA for:
     • st-bnkd-php (CPU-based)
     • Critical workers (queue depth-based with KEDA)

3. Monitoring:

   • Monitor queue depths for consumers
   • Track PDF generation times
   • Monitor Redis memory usage
   • Alert on consumer failures

4. Resource Optimization:

   • Review inactive deployments (consumer-revenue, worker-batch-publisher-notification scaled to 0)
   • Consider removing if permanently unused

5. Logging:

   • Centralize logs from all 38 deployments
   • Implement structured logging
   • Track error rates per component

Troubleshooting

Consumer not processing:

# Check consumer logs
kubectl logs -f deployment/consumer-lis-testresult -n sapoche

# Check if connected to message queue
kubectl exec -it deployment/consumer-lis-testresult -n sapoche -- env | grep QUEUE

# Restart consumer
kubectl rollout restart deployment/consumer-lis-testresult -n sapoche

Worker backlog:

# Check worker logs
kubectl logs deployment/worker-sapoche-pdf -n sapoche --tail=100

# Scale up workers temporarily
kubectl scale deployment/worker-sapoche-pdf -n sapoche --replicas=3

# Check Redis queue depth
kubectl exec -it deployment/redis-master-st-sp -n sapoche -- redis-cli LLEN queue_name

PHP application issues:

# Check PHP pods
kubectl get pods -n sapoche | grep st-bnkd-php

# Check PHP logs
kubectl logs -f deployment/st-bnkd-php -n sapoche

# Check Nginx logs
kubectl logs -f deployment/st-bnkd-php-nginx -n sapoche

# Execute into PHP pod
kubectl exec -it deployment/st-bnkd-php -n sapoche -- bash

DICOM issues:

# Check Orthanc API
kubectl logs -f deployment/orthanc-read-api -n sapoche

# Test Orthanc API
kubectl port-forward -n sapoche service/orthanc-read-api-service 8042:80
# Access http://localhost:8042

Performance Considerations

• Backend: 3 replicas handle concurrent requests
• Workers: Process jobs asynchronously to avoid blocking
• Consumers: Real-time event processing from message queues
• Redis: In-memory cache for fast data access
• PDF Generation: Multiple specialized workers for different tier levels

Dependencies

• Message Queue: Likely Redpanda or Kafka (in separate namespace)
• Databases: External MySQL/PostgreSQL
• Object Storage: For DICOM images and PDFs
• Redis: In-namespace cache