pat--test-result--be

Overview

• Namespace: pat--test-result--be
• Purpose: Patient Portal Test Result Backend - PRODUCTION
• Age: ~3 years
• Status: Active - Patient-facing test result service
• Workloads: 21 deployments (18 active, 3 scaled to 0)
• Environment: PRODUCTION - Critical patient-facing service

Architecture

Patient Portal Test Result Backend manages test results for patient access:

• API: REST API backend (2 replicas with HPA)
• Event Consumers: Process test result events and updates (14 deployments)
• Workers: Background job processors (3 deployments, 2 active)
• Cron Jobs: Scheduled tasks
• Integration: LIS, imaging services, appointments, notifications

Auto-Scaling Configuration

HorizontalPodAutoscalers (8 HPAs)

HPA Name	Target	Min	Max	Current	Metrics	Type
pat--test-result--be--app--prod	API	1	10	2	CPU: 14%/100%, Mem: 106MB/160Mi	Standard HPA
keda-hpa-consumer-imaging-service-report	Imaging reports	1	5	1	Queue: 0/10	KEDA
keda-hpa-consumer-lis-testresult	LIS test results	1	4	1	Queue: 0/10	KEDA
keda-hpa-consumer-pat-notifi-test-result	Test result notifications	1	6	1	Queue: 0/10	KEDA
keda-hpa-consumer-pat-test-master-data	Test master data	1	6	1	Queue: 0/10	KEDA
keda-hpa-consumer-pat-test-result	Patient test results	3	25	3	Queue: 0/10	KEDA
keda-hpa-consumer-pat-test-result-refdoc	Referral doc results	1	10	1	Queue: 0/20	KEDA
keda-hpa-consumer-pat-test-result-walkin	Walk-in test results	1	5	1	Queue: 0/20	KEDA

Scaling Summary:

• API auto-scales based on CPU/memory load
• 7 consumers use KEDA for queue-based autoscaling
• Main consumer (pat-test-result) maintains 3 replicas baseline, scales to 25

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
pat--test-result--be--app--prod	2/2	Running + HPA	Main API (auto-scales 1-10)

Event Consumers (14 deployments)

Process patient test result related events:

Name	Replicas	Status	Purpose
consumer-bookings	1/1	Running	Booking integration
consumer-copper-configuration	1/1	Running	Copper system config
consumer-customer	1/1	Running	Customer data sync
consumer-homekit-register	1/1	Running	Homekit registration
consumer-imaging-service-report	1/1	Running + HPA	Imaging reports (scales 1-5)
consumer-lis-testresult	1/1	Running + HPA	LIS test results (scales 1-4)
consumer-lis-testresult-update	1/1	Running	LIS result updates
consumer-pat-appointment-sync	1/1	Running	Appointment synchronization
consumer-pat-notification-test-result	1/1	Running + HPA	Test result notifications (scales 1-6)
consumer-pat-sync-test-master-data	1/1	Running	Test catalog sync
consumer-pat-test-master-data	1/1	Running + HPA	Test master data (scales 1-6)
consumer-pat-test-result	3/3	Running + HPA	Main consumer (scales 3-25)
consumer-pat-test-result-refdoc	1/1	Running + HPA	Referral doctor results (scales 1-10)
consumer-pat-test-result-walkin	1/1	Running + HPA	Walk-in patient results (scales 1-5)

Scaled to 0:

• consumer-mixpanel-events (x Analytics integration - inactive)
• consumer-pat-clear-order-data (x Data cleanup - inactive)

Workers (3 deployments)

Name	Replicas	Status	Purpose
wrk--batch-publisher	1/1	Running	Batch job publisher
wrk--notifications	1/1	Running	Notification processing
wrk--default	0/0	x Scaled to 0	Default worker queue

Supporting Services (1 deployment)

Name	Replicas	Status	Purpose
cron	1/1	Running	Scheduled cron jobs

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
pat--test-result--be--app--prod	NodePort	10.8.23.32	80	30940	Main API

Access & Management

View all resources:

kubectl get all -n pat--test-result--be

Check API:

# View API pods
kubectl get pods -n pat--test-result--be | grep "app--prod"

# Check HPA status
kubectl describe hpa pat--test-result--be--app--prod -n pat--test-result--be

# View logs
kubectl logs -f deployment/pat--test-result--be--app--prod -n pat--test-result--be

# Test API
kubectl port-forward -n pat--test-result--be service/pat--test-result--be--app--prod 8080:80
# Access http://localhost:8080

Check consumers:

# All active consumers
kubectl get pods -n pat--test-result--be | grep consumer | grep Running

# Main test result consumer
kubectl get pods -n pat--test-result--be | grep "consumer-pat-test-result--prod"

# KEDA-scaled consumers
kubectl get hpa -n pat--test-result--be | grep keda

# Consumer logs
kubectl logs -f deployment/pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

View HPAs:

# All HPAs
kubectl get hpa -n pat--test-result--be

# Detailed HPA status
kubectl describe hpa pat--test-result--be--app--prod -n pat--test-result--be
kubectl describe hpa keda-hpa-pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

Scaling:

# View current scaling
kubectl get hpa -n pat--test-result--be -w

# Check KEDA scaled objects
kubectl get scaledobjects -n pat--test-result--be

Restart services:

# Restart API
kubectl rollout restart deployment/pat--test-result--be--app--prod -n pat--test-result--be

# Restart main consumer
kubectl rollout restart deployment/pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

# Restart all active consumers
kubectl get deployments -n pat--test-result--be | grep consumer | grep -v "0/0" | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n pat--test-result--be

Monitoring

Resource usage:

kubectl top pods -n pat--test-result--be --sort-by=memory
kubectl top pods -n pat--test-result--be --sort-by=cpu

HPA metrics:

# All HPAs
kubectl get hpa -n pat--test-result--be

# Detailed status
kubectl describe hpa pat--test-result--be--app--prod -n pat--test-result--be

Deployment status:

# Active deployments
kubectl get deployments -n pat--test-result--be | grep -v "0/0"

# Check main consumer
kubectl get deployment pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

Events:

kubectl get events -n pat--test-result--be --sort-by='.lastTimestamp' | head -20

Data Flow

Patient Portal Frontend
    ↓
API Gateway (APISIX/Traefik)
    ↓
pat--test-result--be--app--prod (NodePort 30940)
    ↓
API (2-10 replicas via HPA)
    ↓
Database (patient test results)
    ↓
Events Published to Kafka
    ↓
Consumers Process Events (with KEDA autoscaling)
    ↓
    ├─→ LIS Integration (test results from lab)
    ├─→ Imaging Integration (radiology reports)
    ├─→ Notifications (SMS/email to patients)
    ├─→ Appointment Sync
    └─→ Test Master Data (test catalog)

Patient-Facing Workflow

1. Test Result Availability

• LIS publishes test results
• consumer-lis-testresult receives events (KEDA-scaled 1-4)
• Test results stored in patient portal database

2. Patient Access

• Patients log into portal
• API serves test results (2-10 replicas based on load)
• Real-time availability checks

3. Notifications

• consumer-pat-notification-test-result processes (scales 1-6)
• wrk--notifications sends SMS/email
• Patients notified when results ready

4. Result Types

• Main consumer: consumer-pat-test-result (3-25 replicas)
• Referral doctor: consumer-pat-test-result-refdoc (1-10 replicas)
• Walk-in: consumer-pat-test-result-walkin (1-5 replicas)

5. Imaging Reports

• consumer-imaging-service-report processes radiology reports (1-5 replicas)
• Integration with DICOM/Orthanc systems

6. Test Catalog

• consumer-pat-test-master-data syncs test catalog (1-6 replicas)
• consumer-pat-sync-test-master-data keeps catalog updated

Production Considerations

High Availability

Well Configured:

• API: 2 replicas with HPA (scales to 10)
• Main consumer: 3 replicas baseline (scales to 25)
• KEDA autoscaling on 7 consumers
• Multiple specialized consumers for different result types

x Single Points of Failure:

• Most consumers: 1 replica
• Workers: 1 replica each
• Cron: 1 replica

Auto-Scaling Performance

Workload	Type	Current	Min	Max	Load	Status
API	Standard HPA	2	1	10	CPU 14%, Mem 106MB	Low load
Main test result consumer	KEDA	3	3	25	Queue: 0	At baseline
Other consumers	KEDA	1 each	1	1-10	Queue: 0	Low lag

Current State: System running at low/moderate load with significant scaling headroom

Recommendations

1. Main Consumer Capacity:

   • Currently at 3 replicas baseline
   • Can scale to 25 for high load
   • Monitor queue depth during result publication peaks (typically mornings)

2. Consumer Resilience:

   • Critical consumers at 1 replica:
     • consumer-lis-testresult
     • consumer-pat-notification-test-result
     • consumer-imaging-service-report
   • Consider baseline of 2 replicas for critical paths

3. API Scaling:

   • Currently at 2 replicas (low load)
   • CPU at 14% - good headroom
   • Max of 10 replicas - review if sufficient for peak patient access

4. KEDA Thresholds:

   • Main consumer: 10 messages per replica
   • Refdoc/walkin: 20 messages per replica
   • Review if appropriate for your SLAs

5. Resource Cleanup:

   • 3 deployments scaled to 0
   • Review and remove if permanently unused:
     • consumer-mixpanel-events
     • consumer-pat-clear-order-data
     • wrk--default

6. Monitoring Priorities:

   • API response times (patient-facing)
   • Test result availability latency
   • Notification delivery success rate
   • Queue depths and consumer lag
   • Patient access patterns (peak hours)

Troubleshooting

API issues:

# Check API pods
kubectl get pods -n pat--test-result--be | grep "app--prod"

# Check HPA status
kubectl describe hpa pat--test-result--be--app--prod -n pat--test-result--be

# Check logs
kubectl logs -f deployment/pat--test-result--be--app--prod -n pat--test-result--be --tail=100

# Test API endpoint
kubectl port-forward -n pat--test-result--be service/pat--test-result--be--app--prod 8080:80
# Access http://localhost:8080/health

Test results not appearing:

# Check main consumer
kubectl get pods -n pat--test-result--be | grep "consumer-pat-test-result--prod"
kubectl logs -f deployment/pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

# Check LIS consumer
kubectl logs -f deployment/pat--test-result--be--consumer-lis-testresult--prod -n pat--test-result--be

# Check KEDA scaling
kubectl describe hpa keda-hpa-pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

# Check queue depth
kubectl describe scaledobject pat--test-result--be--consumer-pat-test-result--prod -n pat--test-result--be

Notifications not sending:

# Check notification consumer
kubectl logs -f deployment/pat--test-result--be--consumer-pat-notification-test-result--prod -n pat--test-result--be

# Check notification worker
kubectl logs -f deployment/pat--test-result--be--wrk--notifications--prod -n pat--test-result--be

# Restart both
kubectl rollout restart deployment/pat--test-result--be--consumer-pat-notification-test-result--prod -n pat--test-result--be
kubectl rollout restart deployment/pat--test-result--be--wrk--notifications--prod -n pat--test-result--be

Consumer lag issues:

# Check all KEDA HPAs
kubectl get hpa -n pat--test-result--be | grep keda

# Check specific consumer lag
kubectl describe hpa keda-hpa-pat--test-result--be--consumer-lis-testresult--prod -n pat--test-result--be

# Check KEDA operator
kubectl get pods -n keda
kubectl logs -n keda deployment/keda-operator

Imaging reports delayed:

# Check imaging consumer
kubectl logs -f deployment/pat--test-result--be--consumer-imaging-service-report--prod -n pat--test-result--be

# Check HPA status
kubectl describe hpa keda-hpa-pat--test-result--be--consumer-imaging-service-report--prod -n pat--test-result--be

# Check if scaling is needed
kubectl get scaledobject pat--test-result--be--consumer-imaging-service-report--prod -n pat--test-result--be

Performance Metrics

Current Scale

• API: 2 replicas (can scale to 10)
• Main Consumer: 3 replicas (can scale to 25)
• Other Consumers: 1 replica each (scale 1-10 based on type)
• Workers: 1 replica each
• Total Active Pods: ~21-23 pods

Traffic Patterns

Expected Peak Times:

• Morning (8-10 AM): Test results published from overnight lab runs
• Lunch (12-2 PM): Patient portal access during breaks
• Evening (6-8 PM): After-work patient access

Scaling Behavior:

• Main consumer scales based on result publication volume
• API scales based on patient access patterns
• Notification consumer scales based on notification queue

Integration Points

LIS Integration

• consumer-lis-testresult: Receives test results from lab
• consumer-lis-testresult-update: Handles result updates/corrections
• consumer-pat-sync-test-master-data: Syncs test catalog

Imaging Integration

• consumer-imaging-service-report: Receives radiology reports
• Integration with DICOM/Orthanc systems

Patient Portal

• API: Serves test results to patients
• consumer-pat-test-result: Main result processing (3-25 replicas)
• consumer-pat-test-result-refdoc: Referral doctor results
• consumer-pat-test-result-walkin: Walk-in patient results

Notification System

• consumer-pat-notification-test-result: Queues notifications
• wrk--notifications: Sends SMS/email notifications

Appointment System

• consumer-pat-appointment-sync: Links results to appointments
• consumer-bookings: Booking integration

Important Notes

x PRODUCTION ENVIRONMENT:

• This is a PATIENT-FACING CRITICAL SERVICE
• Downtime directly impacts patient access to test results
• SLA-critical: Patients expect immediate access when notified
• Test in staging thoroughly before production deployment
• Monitor patient access patterns carefully
• Have immediate rollback plan ready

x Compliance:

• Handles protected health information (PHI)
• HIPAA compliance required
• Audit logging for patient data access
• Secure data transmission

Patient Impact: This service enables patients to access their medical test results online, reducing phone calls to clinics and improving patient satisfaction.