pat--patient-mgt--be

Overview

• Namespace: pat--patient-mgt--be
• Purpose: Patient Management Backend - PRODUCTION
• Age: ~2 years 196 days (since May 2023)
• Status: Active - Patient data management system
• Workloads: 5 deployments (all active)
• Environment: PRODUCTION - Patient record management

Architecture

Patient management system handling patient data, LIS integration, and notifications:

• Main Application: REST API backend (1 replica)
• Event Consumer: LIS patient data sync (1 deployment)
• Workers: Background job processing (2 deployments)
• Scheduler: Cron jobs for scheduled tasks

Auto-Scaling Configuration

No Auto-Scaling Configured:

• No HorizontalPodAutoscalers (HPAs)
• No KEDA scaled objects
• Fixed replica counts (all at 1 replica)

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
pat--patient-mgt--be--app--prod	1/1	Running	Main patient management API

Event Consumer (1 deployment)

Name	Replicas	Status	Purpose
consumer-lis-patient	1/1	Running	LIS patient data synchronization

Workers (2 deployments)

Name	Replicas	Status	Purpose
wrk--default	1/1	Running	Default worker queue
wrk--notifications	1/1	Running	Notification processing

Scheduler (1 deployment)

Name	Replicas	Status	Purpose
cron--prod	1/1	Running	Scheduled cron jobs

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
pat--patient-mgt--be--app--prod	NodePort	10.8.24.165	80	31418	Main patient management API

Access & Management

View all resources:

kubectl get all -n pat--patient-mgt--be

Check main application:

kubectl get pods -n pat--patient-mgt--be | grep "app--prod"
kubectl logs -f deployment/pat--patient-mgt--be--app--prod -n pat--patient-mgt--be

Check LIS consumer:

kubectl get pods -n pat--patient-mgt--be | grep consumer
kubectl logs -f deployment/pat--patient-mgt--be--consumer-lis-patient--prod -n pat--patient-mgt--be

Check workers:

kubectl get pods -n pat--patient-mgt--be | grep wrk
kubectl logs -f deployment/pat--patient-mgt--be--wrk--notifications--prod -n pat--patient-mgt--be

Restart services:

# Restart main app
kubectl rollout restart deployment/pat--patient-mgt--be--app--prod -n pat--patient-mgt--be

# Restart LIS consumer
kubectl rollout restart deployment/pat--patient-mgt--be--consumer-lis-patient--prod -n pat--patient-mgt--be

# Restart all workers
kubectl get deployments -n pat--patient-mgt--be | grep wrk | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n pat--patient-mgt--be

Monitoring

Resource usage:

kubectl top pods -n pat--patient-mgt--be --sort-by=memory
kubectl top pods -n pat--patient-mgt--be --sort-by=cpu

Events:

kubectl get events -n pat--patient-mgt--be --sort-by='.lastTimestamp' | head -20

Data Flow

Patient Management Request
    ↓
pat--patient-mgt--be--app--prod (NodePort 31418)
    ↓
Main Patient Management API
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
LIS Patient Consumer → consumer-lis-patient
    ↓
Workers Process Background Jobs
    ↓
Patient notifications, data sync

Patient Management Workflow

1. Patient Data Management

• Patient registration and profile updates
• Patient demographics management
• Medical history tracking
• API validates and stores patient data

2. LIS Patient Integration

• Patient data synchronization with LIS
• consumer-lis-patient processes patient events from LIS
• Updates patient records with lab-related information
• Links patients to test orders

3. Background Workers

• wrk--notifications: Processes patient notification jobs
• wrk--default: General background processing
• Patient updates, data validation

4. Scheduled Tasks

• Cron jobs for periodic patient data tasks
• Data cleanup and validation
• Scheduled notifications

Production Considerations

High Availability

x Single Points of Failure:

• Main API: 1 replica (no HA)
• LIS consumer: 1 replica (no HA)
• All workers: 1 replica each
• Cron job: 1 replica

Recommendations

1. Main API Resilience:

   • Currently 1 replica (single point of failure)
   • Increase to 2+ replicas or add HPA
   • Critical for patient data access

2. Add Auto-Scaling:

   • Consider HPA for main API
   • Add KEDA for LIS consumer based on queue depth
   • Scale during peak registration times

3. Consumer Resilience:

   • consumer-lis-patient: 1 replica (single point of failure)
   • Consider 2 replicas for reliability
   • Critical for LIS data synchronization

4. Worker Resilience:

   • wrk--notifications: 1 replica (consider 2 for HA)
   • wrk--default: 1 replica (consider 2 for HA)

5. Monitoring Priorities:

   • API response times
   • Patient registration success rates
   • LIS sync status and lag
   • Worker queue depth

Troubleshooting

Main API issues:

# Check API pod
kubectl get pods -n pat--patient-mgt--be | grep "app--prod"

# Check logs
kubectl logs -f deployment/pat--patient-mgt--be--app--prod -n pat--patient-mgt--be --tail=100

# Test API endpoint
kubectl port-forward -n pat--patient-mgt--be service/pat--patient-mgt--be--app--prod 8080:80
# Access http://localhost:8080

LIS sync issues:

# Check LIS consumer
kubectl logs -f deployment/pat--patient-mgt--be--consumer-lis-patient--prod -n pat--patient-mgt--be

# Check for sync errors
kubectl logs deployment/pat--patient-mgt--be--consumer-lis-patient--prod -n pat--patient-mgt--be --tail=100 | grep -i "error\|sync\|lis"

# Restart consumer
kubectl rollout restart deployment/pat--patient-mgt--be--consumer-lis-patient--prod -n pat--patient-mgt--be

Worker issues:

# Check notification worker
kubectl logs -f deployment/pat--patient-mgt--be--wrk--notifications--prod -n pat--patient-mgt--be

# Check default worker
kubectl logs -f deployment/pat--patient-mgt--be--wrk--default--prod -n pat--patient-mgt--be

# Restart workers
kubectl rollout restart deployment/pat--patient-mgt--be--wrk--notifications--prod -n pat--patient-mgt--be
kubectl rollout restart deployment/pat--patient-mgt--be--wrk--default--prod -n pat--patient-mgt--be

Cron job failures:

# Check cron pod
kubectl get pods -n pat--patient-mgt--be | grep cron

# Check cron logs
kubectl logs -f deployment/pat--patient-mgt--be--cron--prod -n pat--patient-mgt--be

# Restart cron
kubectl rollout restart deployment/pat--patient-mgt--be--cron--prod -n pat--patient-mgt--be

Performance Metrics

Current Scale

• Main API: 1 replica (no HA)
• Consumer: 1 replica (LIS patient sync)
• Workers: 2 workers at 1 replica each
• Total Active Pods: ~5 pods

Stability

• Namespace Age: ~2 years (mature, stable)
• Recent Updates: 205 days ago (stable)
• All Single Replicas: No redundancy configured