pat--notification--be

Overview

• Namespace: pat--notification--be
• Purpose: Patient Portal Notification Backend - PRODUCTION
• Age: ~2 years 266 days (since February 2023)
• Status: Active - Patient notification system
• Workloads: 8 deployments (all active)
• Environment: PRODUCTION - Patient communication platform

Architecture

Patient notification system handling test result notifications, general notifications, and updates:

• Main Application: REST API backend (1 replica)
• Event Consumers: Process notifications and test results (3 deployments, 12 total pods)
• Workers: Background job processing (3 deployments)
• Scheduler: Cron jobs for scheduled tasks

Auto-Scaling Configuration

No Auto-Scaling Configured:

• No HorizontalPodAutoscalers (HPAs)
• No KEDA scaled objects
• Fixed replica counts (high volume handled by multiple replicas)

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
pat--notification--be--app--prod	1/1	Running	Main patient notification API

Event Consumers (3 deployments - 12 total pods)

Name	Replicas	Status	Purpose
consumer-test-result-notification	6/6	Running	Test result notifications (high volume)
consumer-notification	3/3	Running	General patient notifications
consumer-notification-update	3/3	Running	Notification updates

Workers (3 deployments)

Name	Replicas	Status	Purpose
wrk--batch-publisher	1/1	Running	Batch job publisher
wrk--default	1/1	Running	Default worker queue
wrk--notifications	1/1	Running	Notification processing

Scheduler (1 deployment)

Name	Replicas	Status	Purpose
cron--prod	1/1	Running	Scheduled cron jobs

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
pat--notification--be--app--prod	NodePort	10.8.26.180	80	30181	Main notification API

Access & Management

View all resources:

kubectl get all -n pat--notification--be

Check main application:

kubectl get pods -n pat--notification--be | grep "app--prod"
kubectl logs -f deployment/pat--notification--be--app--prod -n pat--notification--be

Check high-volume consumers:

# Test result notifications (6 replicas)
kubectl get pods -n pat--notification--be | grep test-result-notification
kubectl logs -f deployment/pat--notification--be--consumer-test-result-notification--prod -n pat--notification--be

# General notifications (3 replicas)
kubectl get pods -n pat--notification--be | grep "consumer-notification--prod"
kubectl logs -f deployment/pat--notification--be--consumer-notification--prod -n pat--notification--be

Restart services:

# Restart main app
kubectl rollout restart deployment/pat--notification--be--app--prod -n pat--notification--be

# Restart test result consumer (6 replicas)
kubectl rollout restart deployment/pat--notification--be--consumer-test-result-notification--prod -n pat--notification--be

# Restart all consumers
kubectl get deployments -n pat--notification--be | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n pat--notification--be

Monitoring

Resource usage:

kubectl top pods -n pat--notification--be --sort-by=memory
kubectl top pods -n pat--notification--be --sort-by=cpu

Check high-volume consumers:

# Test result consumer (6 replicas)
kubectl top pods -n pat--notification--be | grep test-result

# All notification consumers
kubectl top pods -n pat--notification--be | grep consumer

Events:

kubectl get events -n pat--notification--be --sort-by='.lastTimestamp' | head -20

Data Flow

Patient Notification Events
    ↓
pat--notification--be--app--prod (NodePort 30181)
    ↓
Message Queue (Kafka/Redpanda)
    ↓
Consumers Process Events (12 pods total)
    ├─ Test Results → consumer-test-result-notification (6 replicas - highest volume)
    ├─ General → consumer-notification (3 replicas)
    └─ Updates → consumer-notification-update (3 replicas)
    ↓
Workers Process Background Jobs
    ↓
SMS, Email, Push notifications to patients

Patient Notification Workflow

1. Test Result Notifications (Highest Volume)

• 6 replicas dedicated to test result notifications
• Notifies patients when test results are ready
• Highest priority and volume consumer
• Critical for patient experience

2. General Notifications

• 3 replicas for general patient notifications
• Appointment reminders, confirmations
• General patient communications
• Multi-channel delivery (SMS, email, push)

3. Notification Updates

• 3 replicas for notification status updates
• Delivery confirmations
• Read receipts
• Notification state management

4. Background Workers

• wrk--notifications: Processes notification jobs
• wrk--default: General background processing
• wrk--batch-publisher: Batch job publishing

5. Scheduled Tasks

• Cron jobs for periodic notifications
• Scheduled reminders
• Notification cleanup

Production Considerations

High Availability

Well Configured for Volume:

• consumer-test-result-notification: 6 replicas (high volume)
• consumer-notification: 3 replicas (medium volume)
• consumer-notification-update: 3 replicas (medium volume)
• Total: 12 consumer pods handling patient notifications

x Single Points of Failure:

• Main API: 1 replica (no HA)
• All workers: 1 replica each
• Cron job: 1 replica

Recommendations

1. Main API Resilience:

   • Currently 1 replica (single point of failure)
   • Increase to 2+ replicas or add HPA
   • Critical entry point for patient notifications

2. Add Auto-Scaling:

   • Consider KEDA for queue-based scaling:
     • consumer-test-result-notification (6-15 replicas)
     • consumer-notification (3-10 replicas)
     • consumer-notification-update (3-8 replicas)
   • Currently fixed at high replica counts (may over-provision during low traffic)

3. Worker Resilience:

   • wrk--notifications: 1 replica (consider 2 for HA)
   • wrk--default: 1 replica (consider 2 for HA)

4. Capacity Planning:

   • consumer-test-result-notification: 6 replicas (review if sufficient)
   • Total 12 consumer pods (significant resource allocation)
   • Monitor during peak result delivery times

5. Monitoring Priorities:

   • Test result notification delivery rates
   • Notification delivery success rates
   • Consumer lag (especially test results - 6 replicas)
   • SMS/email/push delivery success

Troubleshooting

Test result notification delays:

# Check all 6 replicas
kubectl get pods -n pat--notification--be | grep test-result-notification

# Check logs from all replicas
kubectl logs deployment/pat--notification--be--consumer-test-result-notification--prod -n pat--notification--be --all-containers=true --tail=100

# Check resource usage
kubectl top pods -n pat--notification--be | grep test-result

# Restart consumer (all 6 replicas)
kubectl rollout restart deployment/pat--notification--be--consumer-test-result-notification--prod -n pat--notification--be

General notification issues:

# Check notification consumer (3 replicas)
kubectl get pods -n pat--notification--be | grep "consumer-notification--prod"

# Check logs
kubectl logs deployment/pat--notification--be--consumer-notification--prod -n pat--notification--be --tail=100

# Restart consumer
kubectl rollout restart deployment/pat--notification--be--consumer-notification--prod -n pat--notification--be

Notification update issues:

# Check update consumer (3 replicas)
kubectl logs deployment/pat--notification--be--consumer-notification-update--prod -n pat--notification--be --tail=100

# Restart consumer
kubectl rollout restart deployment/pat--notification--be--consumer-notification-update--prod -n pat--notification--be

Worker issues:

# Check notification worker
kubectl logs -f deployment/pat--notification--be--wrk--notifications--prod -n pat--notification--be

# Check default worker
kubectl logs -f deployment/pat--notification--be--wrk--default--prod -n pat--notification--be

# Restart workers
kubectl rollout restart deployment/pat--notification--be--wrk--notifications--prod -n pat--notification--be
kubectl rollout restart deployment/pat--notification--be--wrk--default--prod -n pat--notification--be

Performance Metrics

Current Scale

• Main API: 1 replica
• High-Volume Consumer: 6 replicas (test result notifications)
• Medium-Volume Consumers: 2 consumers at 3 replicas each (6 total)
• Workers: 3 workers at 1 replica each
• Total Active Pods: ~18 pods

Volume Distribution

• Test Results: 6 replicas (50% of consumer capacity)
• General Notifications: 3 replicas (25% of consumer capacity)
• Notification Updates: 3 replicas (25% of consumer capacity)

Stability

• Namespace Age: ~2 years (mature, stable)
• Recent Updates: 23 days ago
• High Replica Counts: Indicates high notification volume