Skip to main content

spc--appointment--be

Overview​

  • Namespace: spc--appointment--be
  • Purpose: Sapoche Appointment Management Backend - PRODUCTION
  • Age: 98 days (~3 months, since July 2025)
  • Status: Active - Appointment scheduling and management
  • Workloads: 8 deployments (7 active, 1 scaled to 0)
  • Environment: PRODUCTION - Medical appointment system

Architecture​

Appointment management system handling appointment scheduling, notifications, and integration with LIS and POS:

  • Main Application: REST API backend (2 replicas)
  • Event Consumers: Process appointments, samples, notifications, orders (3 active + 1 scaled to 0)
  • Worker: Batch job publisher (1 deployment)
  • Scheduler: Cron jobs for scheduled tasks (1 deployment)
  • Observability: OpenTelemetry collector for tracing

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed replica counts for all deployments
  • Main app: 2 replicas (manual HA)

Workload Categories​

Main Application (1 deployment)​

NameReplicasStatusPurpose
spc--appointment--be--app--prod2/2RunningMain appointment API (2 replicas for HA)

The main application handles:

  • Appointment scheduling and booking
  • Appointment modifications and cancellations
  • Schedule management
  • Integration with clinic resources
  • RESTful API for frontend and integrations

Event Consumers (4 deployments - 3 active)​

Process appointment-related events from message queues:

NameReplicasStatusPurpose
consumer-lis-sample-status1/1RunningLIS sample status updates
consumer-notification1/1RunningAppointment notifications
consumer-pos-orders1/1RunningPOS order integration
consumer-checkup0/0x Scaled to 0Checkup appointments (inactive - 29 days)

Workers & Scheduler (2 deployments)​

NameReplicasStatusPurpose
wrk--batch-publisher1/1RunningBatch job publisher
cron--prod1/1RunningScheduled cron jobs

Observability (1 deployment)​

NameReplicasStatusPurpose
spc--appointment--be-otel-collector1/1RunningOpenTelemetry trace collector

Services​

NameTypeCluster IPPortsNodePortPurpose
spc--appointment--be--app--prodNodePort10.8.25.1718031432Main appointment API
spc--appointment--be-otel-collectorClusterIP10.8.23.2144317, 4318-OTLP trace collection

Access & Management​

View all resources:​

kubectl get all -n spc--appointment--be

Check main application:​

# View app pods (2 replicas)
kubectl get pods -n spc--appointment--be | grep "app--prod"

# View logs
kubectl logs -f deployment/spc--appointment--be--app--prod -n spc--appointment--be

# Check both replicas
kubectl logs -f deployment/spc--appointment--be--app--prod -n spc--appointment--be --all-containers=true

Check consumers:​

# All active consumers
kubectl get pods -n spc--appointment--be | grep consumer | grep Running

# Scaled to 0
kubectl get deployments -n spc--appointment--be | grep "0/0"

# Consumer logs
kubectl logs -f deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

Check cron jobs:​

# View cron pod
kubectl get pods -n spc--appointment--be | grep cron

# Cron logs
kubectl logs -f deployment/spc--appointment--be--cron--prod -n spc--appointment--be

Restart services:​

# Restart main app
kubectl rollout restart deployment/spc--appointment--be--app--prod -n spc--appointment--be

# Restart specific consumer
kubectl rollout restart deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

# Restart all consumers (active only)
kubectl get deployments -n spc--appointment--be | grep consumer | grep -v "0/0" | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n spc--appointment--be

Monitoring​

Resource usage:​

kubectl top pods -n spc--appointment--be --sort-by=memory
kubectl top pods -n spc--appointment--be --sort-by=cpu

Deployment status:​

# All deployments
kubectl get deployments -n spc--appointment--be

# Active deployments
kubectl get deployments -n spc--appointment--be | grep -v "0/0"

# Scaled to 0
kubectl get deployments -n spc--appointment--be | grep "0/0"

Events:​

kubectl get events -n spc--appointment--be --sort-by='.lastTimestamp' | head -20

Traces (OpenTelemetry):​

# Check OTEL collector
kubectl logs -f deployment/spc--appointment--be-otel-collector -n spc--appointment--be

# Port forward to access OTEL endpoints
kubectl port-forward -n spc--appointment--be deployment/spc--appointment--be-otel-collector 4317:4317

Data Flow​

Appointment Request
    ↓
spc--appointment--be--app--prod (NodePort 31432)
    ↓
Main Appointment API (2 replicas)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Consumers Process Events
    ├─ LIS Sample Status → consumer-lis-sample-status
    ├─ Notifications → consumer-notification
    └─ POS Orders → consumer-pos-orders
    ↓
Batch Publisher Creates Background Jobs
    ↓
Appointment confirmations, reminders, updates

OpenTelemetry Tracing​

Application → OTEL Collector (4317/4318) → Backend (Grafana/Jaeger)

Appointment Workflow​

1. Appointment Scheduling​

  • Patient requests appointment via portal or staff
  • API validates availability and resources
  • Creates appointment in database
  • Publishes events to message queue

2. LIS Sample Integration​

  • Sample status updates from LIS
  • consumer-lis-sample-status processes sample events
  • Updates appointment status based on sample collection
  • Links appointments to lab orders

3. Notification Processing​

  • Appointment confirmations
  • Reminder notifications (SMS, email, push)
  • consumer-notification handles all notifications
  • Cancellation and reschedule alerts

4. POS Order Integration​

  • Integration with POS for billing
  • consumer-pos-orders processes order events
  • Links appointments to billing records
  • Payment confirmations

5. Checkup Appointments (Inactive)​

  • consumer-checkup scaled to 0 (29 days)
  • Possibly deprecated or seasonal
  • Review if still needed

6. Scheduled Tasks​

  • Cron job for periodic tasks
  • Appointment reminders
  • Status updates
  • Data cleanup

Production Considerations​

High Availability​

Well Configured:

  • Main API: 2 replicas for redundancy
  • Multiple independent consumers

x Single Points of Failure:

  • All consumers: 1 replica each
  • Batch publisher: 1 replica
  • Cron job: 1 replica
  • OTEL collector: 1 replica

No Auto-Scaling​

  • No HPAs or KEDA configured
  • Fixed replica counts
  • Cannot automatically scale based on load
  • May struggle during peak booking times

Recommendations​

  1. Add Auto-Scaling:

    • Consider HPA for main API (currently 2 fixed replicas)
    • Add KEDA for consumers based on queue depth:
      • consumer-notification (likely high volume)
      • consumer-pos-orders
      • consumer-lis-sample-status

  2. Consumer Resilience:

    • All consumers at 1 replica
    • Consider 2 replicas for critical consumers:
      • consumer-notification (critical for patient communication)
      • consumer-pos-orders (critical for billing)

  3. Cleanup Opportunity:

    • consumer-checkup scaled to 0 for 29 days
    • Review and remove if permanently unused

  4. Main API Scaling:

    • Currently fixed at 2 replicas (good baseline)
    • Add HPA to scale during peak booking hours
    • Target: 2-10 replicas based on CPU/memory

  5. Observability:

    • OTEL collector at 1 replica
    • Consider 2+ replicas for reliability
    • Monitor trace collection lag

  6. Monitoring Priorities:

    • API response times
    • Appointment booking success rates
    • Notification delivery success
    • LIS integration status
    • Consumer lag monitoring

Troubleshooting​

Main API issues:​

# Check both API pods
kubectl get pods -n spc--appointment--be | grep "app--prod"

# Check logs from both replicas
kubectl logs deployment/spc--appointment--be--app--prod -n spc--appointment--be --all-containers=true --tail=100

# Test API endpoint
kubectl port-forward -n spc--appointment--be service/spc--appointment--be--app--prod 8080:80
# Access http://localhost:8080

Consumer not processing:​

# Check consumer status
kubectl get pods -n spc--appointment--be | grep consumer-notification

# Check logs
kubectl logs -f deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

# Check for errors
kubectl logs deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be --tail=100 | grep -i error

# Restart consumer
kubectl rollout restart deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

Notification delays:​

# Check notification consumer
kubectl logs -f deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

# Check resource usage
kubectl top pods -n spc--appointment--be | grep notification

# Check if stuck
kubectl get pods -n spc--appointment--be | grep notification

# Restart notification consumer
kubectl rollout restart deployment/spc--appointment--be--consumer-notification--prod -n spc--appointment--be

LIS sample status issues:​

# Check LIS consumer
kubectl logs -f deployment/spc--appointment--be--consumer-lis-sample-status--prod -n spc--appointment--be

# Check for LIS connectivity
kubectl logs deployment/spc--appointment--be--consumer-lis-sample-status--prod -n spc--appointment--be --tail=100 | grep -i "lis\|sample\|status"

# Restart consumer
kubectl rollout restart deployment/spc--appointment--be--consumer-lis-sample-status--prod -n spc--appointment--be

POS integration issues:​

# Check POS consumer
kubectl logs -f deployment/spc--appointment--be--consumer-pos-orders--prod -n spc--appointment--be

# Check for POS connectivity
kubectl logs deployment/spc--appointment--be--consumer-pos-orders--prod -n spc--appointment--be --tail=100 | grep -i "pos\|order\|billing"

# Restart consumer
kubectl rollout restart deployment/spc--appointment--be--consumer-pos-orders--prod -n spc--appointment--be

Cron job failures:​

# Check cron pod
kubectl get pods -n spc--appointment--be | grep cron

# Check cron logs
kubectl logs -f deployment/spc--appointment--be--cron--prod -n spc--appointment--be

# Restart cron
kubectl rollout restart deployment/spc--appointment--be--cron--prod -n spc--appointment--be

Scaled-to-0 investigation:​

# Check checkup consumer
kubectl get deployment spc--appointment--be--consumer-checkup--prod -n spc--appointment--be

# View deployment history
kubectl rollout history deployment/spc--appointment--be--consumer-checkup--prod -n spc--appointment--be

# Scale up if needed for testing
kubectl scale deployment spc--appointment--be--consumer-checkup--prod -n spc--appointment--be --replicas=1

Tracing issues:​

# Check OTEL collector
kubectl logs -f deployment/spc--appointment--be-otel-collector -n spc--appointment--be

# Check collector metrics
kubectl port-forward -n spc--appointment--be deployment/spc--appointment--be-otel-collector 8888:8888
# Access http://localhost:8888/metrics

Performance Metrics​

Current Scale​

  • Main API: 2 replicas (fixed, no HPA)
  • Consumers: 3 active at 1 replica each
  • Workers: 1 batch publisher, 1 cron
  • Observability: 1 OTEL collector
  • Total Active Pods: ~8 pods

Stability​

  • Namespace Age: 98 days (~3 months) - relatively new
  • Recent Redeployments: 27 days ago (stable)
  • consumer-checkup: Scaled to 0 for 29 days (review needed)

Integration Points​

External Systems​

  1. Laboratory Information System (LIS):

    • Sample status updates
    • Lab order linking
    • consumer-lis-sample-status handles LIS events

  2. Point of Sale (POS):

    • Billing integration
    • Order processing
    • consumer-pos-orders handles POS events

  3. Notification Systems:

    • SMS notifications
    • Email notifications
    • Push notifications
    • consumer-notification handles all notifications

  4. Patient Portal:

    • Appointment booking interface
    • Schedule viewing
    • Appointment management

Internal Systems​

  • Appointment Database: Central appointment data store
  • Schedule Service: Resource and schedule management
  • Billing Service: Integration with financial systems
  • Patient Service: Patient data integration

Important Notes​

x PRODUCTION ENVIRONMENT:

  • This is a CRITICAL APPOINTMENT system
  • Downtime impacts patient scheduling and clinic operations
  • Changes must be tested in staging first
  • Monitor appointment booking success rates carefully
  • Have immediate rollback plan ready

x No Auto-Scaling:

  • No HPAs or KEDA configured
  • Fixed replica counts cannot adapt to load
  • May struggle during peak booking times
  • Consider adding auto-scaling

x Cleanup Opportunity:

  • consumer-checkup scaled to 0 for 29 days
  • Review and remove if permanently unused

x Single Replica Consumers:

  • All 3 active consumers at 1 replica
  • No redundancy for notification or POS integration
  • Consider 2 replicas for critical consumers