Skip to main content

pat--authenticate

Overview​

  • Namespace: pat--authenticate
  • Purpose: Patient Authentication Backend - PRODUCTION
  • Age: ~3 years 60 days (since August 2022)
  • Status: Active - Patient authentication and authorization system
  • Workloads: 3 deployments + 1 CronJob (all active)
  • Environment: PRODUCTION - Critical authentication service

Architecture​

Patient authentication system handling login, registration, and LIS patient synchronization:

  • Main Application: REST API backend (3 replicas) - High Availability
  • Event Consumer: LIS patient data sync (3 replicas) - High Availability
  • Worker: Background job processing (1 deployment)
  • CronJob: Scheduled task (runs every minute)

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed replica counts (Main app: 3, Consumer: 3, Worker: 1)

Workload Categories​

Main Application (1 deployment)​

NameReplicasStatusPurpose
pat--authenticate--be--app--prod3/3RunningMain auth API (HA configured)

Event Consumer (1 deployment)​

NameReplicasStatusPurpose
consumer-lis-patient3/3RunningLIS patient sync (HA configured)

Workers (1 deployment)​

NameReplicasStatusPurpose
wrk--default1/1RunningDefault worker queue

Scheduler (1 CronJob)​

NameScheduleStatusPurpose
cron--prod* * * * * (every minute)ActiveScheduled tasks (high frequency)

Services​

NameTypeCluster IPPortsPurpose
pat--authenticate--be--app--prodClusterIP10.8.21.18280Main authentication API (internal)

Access & Management​

View all resources:​

kubectl get all -n pat--authenticate

Check main application:​

# View app pods (3 replicas)
kubectl get pods -n pat--authenticate | grep "app--prod"

# View logs from all replicas
kubectl logs -f deployment/pat--authenticate--be--app--prod -n pat--authenticate

# Check specific replica
kubectl logs -f deployment/pat--authenticate--be--app--prod -n pat--authenticate --all-containers=true

Check LIS consumer:​

# View consumer pods (3 replicas)
kubectl get pods -n pat--authenticate | grep consumer

# View logs from all consumer replicas
kubectl logs -f deployment/pat--authenticate--be--consumer-lis-patient--prod -n pat--authenticate

# Check specific consumer pod
for pod in $(kubectl get pods -n pat--authenticate | grep consumer | awk '{print $1}'); do
  echo "=== $pod ==="
  kubectl logs $pod -n pat--authenticate --tail=20
done

Check CronJob:​

# View CronJob
kubectl get cronjob -n pat--authenticate

# View recent job runs
kubectl get jobs -n pat--authenticate --sort-by=.status.startTime

# View CronJob pods
kubectl get pods -n pat--authenticate | grep cron

Restart services:​

# Restart main app (all 3 replicas)
kubectl rollout restart deployment/pat--authenticate--be--app--prod -n pat--authenticate

# Restart LIS consumer (all 3 replicas)
kubectl rollout restart deployment/pat--authenticate--be--consumer-lis-patient--prod -n pat--authenticate

# Restart worker
kubectl rollout restart deployment/pat--authenticate--be--wrk--default--prod -n pat--authenticate

Monitoring​

Resource usage:​

kubectl top pods -n pat--authenticate --sort-by=memory
kubectl top pods -n pat--authenticate --sort-by=cpu

Check CronJob executions:​

# Recent jobs
kubectl get jobs -n pat--authenticate --sort-by=.status.startTime | tail -10

# Failed jobs
kubectl get jobs -n pat--authenticate --field-selector status.successful=0

# Job logs
kubectl logs -n pat--authenticate job/<job-name>

Events:​

kubectl get events -n pat--authenticate --sort-by='.lastTimestamp' | head -20

Data Flow​

Patient Auth Request
    ↓
pat--authenticate--be--app--prod (ClusterIP - internal)
    ↓
Main Authentication API (3 replicas - HA)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
LIS Patient Consumer → consumer-lis-patient (3 replicas)
    ↓
Worker Processes Background Jobs
    ↓
CronJob → Scheduled Tasks (every minute)
    ↓
Patient auth updates, session management

Authentication Workflow​

1. Authentication API (High Availability)​

  • 3 replicas for redundancy and load distribution
  • Patient login and registration
  • Session management
  • Token generation and validation
  • Password management
  • Multi-factor authentication
  • OAuth/SSO integration

2. LIS Patient Synchronization (High Availability)​

  • 3 replicas for reliability
  • consumer-lis-patient processes patient sync events from LIS
  • Updates patient authentication records
  • Links patient accounts with lab system
  • Critical for cross-system authentication

3. Background Worker​

  • Async job processing
  • User provisioning tasks
  • Email/notification sending
  • Session cleanup

4. Scheduled Tasks (High Frequency)​

  • CronJob runs every minute (very high frequency)
  • Session expiration checks
  • Token cleanup
  • User sync operations
  • Health checks

Production Considerations​

High Availability​

Excellent Configuration:

  • Main API: 3 replicas for redundancy
  • LIS consumer: 3 replicas for reliability
  • ClusterIP service (internal only)
  • Very mature namespace (~3 years)

x Single Points of Failure:

  • Worker: 1 replica (consider 2 for HA)

x CronJob Frequency:

  • Runs every minute (very high frequency)
  • Monitor for resource impact
  • Review if all runs are necessary

Recommendations​

  1. Auto-Scaling (Optional):

    • Currently fixed at 3 replicas (good baseline)
    • Consider HPA to scale during peak auth times
    • Target: 3-10 replicas based on load

  2. Worker Resilience:

    • Currently 1 replica
    • Consider 2 replicas for HA
    • Critical for async auth operations

  3. CronJob Review:

    • Runs every minute (very frequent)
    • Review if frequency can be reduced
    • Monitor resource usage from cron jobs
    • Consider if some tasks can be event-driven

  4. Monitoring Priorities:

    • API response times (3 replicas handling load)
    • Authentication success/failure rates
    • LIS sync status (3 consumer replicas)
    • CronJob execution success rate
    • Session management performance

  5. Security:

    • ClusterIP service (internal only) - good security
    • Monitor authentication patterns
    • Track failed login attempts
    • Session security audits

Troubleshooting​

Main API issues:​

# Check all 3 API pods
kubectl get pods -n pat--authenticate | grep "app--prod"

# Check logs from all replicas
kubectl logs deployment/pat--authenticate--be--app--prod -n pat--authenticate --all-containers=true --tail=100

# Check specific pod
POD_NAME=$(kubectl get pods -n pat--authenticate | grep "app--prod" | head -1 | awk '{print $1}')
kubectl logs $POD_NAME -n pat--authenticate --tail=100

# Test API endpoint (internal - need port-forward)
kubectl port-forward -n pat--authenticate service/pat--authenticate--be--app--prod 8080:80
# Access http://localhost:8080

LIS sync issues:​

# Check all 3 consumer pods
kubectl get pods -n pat--authenticate | grep consumer

# Check logs from all consumer replicas
kubectl logs deployment/pat--authenticate--be--consumer-lis-patient--prod -n pat--authenticate --all-containers=true --tail=100

# Check each replica separately
for pod in $(kubectl get pods -n pat--authenticate | grep consumer | awk '{print $1}'); do
  echo "=== $pod ==="
  kubectl logs $pod -n pat--authenticate --tail=50 | grep -i "error\|sync\|lis"
done

# Restart all consumer replicas
kubectl rollout restart deployment/pat--authenticate--be--consumer-lis-patient--prod -n pat--authenticate

Authentication failures:​

# Check API logs for auth errors
kubectl logs deployment/pat--authenticate--be--app--prod -n pat--authenticate --tail=200 | grep -i "auth\|login\|fail\|error"

# Check across all replicas
for pod in $(kubectl get pods -n pat--authenticate | grep "app--prod" | awk '{print $1}'); do
  echo "=== $pod ==="
  kubectl logs $pod -n pat--authenticate --tail=50 | grep -i "401\|403\|auth.*fail"
done

CronJob issues:​

# Check CronJob status
kubectl get cronjob -n pat--authenticate

# Check recent jobs
kubectl get jobs -n pat--authenticate --sort-by=.status.startTime | tail -20

# Check failed jobs
kubectl get jobs -n pat--authenticate --field-selector status.successful=0

# Check specific job logs
kubectl logs -n pat--authenticate job/<job-name>

# Delete old completed jobs (if too many)
kubectl delete jobs -n pat--authenticate --field-selector status.successful=1

Worker issues:​

# Check worker
kubectl logs -f deployment/pat--authenticate--be--wrk--default--prod -n pat--authenticate

# Check for job processing errors
kubectl logs deployment/pat--authenticate--be--wrk--default--prod -n pat--authenticate --tail=100 | grep -i "error\|fail"

# Restart worker
kubectl rollout restart deployment/pat--authenticate--be--wrk--default--prod -n pat--authenticate

Load distribution issues:​

# Check resource usage across API replicas
kubectl top pods -n pat--authenticate | grep app--prod

# Check resource usage across consumer replicas
kubectl top pods -n pat--authenticate | grep consumer

# Restart all to redistribute load
kubectl rollout restart deployment/pat--authenticate--be--app--prod -n pat--authenticate
kubectl rollout restart deployment/pat--authenticate--be--consumer-lis-patient--prod -n pat--authenticate

Performance Metrics​

Current Scale​

  • Main API: 3 replicas (excellent HA)
  • LIS Consumer: 3 replicas (excellent HA)
  • Worker: 1 replica
  • CronJob: Runs every minute (high frequency)
  • Total Active Pods: ~7 pods + CronJob pods

Stability​

  • Namespace Age: ~3 years (very mature, stable)
  • Recent Updates: 205 days ago (very stable)
  • HA Configuration: 3+3 replicas (excellent)
  • Service Type: ClusterIP (internal only - good security)