rfd--doctor-statement

Overview

• Namespace: rfd--doctor-statement
• Purpose: Referring Doctor Statement Generation Backend - PRODUCTION
• Age: ~2 years 251 days (since February 2023)
• Status: Active - Doctor financial statements
• Workloads: 6 deployments (all active)
• Environment: PRODUCTION - Financial statement processing

Architecture

Doctor statement system handling statement generation, PDF creation, OTP verification, and link events:

• Main Application: REST API backend (1 replica)
• Event Consumers: Statement processing, PDF generation, OTP, link events (4 deployments)
• Worker: Batch job publisher (1 deployment)

Auto-Scaling Configuration

No Auto-Scaling Configured:

• No HorizontalPodAutoscalers (HPAs)
• No KEDA scaled objects
• Fixed replica counts (all at 1 replica)

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
rfd--doctor-statement--be--app--prod	1/1	Running	Main statement API

Event Consumers (4 deployments)

Name	Replicas	Status	Purpose
consumer-doctor-statement	1/1	Running	Doctor statement processing
consumer-generate-statement-pdf	1/1	Running	PDF statement generation
consumer-send-otp	1/1	Running	OTP for statement access
consumer-statement-link-event	1/1	Running	Statement link event tracking

Workers (1 deployment)

Name	Replicas	Status	Purpose
wrk--batch-publisher	1/1	Running	Batch job publisher

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
rfd--doctor-statement--be--app--prod	NodePort	10.8.16.166	80	30508	Main statement API

Access & Management

View all resources:

kubectl get all -n rfd--doctor-statement

Check main application:

kubectl get pods -n rfd--doctor-statement | grep "app--prod"
kubectl logs -f deployment/rfd--doctor-statement--be--app--prod -n rfd--doctor-statement

Check consumers:

# All consumers
kubectl get pods -n rfd--doctor-statement | grep consumer

# Statement processing
kubectl logs -f deployment/rfd--doctor-statement--be--consumer-doctor-statement--prod -n rfd--doctor-statement

# PDF generation
kubectl logs -f deployment/rfd--doctor-statement--be--consumer-generate-statement-pdf--prod -n rfd--doctor-statement

Restart services:

# Restart main app
kubectl rollout restart deployment/rfd--doctor-statement--be--app--prod -n rfd--doctor-statement

# Restart all consumers
kubectl get deployments -n rfd--doctor-statement | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n rfd--doctor-statement

Monitoring

Resource usage:

kubectl top pods -n rfd--doctor-statement --sort-by=memory
kubectl top pods -n rfd--doctor-statement --sort-by=cpu

Events:

kubectl get events -n rfd--doctor-statement --sort-by='.lastTimestamp' | head -20

Data Flow

Doctor Statement Request
    ↓
rfd--doctor-statement--be--app--prod (NodePort 30508)
    ↓
Main Statement API
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ↓
Consumers Process Events
    ├─ Statement Data → consumer-doctor-statement
    ├─ PDF Generation → consumer-generate-statement-pdf
    ├─ OTP Sending → consumer-send-otp
    └─ Link Tracking → consumer-statement-link-event
    ↓
Batch Publisher Creates Background Jobs
    ↓
Financial statements, PDF delivery, access control

Doctor Statement Workflow

1. Statement API

• Financial statement requests
• Statement data aggregation
• Period selection
• Export options
• Access control

2. Statement Processing

• consumer-doctor-statement processes statement events
• Financial data aggregation
• Transaction summaries
• Commission calculations

3. PDF Generation

• consumer-generate-statement-pdf creates PDF statements
• Financial report formatting
• Template rendering
• PDF storage and delivery

4. OTP Verification

• consumer-send-otp handles secure access
• OTP generation for sensitive statements
• SMS/Email delivery
• Access authorization

5. Link Event Tracking

• consumer-statement-link-event tracks statement access
• Download tracking
• View analytics
• Audit trail

6. Batch Publisher

• Creates background jobs for large statements
• Scheduled statement generation
• Batch processing

Production Considerations

High Availability

Single Points of Failure:

• Main API: 1 replica (no HA)
• All consumers: 1 replica each
• Batch publisher: 1 replica
• No redundancy for financial processing

Recommendations

1. Main API Resilience:

   • Currently 1 replica (single point of failure)
   • Increase to 2+ replicas or add HPA
   • Critical for financial statement access

2. Add Auto-Scaling:

   • Consider HPA for main API
   • Add KEDA for consumers based on queue depth
   • Scale during month-end statement generation

3. Consumer Resilience:

   • All consumers at 1 replica
   • Consider 2 replicas for critical consumers:
     • consumer-generate-statement-pdf (PDF generation)
     • consumer-doctor-statement (financial processing)

4. Recent Updates:

   • Recent deployment: 29 days ago
   • Regular maintenance updates
   • Monitor stability

5. Monitoring Priorities:

   • Statement generation success rates
   • PDF creation performance
   • OTP delivery success
   • Financial data accuracy
   • Consumer lag monitoring

Troubleshooting

Main API issues:

kubectl logs -f deployment/rfd--doctor-statement--be--app--prod -n rfd--doctor-statement --tail=100
kubectl port-forward -n rfd--doctor-statement service/rfd--doctor-statement--be--app--prod 8080:80

Statement processing issues:

kubectl logs -f deployment/rfd--doctor-statement--be--consumer-doctor-statement--prod -n rfd--doctor-statement
kubectl logs deployment/rfd--doctor-statement--be--consumer-doctor-statement--prod -n rfd--doctor-statement --tail=100 | grep -i "error\|statement\|fail"
kubectl rollout restart deployment/rfd--doctor-statement--be--consumer-doctor-statement--prod -n rfd--doctor-statement

PDF generation issues:

kubectl logs -f deployment/rfd--doctor-statement--be--consumer-generate-statement-pdf--prod -n rfd--doctor-statement
kubectl logs deployment/rfd--doctor-statement--be--consumer-generate-statement-pdf--prod -n rfd--doctor-statement --tail=100 | grep -i "error\|pdf\|fail"
kubectl rollout restart deployment/rfd--doctor-statement--be--consumer-generate-statement-pdf--prod -n rfd--doctor-statement

OTP issues:

kubectl logs -f deployment/rfd--doctor-statement--be--consumer-send-otp--prod -n rfd--doctor-statement
kubectl rollout restart deployment/rfd--doctor-statement--be--consumer-send-otp--prod -n rfd--doctor-statement

Performance Metrics

Current Scale

• Main API: 1 replica (no HA)
• Consumers: 4 consumers at 1 replica each
• Worker: 1 batch publisher
• Total Active Pods: ~6 pods

Stability

• Namespace Age: ~2 years (mature)
• Recent Updates: 29 days ago
• All Single Replicas: No redundancy configured