spc--pdf-generate--be

Overview

Namespace: spc--pdf-generate--be
Purpose: Sapoche PDF Generation Service - PRODUCTION
Age: 724 days (~2 years, since November 2023)
Status: Active - Medical document PDF generation
Workloads: 10 deployments (all active)
Environment: PRODUCTION - Critical document generation service

Architecture

Dedicated PDF generation service handling medical reports, test results, imaging reports, and billing documents:

Main Application: REST API backend (1 replica)
Event Consumers: Process PDF generation requests (8 deployments with KEDA autoscaling)
Worker: Batch job publisher (1 deployment)
VIP Tier Support: Dedicated consumers for SVIP and VIP customers

Auto-Scaling Configuration

HorizontalPodAutoscalers (7 KEDA HPAs)

HPA Name	Target	Min	Max	Current	Queue Depth	Status
keda-hpa-consumer-general	General PDFs	3	20	3	3/10	Active
keda-hpa-consumer-order-billing	Billing PDFs	1	5	1	0/10	⏸️ Idle
keda-hpa-consumer-svip-1	SVIP Tier 1	1	15	1	0/10	⏸️ Idle
keda-hpa-consumer-svip-2	SVIP Tier 2	1	15	1	0/10	⏸️ Idle
keda-hpa-consumer-svip-imaging	SVIP Imaging	1	3	1	0/20	⏸️ Idle
keda-hpa-consumer-svip-regen	SVIP Regeneration	10	10	10	200m/1	🔒 Locked
keda-hpa-consumer-vip-1	VIP Tier 1	1	3	1	0/10	⏸️ Idle

Scaling Summary:

Heavy KEDA usage: 7 of 10 deployments use queue-based autoscaling
consumer-general: Currently processing (3/10 queue depth)
consumer-svip-regen: Locked at 10 replicas (min=max=10)
All other consumers at minimum replicas (idle)

Workload Categories

Main Application (1 deployment)

Name	Replicas	Status	Purpose
spc--pdf-generate--be--app--prod	1/1	Running	Main PDF generation API

PDF Generation Consumers (8 deployments)

Process PDF generation requests from message queues:

Name	Replicas	Status	KEDA	Purpose
consumer-general	3/6	Running + HPA	3-20	General PDF generation (active)
consumer-order-billing	1/1	Running + HPA	1-5	Billing document PDFs
consumer-order-billing-high-priority	1/1	Running	None	High-priority billing (newer - 114 days)
consumer-svip-1	1/1	Running + HPA	1-15	SVIP Tier 1 customer PDFs
consumer-svip-2	1/1	Running + HPA	1-15	SVIP Tier 2 customer PDFs
consumer-svip-imaging	1/1	Running + HPA	1-3	SVIP imaging report PDFs
consumer-svip-regen	10/10	Running + HPA	10-10	SVIP PDF regeneration (locked at 10)
consumer-vip-1	1/1	Running + HPA	1-3	VIP Tier 1 customer PDFs

Worker (1 deployment)

Name	Replicas	Status	Purpose
wrk--batch-publisher	1/1	Running	Batch job publisher

Services

Name	Type	Cluster IP	Ports	NodePort	Purpose
spc--pdf-generate--be--app--prod	NodePort	10.8.18.64	80	31754	Main PDF generation API

Access & Management

View all resources:

kubectl get all -n spc--pdf-generate--be

Check main application:

# View app pod
kubectl get pods -n spc--pdf-generate--be | grep "app--prod"

# View logs
kubectl logs -f deployment/spc--pdf-generate--be--app--prod -n spc--pdf-generate--be

Check PDF generation consumers:

# All consumers
kubectl get pods -n spc--pdf-generate--be | grep consumer

# KEDA-scaled consumers
kubectl get hpa -n spc--pdf-generate--be

# Active consumer (general)
kubectl logs -f deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# High-volume SVIP regen (10 replicas)
kubectl logs -f deployment/spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be

Check KEDA scaling:

# All KEDA scaled objects
kubectl get scaledobjects -n spc--pdf-generate--be

# All HPAs
kubectl get hpa -n spc--pdf-generate--be

# Detailed HPA status for general consumer
kubectl describe hpa keda-hpa-spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# SVIP regen (locked at 10)
kubectl describe hpa keda-hpa-spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be

Restart services:

# Restart main app
kubectl rollout restart deployment/spc--pdf-generate--be--app--prod -n spc--pdf-generate--be

# Restart specific consumer
kubectl rollout restart deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# Restart all consumers
kubectl get deployments -n spc--pdf-generate--be | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n spc--pdf-generate--be

Monitoring

Resource usage:

kubectl top pods -n spc--pdf-generate--be --sort-by=memory
kubectl top pods -n spc--pdf-generate--be --sort-by=cpu

KEDA metrics:

# All HPAs with current metrics
kubectl get hpa -n spc--pdf-generate--be

# Watch scaling in real-time
kubectl get hpa -n spc--pdf-generate--be -w

# Detailed scaled object status
kubectl get scaledobjects -n spc--pdf-generate--be -o wide

Deployment status:

kubectl get deployments -n spc--pdf-generate--be

Events:

kubectl get events -n spc--pdf-generate--be --sort-by='.lastTimestamp' | head -20

Queue depth monitoring:

# Check KEDA metrics for all consumers
for hpa in $(kubectl get hpa -n spc--pdf-generate--be -o name); do
  echo "=== $hpa ==="
  kubectl describe $hpa -n spc--pdf-generate--be | grep -A 5 "Metrics:"
done

Data Flow

PDF Generation Request
    ↓
spc--pdf-generate--be--app--prod (NodePort 31754)
    ↓
Message Queue (Kafka/Redpanda)
    ↓
    ├─ General Queue → consumer-general (KEDA 3-20, currently 3 replicas)
    ├─ Billing Queue → consumer-order-billing (KEDA 1-5)
    ├─ High-Priority Billing → consumer-order-billing-high-priority
    ├─ SVIP Tier 1 → consumer-svip-1 (KEDA 1-15)
    ├─ SVIP Tier 2 → consumer-svip-2 (KEDA 1-15)
    ├─ SVIP Imaging → consumer-svip-imaging (KEDA 1-3)
    ├─ SVIP Regen → consumer-svip-regen (10 replicas, locked)
    └─ VIP Tier 1 → consumer-vip-1 (KEDA 1-3)
    ↓
PDF Generation Processing
    ↓
Object Storage (S3/GCS)
    ↓
PDF Available for Download

PDF Generation Workflow

1. General PDFs (consumer-general)

Test results, examination reports, general medical documents
KEDA 3-20 replicas (currently 3, processing 3/10 queue depth)
Handles majority of PDF generation load
Auto-scales based on queue depth

2. Billing Documents

consumer-order-billing: Standard billing PDFs (KEDA 1-5)
consumer-order-billing-high-priority: Priority billing (no KEDA, fixed 1 replica)
Invoices, receipts, payment records

3. VIP Customer PDFs (consumer-vip-1)

Dedicated consumer for VIP tier 1 customers
KEDA 1-3 replicas (currently 1, idle)
Priority processing for VIP customers

4. SVIP Customer PDFs (Multiple Consumers)

consumer-svip-1: SVIP Tier 1 (KEDA 1-15, currently 1)
consumer-svip-2: SVIP Tier 2 (KEDA 1-15, currently 1)
consumer-svip-imaging: SVIP imaging reports (KEDA 1-3, currently 1)
consumer-svip-regen: PDF regeneration (🔒 locked at 10 replicas)
Highest priority processing for SVIP customers

5. PDF Regeneration (consumer-svip-regen)

Always runs at 10 replicas (min=max=10)
Handles PDF regeneration requests
High volume workload requiring dedicated capacity
Cannot scale up or down (locked configuration)

Production Considerations

High Availability

Well Configured:

Heavy KEDA usage (7 of 10 deployments)
consumer-general: 3 replicas baseline (can scale to 20)
consumer-svip-regen: 10 replicas dedicated capacity
Queue-based autoscaling for demand-driven scaling

x Single Points of Failure:

Main API: 1 replica (no HPA)
Most consumers: 1 replica when idle
Batch publisher: 1 replica
consumer-order-billing-high-priority: 1 replica (no KEDA)

KEDA Scaling Configuration

Consumer	Min	Max	Current	Queue Threshold	Notes
general	3	20	3	10 messages	Baseline 3 replicas
order-billing	1	5	1	10 messages	Standard scaling
svip-1	1	15	1	10 messages	SVIP tier 1
svip-2	1	15	1	10 messages	SVIP tier 2
svip-imaging	1	3	20 messages	Higher threshold
svip-regen	10	10	10	1 message	🔒 Locked at 10
vip-1	1	3	1	10 messages	VIP tier 1

Recommendations

Main API Scaling:
- Currently at 1 replica (no HPA)
- Consider adding HPA for API availability
- Or increase to 2 replicas for baseline HA
Consumer Resilience:
- consumer-general: 3 replicas baseline (good)
- consumer-svip-regen: 10 replicas dedicated (good)
- Consider baseline of 2 replicas for critical consumers:
  - consumer-order-billing
  - consumer-svip-1
  - consumer-svip-2
SVIP Regen Configuration:
- Locked at 10 replicas (min=max=10)
- Review if this is intentional or over-provisioned
- If regeneration is rare, consider allowing scale-down to save resources
- If regeneration is constant, configuration makes sense
High-Priority Billing:
- consumer-order-billing-high-priority: 1 replica, no KEDA
- Newer deployment (114 days)
- Consider adding KEDA for consistency
- Or increase to 2 replicas for HA
Queue Threshold Review:
- Most consumers: 10 messages per replica
- svip-imaging: 20 messages per replica (higher)
- svip-regen: 1 message per replica (very sensitive)
- Review if thresholds match actual processing capacity
Consumer Dependency:
- 8 consumers depend on message queue availability
- Ensure queue (Kafka/Redpanda) is highly available
- Monitor queue lag and consumer group health
Monitoring Priorities:
- PDF generation success rates
- Queue depths and consumer lag
- consumer-general active processing (currently 3/10 depth)
- SVIP regen utilization (10 replicas always running)
- Generation time SLAs
- Failed PDF generation alerts

Troubleshooting

Main API issues:

# Check API pod
kubectl get pods -n spc--pdf-generate--be | grep "app--prod"

# Check logs
kubectl logs -f deployment/spc--pdf-generate--be--app--prod -n spc--pdf-generate--be --tail=100

# Test API endpoint
kubectl port-forward -n spc--pdf-generate--be service/spc--pdf-generate--be--app--prod 8080:80
# Access http://localhost:8080

Consumer not processing PDFs:

# Check consumer status
kubectl get pods -n spc--pdf-generate--be | grep consumer-general

# Check KEDA scaler
kubectl describe scaledobject spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# Check consumer logs
kubectl logs -f deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# Check for errors
kubectl logs deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be --tail=100 | grep -i error

# Restart consumer
kubectl rollout restart deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

KEDA not scaling:

# Check all HPAs
kubectl get hpa -n spc--pdf-generate--be

# Check specific HPA
kubectl describe hpa keda-hpa-spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

# Check KEDA operator
kubectl get pods -n keda
kubectl logs -n keda deployment/keda-operator | grep spc--pdf-generate

# Check scaled object status
kubectl get scaledobject -n spc--pdf-generate--be -o yaml

High queue lag:

# Check all KEDA HPAs for queue depth
kubectl get hpa -n spc--pdf-generate--be

# Check consumer processing
for consumer in general order-billing svip-1 svip-2 svip-imaging svip-regen vip-1; do
  echo "=== consumer-$consumer ==="
  kubectl logs deployment/spc--pdf-generate--be--consumer-$consumer--prod -n spc--pdf-generate--be --tail=10
done

# Manual scale if needed (KEDA will override based on queue)
kubectl scale deployment spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be --replicas=10

SVIP regen over-provisioned:

# Check SVIP regen utilization
kubectl top pods -n spc--pdf-generate--be | grep svip-regen

# Check logs for activity
kubectl logs deployment/spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be --tail=100

# Check scaled object configuration
kubectl get scaledobject spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be -o yaml

# If over-provisioned, update KEDA scaled object to allow scale-down
# kubectl edit scaledobject spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be

PDF generation failures:

# Check consumer logs for failures
kubectl logs deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be | grep -i "fail\|error\|exception"

# Check batch publisher
kubectl logs -f deployment/spc--pdf-generate--be--wrk--batch-publisher--prod -n spc--pdf-generate--be

# Check object storage connectivity
kubectl exec -it deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be -- env | grep -i "s3\|gcs\|storage"

# Restart problematic consumer
kubectl rollout restart deployment/spc--pdf-generate--be--consumer-general--prod -n spc--pdf-generate--be

Billing PDF delays:

# Check billing consumers
kubectl get pods -n spc--pdf-generate--be | grep order-billing

# Check standard billing
kubectl logs -f deployment/spc--pdf-generate--be--consumer-order-billing--prod -n spc--pdf-generate--be

# Check high-priority billing
kubectl logs -f deployment/spc--pdf-generate--be--consumer-order-billing-high-priority--prod -n spc--pdf-generate--be

# Check queue depth
kubectl describe hpa keda-hpa-spc--pdf-generate--be--consumer-order-billing--prod -n spc--pdf-generate--be

VIP/SVIP processing issues:

# Check all VIP/SVIP consumers
kubectl get pods -n spc--pdf-generate--be | grep -E "svip|vip"

# Check SVIP tier 1
kubectl logs -f deployment/spc--pdf-generate--be--consumer-svip-1--prod -n spc--pdf-generate--be

# Check SVIP tier 2
kubectl logs -f deployment/spc--pdf-generate--be--consumer-svip-2--prod -n spc--pdf-generate--be

# Check SVIP imaging
kubectl logs -f deployment/spc--pdf-generate--be--consumer-svip-imaging--prod -n spc--pdf-generate--be

# Check SVIP regen (10 replicas)
kubectl logs deployment/spc--pdf-generate--be--consumer-svip-regen--prod -n spc--pdf-generate--be --tail=50

# Check VIP tier 1
kubectl logs -f deployment/spc--pdf-generate--be--consumer-vip-1--prod -n spc--pdf-generate--be

Performance Metrics

Current Scale (Production Load)

Main API: 1 replica (no scaling)
Active Consumers:
- consumer-general: 3 replicas (processing, 3/10 queue depth)
- consumer-svip-regen: 10 replicas (locked, always running)
Idle Consumers: 6 consumers at 1 replica each
Total Active Pods: ~22 pods

Scaling Behavior

KEDA-based: 7 of 10 deployments use queue-based autoscaling
consumer-general: Baseline 3, scales to 20 based on queue
consumer-svip-regen: Locked at 10 (cannot scale)
Queue thresholds: 10-20 messages per replica (except svip-regen at 1)

Resource Allocation

Baseline: ~16 pods (3 general + 10 svip-regen + 1 each for others + 1 API + 1 worker)
Peak capacity: Up to 60+ pods (if all consumers scale to max)
Current utilization: Low (most consumers idle, only general processing)

Integration Points

External Systems

Medical Records System:
- Source of patient data
- Test results, imaging reports
- Examination data
Billing System:
- Order billing documents
- High-priority billing PDFs
- Invoices and receipts
Object Storage:
- PDF storage (S3/GCS)
- Patient document retrieval
- Long-term archival
Patient Portal:
- PDF download requests
- Document availability notifications

Internal Systems

Test Result Services: Source data for test result PDFs
Imaging Services: Source data for imaging report PDFs
Billing Services: Source data for billing documents
Notification Services: PDF availability alerts

Important Notes

x PRODUCTION ENVIRONMENT:

This is a CRITICAL DOCUMENT GENERATION service
Downtime impacts patient access to medical records
Changes must be tested in staging first
Monitor PDF generation success rates carefully
Have immediate rollback plan ready

x SVIP Regen Over-Provisioning:

consumer-svip-regen locked at 10 replicas (min=max=10)
Always running, cannot scale down
Review if intentional or can be optimized
Significant resource allocation

x Heavy KEDA Usage:

7 of 10 deployments use KEDA
Dependency on KEDA operator availability
Monitor KEDA operator health
Queue availability critical

Compliance: Medical documents subject to regulatory requirements (HIPAA, data retention policies)

System Purpose

The PDF generation service provides:

General PDFs: Test results, medical reports (consumer-general)
Billing Documents: Invoices, receipts, billing statements
VIP Services: Priority PDF generation for VIP customers
SVIP Services: Highest priority for SVIP customers with dedicated resources
PDF Regeneration: Re-generation of existing PDFs (dedicated 10 replicas)
Imaging Reports: Medical imaging PDF reports

Key Role: Centralized, scalable PDF generation platform for all medical documentation with tiered service levels (General, VIP, SVIP) and dedicated resources for high-volume regeneration tasks.

Overview​

Architecture​

Auto-Scaling Configuration​

HorizontalPodAutoscalers (7 KEDA HPAs)​

Workload Categories​

Main Application (1 deployment)​

PDF Generation Consumers (8 deployments)​

Worker (1 deployment)​

Services​

Access & Management​

View all resources:​

Check main application:​

Check PDF generation consumers:​

Check KEDA scaling:​

Restart services:​

Monitoring​

Resource usage:​

KEDA metrics:​

Deployment status:​

Events:​

Queue depth monitoring:​

Data Flow​

PDF Generation Workflow​

1. General PDFs (consumer-general)​

2. Billing Documents​

3. VIP Customer PDFs (consumer-vip-1)​

4. SVIP Customer PDFs (Multiple Consumers)​

5. PDF Regeneration (consumer-svip-regen)​

Production Considerations​

High Availability​

KEDA Scaling Configuration​

Recommendations​

Troubleshooting​

Main API issues:​

Consumer not processing PDFs:​

KEDA not scaling:​

High queue lag:​

SVIP regen over-provisioned:​

PDF generation failures:​

Billing PDF delays:​

VIP/SVIP processing issues:​

Performance Metrics​

Current Scale (Production Load)​

Scaling Behavior​

Resource Allocation​

Integration Points​

External Systems​

Internal Systems​

Important Notes​

System Purpose​