Skip to main content

spc--audit

Overview​

  • Namespace: spc--audit
  • Purpose: Sapoche Audit System Backend - PRODUCTION
  • Age: ~220 days (since March 2024)
  • Status: Active - Audit logging and compliance tracking
  • Workloads: 6 deployments (all active)
  • Environment: PRODUCTION - Audit trail and compliance

Architecture​

Audit system tracking LIS, orders, and general audit events:

  • Main Application: REST API backend (1 replica)
  • Event Consumers: LIS test results, LIS VID status, orders, general audit, VID audit (5 deployments)
  • General Audit Consumer: 3 replicas - Good HA for critical audit logs

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed replica counts

Workload Categories​

Main Application (1 deployment)​

NameReplicasStatusPurpose
spc--audit--be--app--prod1/1RunningMain audit API

Event Consumers (5 deployments)​

NameReplicasStatusPurpose
consumer-lis-test-result1/1RunningLIS test result audit
consumer-lis-vid-status1/1RunningLIS VID status audit
consumer-order1/1RunningOrder audit logging
consumer-spc-audit-general3/3RunningGeneral audit ( Good HA)
consumer-spc-audit-vid1/1RunningVID audit logging

Services​

NameTypeCluster IPPortsNodePortPurpose
spc--audit--be--app--prodNodePort10.8.17.1928030105Main audit API

Access & Management​

View all resources:​

kubectl get all -n spc--audit

Check main application:​

# View app pod
kubectl get pods -n spc--audit | grep "app--prod"

# View logs
kubectl logs -f deployment/spc--audit--be--app--prod -n spc--audit

Check consumers:​

# LIS test result audit
kubectl logs -f deployment/spc--audit--be--consumer-lis-test-result--prod -n spc--audit

# LIS VID status audit
kubectl logs -f deployment/spc--audit--be--consumer-lis-vid-status--prod -n spc--audit

# Order audit
kubectl logs -f deployment/spc--audit--be--consumer-order--prod -n spc--audit

# General audit (3 replicas)
kubectl logs -f deployment/spc--audit--be--consumer-spc-audit-general--prod -n spc--audit

# VID audit
kubectl logs -f deployment/spc--audit--be--consumer-spc-audit-vid--prod -n spc--audit

Restart services:​

# Restart main app
kubectl rollout restart deployment/spc--audit--be--app--prod -n spc--audit

# Restart all consumers
kubectl get deployments -n spc--audit | grep consumer | awk '{print $1}' | xargs -I {} kubectl rollout restart deployment/{} -n spc--audit

# Restart general audit consumer (3 replicas)
kubectl rollout restart deployment/spc--audit--be--consumer-spc-audit-general--prod -n spc--audit

Monitoring​

Resource usage:​

kubectl top pods -n spc--audit --sort-by=memory
kubectl top pods -n spc--audit --sort-by=cpu

Events:​

kubectl get events -n spc--audit --sort-by='.lastTimestamp' | head -20

Data Flow​

Audit Event Trigger
    ↓
spc--audit--be--app--prod (NodePort 30105)
    ↓
Main Audit API (1 replica)
    ↓
Database (external)
    ↓
Events Published to Message Queue
    ├─ LIS Test Results → consumer-lis-test-result
    ├─ LIS VID Status → consumer-lis-vid-status
    ├─ Order Events → consumer-order
    ├─ General Audit → consumer-spc-audit-general (3 replicas)
    └─ VID Audit → consumer-spc-audit-vid
    ↓
Audit Trail & Compliance Logs

Audit System Workflow​

1. Audit API​

  • 1 replica (no HA)
  • Audit event collection
  • Query audit logs
  • Compliance reporting
  • Audit trail API

2. LIS Test Result Audit Consumer​

  • consumer-lis-test-result tracks LIS test result events
  • Test result access logging
  • Laboratory information audit
  • Result modification tracking

3. LIS VID Status Audit Consumer​

  • consumer-lis-vid-status tracks VID (Venipuncture ID) status
  • Sample collection audit
  • VID status changes
  • Phlebotomist activity logging

4. Order Audit Consumer​

  • consumer-order logs order events
  • Order creation and updates
  • Order status changes
  • Order cancellations
  • Order modifications

5. General Audit Consumer (Critical - 3 Replicas)​

  • consumer-spc-audit-general handles general audit events - 3 replicas for HA
  • System-wide audit logging
  • User activity tracking
  • Compliance events
  • Security audit logs
  • Most critical audit consumer

6. VID Audit Consumer​

  • consumer-spc-audit-vid tracks VID-specific audit
  • Sample tracking
  • Chain of custody
  • VID assignment and lifecycle

Production Considerations​

High Availability​

Good General Audit Configuration:

  • General audit consumer: 3 replicas (most critical audit logs)
  • Ensures audit compliance even with failures

Single Points of Failure:

  • Main API: 1 replica (no HA)
  • LIS test result consumer: 1 replica
  • LIS VID status consumer: 1 replica
  • Order consumer: 1 replica
  • VID audit consumer: 1 replica

Recommendations​

  1. Main API Resilience:

    • Main API: 1 replica (consider 2+ for HA)
    • Audit API critical for compliance
    • Query access important for reporting

  2. Consumer Resilience:

    • consumer-spc-audit-general: 3 replicas (excellent for critical audit)
    • consumer-order: 1 replica (consider 2+ for order audit reliability)
    • consumer-lis-test-result: 1 replica (consider 2 for LIS audit)
    • consumer-lis-vid-status: 1 replica (consider 2 for VID tracking)
    • consumer-spc-audit-vid: 1 replica (consider 2 for VID audit)

  3. Consider Auto-Scaling:

    • Main API: Add HPA (1-3 replicas)
    • General audit consumer: Consider KEDA for queue-based scaling
    • Order consumer: Consider KEDA for order event spikes

  4. Recent Activity:

    • Main API: Updated 148 days ago (stable)
    • General audit: Updated 148 days ago (stable)
    • VID audit: Updated 18 days ago (active)
    • LIS consumers: Updated 82 days ago (stable)

  5. Monitoring Priorities:

    • Audit event processing lag
    • Consumer lag for all audit streams
    • Audit log completeness
    • Compliance reporting accuracy
    • Storage capacity for audit logs

Troubleshooting​

Main API issues:​

# Check audit API pod
kubectl get pods -n spc--audit | grep "app--prod"

# Check logs
kubectl logs -f deployment/spc--audit--be--app--prod -n spc--audit

# Test API endpoint
kubectl port-forward -n spc--audit service/spc--audit--be--app--prod 8080:80

General audit consumer issues:​

# Check general audit consumer (3 replicas)
kubectl get pods -n spc--audit | grep "audit-general"

# Check logs from all replicas
kubectl logs -f deployment/spc--audit--be--consumer-spc-audit-general--prod -n spc--audit

# Check for audit errors
kubectl logs deployment/spc--audit--be--consumer-spc-audit-general--prod -n spc--audit --tail=100 | grep -i "error\|audit\|fail"

# Restart consumer
kubectl rollout restart deployment/spc--audit--be--consumer-spc-audit-general--prod -n spc--audit

LIS audit issues:​

# Check LIS test result consumer
kubectl logs -f deployment/spc--audit--be--consumer-lis-test-result--prod -n spc--audit

# Check LIS VID status consumer
kubectl logs -f deployment/spc--audit--be--consumer-lis-vid-status--prod -n spc--audit

# Check for LIS audit errors
kubectl logs deployment/spc--audit--be--consumer-lis-test-result--prod -n spc--audit --tail=100 | grep -i "error\|lis\|fail"

Order audit issues:​

# Check order audit consumer
kubectl logs -f deployment/spc--audit--be--consumer-order--prod -n spc--audit

# Check for order audit errors
kubectl logs deployment/spc--audit--be--consumer-order--prod -n spc--audit --tail=100 | grep -i "error\|order\|fail"

# Restart consumer
kubectl rollout restart deployment/spc--audit--be--consumer-order--prod -n spc--audit

VID audit issues:​

# Check VID audit consumer
kubectl logs -f deployment/spc--audit--be--consumer-spc-audit-vid--prod -n spc--audit

# Check for VID errors
kubectl logs deployment/spc--audit--be--consumer-spc-audit-vid--prod -n spc--audit --tail=100 | grep -i "error\|vid\|fail"

Performance Metrics​

Current Scale​

  • Main API: 1 replica (no HA)
  • General Audit Consumer: 3 replicas ( good HA for critical audit)
  • Other Consumers: 4 consumers at 1 replica each
  • Total Active Pods: ~8 pods

Stability​

  • Namespace Age: ~220 days (~7 months)
  • Main API: 148 days since update (stable)
  • General Audit: 148 days since update (stable)
  • VID Audit: 18 days since update (active)
  • Critical Component: General audit consumer at 3 replicas