Skip to main content

spc--websocket-server

Overview​

  • Namespace: spc--websocket-server
  • Purpose: Sapoche WebSocket Server - PRODUCTION
  • Age: ~417 days (since December 2023)
  • Status: Active - Real-time WebSocket communication
  • Workloads: 2 deployments (both active)
  • Environment: PRODUCTION - Real-time messaging and notifications

Architecture​

WebSocket server for real-time communication using Laravel Reverb:

  • Main Application: REST API backend (1 replica)
  • Reverb Server: Laravel Reverb WebSocket server (1 replica)

Auto-Scaling Configuration​

No Auto-Scaling Configured:

  • No HorizontalPodAutoscalers (HPAs)
  • No KEDA scaled objects
  • Fixed 1 replica for both deployments

Workload Categories​

Main Application (1 deployment)​

NameReplicasStatusPurpose
spc--websocket-server--be--app1/1RunningWebSocket API backend

WebSocket Server (1 deployment)​

NameReplicasStatusPurpose
spc--websocket-server--be--reverb1/1RunningLaravel Reverb WebSocket server

Services​

NameTypeCluster IPPortsNodePortPurpose
spc--websocket-server--be--app--prodNodePort10.8.24.1548030173WebSocket API
spc--websocket-server--be--reverb--prodNodePort10.8.19.1798031443Reverb WebSocket server

Access & Management​

View all resources:​

kubectl get all -n spc--websocket-server

Check applications:​

# Main API
kubectl get pods -n spc--websocket-server | grep "app--prod"
kubectl logs -f deployment/spc--websocket-server--be--app--prod -n spc--websocket-server

# Reverb WebSocket server
kubectl get pods -n spc--websocket-server | grep "reverb"
kubectl logs -f deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server

Restart services:​

# Restart main API
kubectl rollout restart deployment/spc--websocket-server--be--app--prod -n spc--websocket-server

# Restart Reverb server
kubectl rollout restart deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server

# Restart all
kubectl rollout restart deployment --all -n spc--websocket-server

Access applications:​

# Port forward WebSocket API
kubectl port-forward -n spc--websocket-server service/spc--websocket-server--be--app--prod 8080:80

# Port forward Reverb WebSocket server
kubectl port-forward -n spc--websocket-server service/spc--websocket-server--be--reverb--prod 8081:80

Monitoring​

Resource usage:​

kubectl top pods -n spc--websocket-server --sort-by=memory
kubectl top pods -n spc--websocket-server --sort-by=cpu

Events:​

kubectl get events -n spc--websocket-server --sort-by='.lastTimestamp' | head -20

Data Flow​

Client WebSocket Connection
    ↓
Load Balancer / Ingress
    ↓
spc--websocket-server--be--reverb--prod (NodePort 31443)
    ↓
Laravel Reverb WebSocket Server (1 replica)
    ↓
Persistent WebSocket Connection
    ↓
Real-time message broadcasting
    ↓
API Backend → spc--websocket-server--be--app--prod (NodePort 30173)
    ↓
Message publishing and management

WebSocket Server Workflow​

1. WebSocket API Backend​

  • 1 replica (no HA)
  • WebSocket connection management API
  • Message publishing endpoints
  • Channel subscription management
  • Authentication for WebSocket connections
  • Connection state management

2. Laravel Reverb WebSocket Server​

  • 1 replica (no HA)
  • Laravel Reverb - Modern WebSocket server for Laravel
  • Real-time message broadcasting
  • Channel management (public, private, presence)
  • WebSocket connection handling
  • Pub/sub messaging
  • Event broadcasting

Production Considerations​

High Availability​

Single Points of Failure:

  • Main API: 1 replica (no HA)
  • Reverb server: 1 replica (no HA, critical for real-time)
  • WebSocket connections will drop if pod restarts
  • No redundancy for real-time messaging

Recommendations​

  1. URGENT: WebSocket Server Resilience:

    • Reverb server: 1 replica ( critical issue)
    • Increase to 2+ replicas for HA
    • WebSocket servers need redundancy for connection failover
    • Pod restart = all active connections dropped
    • Consider sticky sessions or connection draining

  2. API Backend Resilience:

    • Main API: 1 replica (consider 2+ for HA)
    • Less critical than Reverb but still important
    • Handles message publishing

  3. Consider Auto-Scaling:

    • Reverb: Add HPA based on active connections (2-5 replicas)
    • Main API: Add HPA based on CPU/Memory (1-3 replicas)
    • WebSocket connections can spike during peak usage

  4. WebSocket-Specific Considerations:

    • Connection draining on pod updates
    • Sticky sessions for connection affinity
    • Redis for broadcasting across multiple Reverb instances
    • Connection reconnection logic in clients
    • Health checks that don't break WebSocket connections

  5. Recent Activity:

    • Main API: Updated 254 days ago (very stable)
    • Reverb: Updated 206 days ago (very stable)
    • Mature and stable but lacks HA

  6. Monitoring Priorities:

    • Active WebSocket connection count
    • Connection drop rate
    • Message delivery latency
    • Broadcasting success rate
    • Pod restart impact on connections

Troubleshooting​

Main API issues:​

# Check API pod
kubectl get pods -n spc--websocket-server | grep "app--prod"

# Check logs
kubectl logs -f deployment/spc--websocket-server--be--app--prod -n spc--websocket-server

# Test API endpoint
kubectl port-forward -n spc--websocket-server service/spc--websocket-server--be--app--prod 8080:80
curl http://localhost:8080/health

Reverb WebSocket server issues:​

# Check Reverb pod
kubectl get pods -n spc--websocket-server | grep "reverb"

# Check Reverb logs
kubectl logs -f deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server

# Check for WebSocket errors
kubectl logs deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server --tail=100 | grep -i "error\|websocket\|connection\|fail"

# Restart Reverb ( will drop all connections)
kubectl rollout restart deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server

WebSocket connection issues:​

# Check active connections
kubectl exec -it deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server -- netstat -an | grep ESTABLISHED | wc -l

# Test WebSocket connection
kubectl port-forward -n spc--websocket-server service/spc--websocket-server--be--reverb--prod 8081:80
# Use WebSocket client tool to connect to ws://localhost:8081

# Check Reverb metrics
kubectl logs deployment/spc--websocket-server--be--reverb--prod -n spc--websocket-server | grep -i "connection\|subscribe\|broadcast"

Performance Metrics​

Current Scale​

  • Main API: 1 replica (no HA)
  • Reverb Server: 1 replica ( no HA for WebSocket)
  • Total Active Pods: 2 pods

Stability​

  • Namespace Age: ~417 days (~1 year 52 days, mature)
  • Main API: 254 days since update (very stable)
  • Reverb: 206 days since update (very stable)
  • Critical Issue: No redundancy for real-time WebSocket connections

Architecture Notes​

  • Laravel Reverb: Modern WebSocket server for Laravel applications
  • Real-time Broadcasting: Event-driven pub/sub messaging
  • Channel Types: Public, private, and presence channels
  • Persistent Connections: Long-lived WebSocket connections
  • State Management: Connection state and subscriptions
  • CRITICAL: Single replica = all connections drop on restart