The Context: "The 3 AM Trigger"
It was Valentine's Day 2024. Our dating app client, "HeartConnect" (name changed), was experiencing their highest traffic spike of the decade. Their infrastructure relied on HTTP Long-Polling for chat notifications. Every 5 seconds, 2 million active iPhones pinged the backend asking, "Do I have a new message?" This equated to 400,000 HTTP requests per second hitting the load balancers, authenticating, opening an Oracle database connection, finding 0 new messages, and returning an empty 200 OK. The Oracle connection pool completely saturated, locking out actual real-time chat insertions. The database wasn't failing because of chats; it was failing because of the question, "Are there chats?" Total baigan design. We had to pivot to persistent WebSocket connections immediately to survive the evening.
The Problem Statement
Our real-time push constraints:
- Connection Scale: Maintain 2 million concurrent persistent TCP WebSocket connections.
- Statefulness: Because WebSockets are inherently stateful, a user might be connected to Pod A, but the user's match might send a message to Pod B. The message must instantly cross the boundary to Pod A.
- Latency: Push delivery under 100ms globally.
- Hardware Profile: Minimize the memory footprint per idle connection on the JVM.
Visual Architecture
The Core Logic / Process
Data Flow
We replaced the HTTP Long-Polling with persistent WebSockets. We deployed a specialized layer of Spring WebFlux WebSocket Gateway microservices. When User A connects, the gateway authenticates the JWT and registers a mapping in local memory: userId_123 -> WebSocketSession.
When User B sends a message to User A, the request hits the Chat Core Service. It saves the message to Oracle, and then publishes an event to a Redis Pub/Sub channel called notifications.
All 50 instances of the WebSocket Gateway are subscribed to this Redis channel. They receive the broadcast, but only the specific Gateway Pod that holds the active WebSocketSession for User A will actually push the payload over the wire. The other 49 pods quietly drop the message.
Concurrency & Locking
WebSockets using standard Tomcat holding a thread-per-connection would require 2 million Java threads, which would instantly OOM the JVM. By switching to Spring WebFlux, we utilize Project Reactor and Netty. The Netty event loop uses epoll/kqueue to handle tens of thousands of idle connections on a single thread. The only locking required is a lightweight ConcurrentHashMap within each WebFlux pod mapping userId to the FluxSink or session object.
Failure Modes
What happens if the Load Balancer cuts the persistent connection? The iOS client implements a backoff reconnect loop. What if a message is published to Redis exactly when the user's WebSocket is reconnecting? Redis Pub/Sub is "fire and forget"—it doesn't queue. To fix this, the iOS client, immediately upon successful reconnection, fires a lightweight REST call to fetch "Messages since timestamp $T$." This covers the transient gap gracefully.
The "Baigan" Take (Expert Insights)
Look, everyone knows you need Redis Pub/Sub to scale WebSockets. But the real landmine is the "Thundering Herd on Reconnect".
If your Kubernetes cluster decides to cycle a WebSocket pod out of the load balancer, 40,000 users will disconnect simultaneously. Their iOS devices will instantly say "Oh, connection lost, let's reconnect!" and all 40,000 will slam the remaining pods at the exact same millisecond. We brought down our own Identity Provider because the reconnection stampede caused 40k simultaneous JWT validations.
You must enforce randomized exponential backoff on the client-side reconnect. Do not let your frontend engineers just write setTimeout(reconnect, 1000). It must be waitTime = baseDelay * 2^retry + jitter. That jitter is the only thing standing between a pod restart and a cascading total-platform failure.
Performance Checklist
- Use Spring WebFlux and Netty to handle >50k connections per JVM using non-blocking I/O.
- Use Redis Pub/Sub to broadcast state changes to the WebSocket Gateway fleet.
- Implement a REST fallback/sync on connection to retrieve messages missed during micro-outages.
- MANDATORY: Code randomized jitter into frontend socket reconnection loops.