Request Collapsing at the Edge with Spring WebFlux

The Context: "The 3 AM Trigger"

It was February 2024. A famous influencer linked directly to a specific product page on our e-commerce platform, "BazaarPrime" (name changed). At exactly 3:00 AM, 150,000 users clicked the link within a 10-second window. All 150,000 HTTP GET requests hit our Spring Cloud Gateway instances. The gateway perfectly routed all 150,000 requests to the Catalog Service. The Catalog Service dutifully executed 150,000 identical SELECT queries against the primary Oracle RAC database to fetch the exact same product ID. The database CPU pegged at 100%, and the connection pool exhausted instantly. We tried to deploy Redis caching, but during the 50ms it takes to populate the cache on the very first miss (the "cache stampede"), 10,000 requests still slipped through to the database. Total baigan scaling. We didn't just need a cache; we needed Request Collapsing.

The Problem Statement

Our constraints for the Edge architecture:

• Database Protection: Prevent "Cache Stampedes". Only exactly one query should hit the database for a specific product ID, no matter how many users request it simultaneously.
• Latency: Wait times for concurrent users should not exceed the database latency (e.g., if the DB takes 40ms, all 10,000 users get their response in exactly 40ms).
• Tooling: Implement this entirely within the Spring WebFlux (Project Reactor) framework without deploying new infrastructure proxy layers.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We implemented Request Collapsing using the reactive primitives natively available in Project Reactor, specifically utilizing .cache(Duration).

When the first request for ProductID=123 hits the Spring Cloud Gateway, the gateway initiates the reactive chain to fetch the data. The .cache() operator intercepts the Mono<ProductPayload>. Over the next 40 milliseconds (before the Oracle DB returns the data), 9,999 more requests arrive for ProductID=123. The .cache() operator recognizes that there is an "In-Flight" Mono for that specific key. Instead of executing 9,999 new database calls, it simply subscribes all 9,999 new requests to the original in-flight Mono.

When Oracle finally returns the payload, the reactive stream emits the payload once, and Project Reactor automatically fans out the bytes to all 10,000 waiting HTTP connections simultaneously.

Concurrency & Locking

This is where Java 21 Virtual Threads and WebFlux shine. If we used standard Tomcat threads, holding 10,000 HTTP connections open waiting for 40ms would require 10,000 OS threads, crushing the JVM via Context Switching overhead. Because WebFlux uses Netty's non-blocking Event Loop, 10,000 connections cost virtually zero memory or CPU while parked. There are no explicit Java ReentrantLocks or synchronized blocks used for the collapsing; it is entirely handled by the thread-safe state machine inside Reactor's Mono.cache() implementation.

Failure Modes

What happens if the single Oracle database query fails (e.g., a SQL timeout)? Because 10,000 users are subscribed to that single Mono, the onError signal is propagated to all 10,000 users simultaneously. To mitigate this catastrophic fan-out of errors, we chain .onErrorResume() after the cache layer. If the DB fails, we instantly route the 10,000 users to a stale Redis fallback, ensuring they still get a 200 OK.

The "Baigan" Take (Expert Insights)

Look, anyone can add Redis caching, but Redis doesn't stop the 50ms window where the cache is empty.

The baigan truth is that Request Collapsing requires you to manage memory aggressively. If you collapse routing keys poorly (e.g., collapsing by userId instead of productId), your JVM heap will fill up with millions of Mono objects. We had to use an LRU Cache (like Caffeine) strictly as a front-end to hold the Mono references, ensuring that we only collapse identical requests that arrive within the exact same temporal sliver window. We configured Caffeine to evict the Mono explicitly 500ms after completion.

Performance Checklist

• Adopt Spring WebFlux natively to handle parked connections without catastrophic OS Thread Context Switching.
• Use Mono.cache(Duration) bound to a specific routing key to trap concurrent requests.
• Employ Caffeine Cache to hold the in-flight Mono references to prevent Heap exhaustion.
• Ensure your onError logic is resilient enough to handle a single DB failure affecting 10,000 concurrent subscribers.