Defeating Thundering Herds with Multi-Tier Caching

The Context: "The 3 AM Trigger"

It was December 2023. We had just rolled out a new dynamic pricing engine for a B2B supply chain client, "LogisCore" (name changed). The catalog metadata service was aggressively cached in Redis to spare the primary Oracle DB from millions of redundant read-only queries. It all worked beautifully until the Redis cluster underwent a scheduled master-slave failover. During that 2-second blip, the cache entries expired without being refreshed. When the network re-established, 30,000 parallel containerized microservices experienced a cache miss simultaneously. Instead of querying Redis, they all sent SELECT queries to the Oracle primary. The DB connection pool exhausted instantly, CPU spiked to 100%, and the database locked up completely. A classic thundering herd. Total baigan. We needed a shock-absorber.

The Problem Statement

Our constraints going forward:

• Resilience: Must survive a total cache eviction without crashing the database.
• Latency: P99 read latency must stay under 5ms.
• Complexity: Keep the implementation straightforward using existing Spring Boot capabilities.
• Staleness: Slight eventual consistency (up to 5 seconds stale) is acceptable for catalog metadata.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We shifted from a standard look-aside cache to a multi-tiered caching architecture (L1 + L2). When a request arrives, the Spring Boot application first checks an in-memory Caffeine cache (L1). If the key is present, it's served in sub-milliseconds without network I/O. If L1 misses, we query Redis (L2). If L2 misses, only then do we query the Oracle Database.

Concurrency & Locking

To fix the actual "thundering herd," we implemented a Cache Jitter and a distributed lock using Redisson. We added a $\pm$ 20% random variance to our Redis TTLs so they don't all expire at the exact same millisecond. More importantly, if L2 misses, the application must acquire a Redisson RLock for the specific catalog item before querying Oracle. The first thread to acquire the lock queries the database and populates Redis. The remaining 29,999 blocked threads wait 50ms, check Redis again, and instantly find the newly populated data without ever hitting the DB.

Failure Modes

If the Redis cluster undergoes failover, the L1 Caffeine cache absorbs 95% of the read load since we configured it to hold the hottest 10,000 SKUs. We also utilize Caffeine's refreshAfterWrite feature. By returning slightly stale L1 data while asynchronously refreshing it via a background virtual thread, user requests remain non-blocking even when the L2 cache goes temporarily dark.

The "Baigan" Take (Expert Insights)

Look, everyone throws @Cacheable on a method, hooks it to Redis, and thinks they've scaled. But a single massive cache is a single massive point of failure.

The real secret isn't just the multi-tier cache; it's abandoning synchronous blocking for cache refreshes. Once we moved to Spring WebFlux and Java 21, we handed off the cache repopulation to background virtual threads. The main client gets the slightly stale data instantly, and the DB doesn't get flooded. The only tradeoff here is managing L1 cache invalidation across distributed nodes, which we handle using a lightweight Kafka broadcast topic.

Performance Checklist

• Introduce Caffeine as an L1 cache to decouple read latency from the network layer.
• Use Redisson's distributed RLock to serialize DB queries upon cache-miss.
• Add random jitter (variance) to your cache TTLs to avoid synchronized expirations.
• Configure refreshAfterWrite instead of expireAfterWrite for hot, latency-sensitive data.