Handling Limits for High-Volume Systems: Rate Limiting & Load Shedding

The Context: "The DDoS of Success"

It was October 2024. A mega-merchant launch went live, and their automated ERP system began polling our balance API every 10ms for 5,000 different sub-accounts. In 60 seconds, our API latency jumped from 50ms to 5,000ms. The "Thundering Herd" didn't just slow us down; it exhausted the thread pool of our Edge Gateway, effectively taking down the entire platform for all other customers.

We didn't have a "Limit" problem; we had a "Fairness" problem. This article explores the architecture of Rate Limiting and Load Shedding in high-volume banking systems where mission-critical traffic must be protected from "noisy neighbors" and automated polling.

The Problem Statement

In a high-scale system (25k+ TPS burst), you cannot simply "Add more servers". You must protect existing resources from being overwhelmed.

Goal: Protect System Stability & Ensure Fairness.
Latency: The limit-checker must add < 2ms overhead.
Challenge: The "Distributed Rate Limit". Checking a limit across 100 API nodes requires a shared state (Redis), but a shared state can become its own bottleneck.

Visual Architecture: The Multi-Layer Throttle

The Core Logic: Algorithms of Control

1. The Token Bucket (The Classic)

We use the Token Bucket algorithm for most customer-facing rate limits. A merchant has a "Bucket" of $B$ tokens. Every request consumes one. Tokens refill at a rate of $R$ per second.

Advantage: Allows bursts (up to $B$) but ensures the average rate never exceeds $R$.
Redis Implementation: We use a Lua script to atomically check and decrement tokens to ensure the check is thread-safe across all API nodes.

2. Adaptive Load Shedding (The Emergency Brake)

Rate Limiting protects against a specific customer. Load Shedding protects the whole system. If our JVM CPU usage exceeds 85% or our P99 latency exceeds 1s, the "Adaptive Load Shedder" begins rejecting low-priority traffic (e.g., historical reports) while allowing high-priority traffic (e.g., payment authorizations).

3. Backpressure with Project Reactor (Spring WebFlux)

In a high-volume Java system, we avoid "Blocking threads". Using Spring WebFlux and RSocket, we implement backpressure. If the downstream DB is slow, the subscriber notifies the publisher to "Slow down", preventing a build-up of messages in memory that leads to an OutOfMemoryError.

The "Baigan" Take: Insights from the Trenches

Look, putting a rate limit on the "Public API" is easy. But baigan ka logic hai wo (that's nonsense) if your internal microservices can still DDoS each other.

Never use a 'Fixed Window' counter. If you say "100 requests per minute," and a customer sends 100 requests at 11:59:59 PM and another 100 at 12:00:01 AM, they just sent 200 requests in 2 seconds. This is the "Spiking" problem.

Rule: Use Sliding Window Logs or a Token Bucket. It's more CPU intensive but far more protective.
Rule: Always return X-RateLimit-Reset headers. Don't be that API that just says 429 and leaves the developer guessing when they can try again.

Performance & Implementation Checklist

Redis Pipelining: If your API node checks multiple limits (Global, Merchant, Method), pipeline the Redis commands to reduce RTT (Round Trip Time).
Local Cache Bypass: For extremely high volume (1M+ hits/sec), use a local "In-memory" cache to reject traffic before even hitting Redis for the first 90% of the limit.
Priority Queues: Use Kafka partitions to ensure "VIP Merchants" get their own dedicated processing threads, isolated from standard merchants.
Chaos Engineering: Run "Kill Redis" drills. Does your system fail open (dangerous) or fail closed (safe)? We prefer "Fail Open but Alert Hard" for banking, as some availability is better than zero.

Conclusion

Rate limiting is not about "Restricting Customers"; it is about Predictable Capacity. In a banking environment, an unregulated system is a liability. By implementing Token Buckets and Adaptive Load Shedding, you transform a fragile pipeline into a resilient infrastructure that can survive the world's most aggressive thundering herds.

Verified for Technical Nexus v1.9.0 | Part of the System Design Vault archive.