Cell-Based Distributed Rate Limiting Across Multi-Cluster Kubernetes

The Context: "The Centralized Redis Cascade"

At 11:45 PM on July 31, 2026, a sudden burst of automated API scraping targeted our catalog service across three Kubernetes production clusters. Over 180 Envoy proxy sidecars concurrently bombarded our centralized 6-node Redis cluster with token-bucket evaluation commands (EVALSHA).

Within 90 seconds, the central Redis cluster's primary nodes spiked to 100% single-threaded CPU utilization. Latencies skyrocketed from 0.8ms to 4,200ms.

Because our API gateways were synchronously waiting on the rate-limiter response before forwarding client traffic, the slow rate-limiter became the single point of global failure. Every API endpoint in the platform ground to a dead halt. The service was brought down by the very component designed to protect it. Total baigan ka bottleneck (the ultimate pointless self-inflicted wound).

The Problem Statement

Designing a high-throughput, horizontally scalable rate limiting system capable of handling multi-tenant tiering across geographically distributed Kubernetes clusters without a centralized choke point:

• Target Ingress: 350,000 Requests Per Second (RPS) peak aggregate traffic.
• Decision Overhead: Rate check decision must execute in under 1.5ms (P99).
• Blast Radius: A complete failure of rate-limiting components must degrade gracefully to local token limits without interrupting normal user traffic.
• Multi-Tenancy: Support hierarchical tier quotas (e.g. Bronze: 500 RPS, Enterprise: 50,000 RPS).

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

1. Cellular Isolation and Quota Slicing

Instead of dispatching an RPC to a central store for every individual HTTP request, the global quota for each tenant is pre-allocated and divided among autonomous Cells (individual Kubernetes clusters or availability zones).

If an Enterprise client has an allowed global quota of 60,000 RPS:

• Cell 1 (EU-West) receives an initial slice of 30,000 RPS.
• Cell 2 (AP-South) receives an initial slice of 30,000 RPS.

Each Envoy proxy queries an in-process, shared-memory C++ token bucket filter via gRPC local socket (/run/ratelimit.sock). The lookup takes less than 80 microseconds, entirely offline from the global network.

2. Asynchronous Token Borrowing & Dynamic Rebalancing

What happens if traffic is heavily skewed towards Asia while Europe is asleep?

1. Each cell's Cell Sync Agent tracks consumption velocity over a sliding 500ms window.
2. If Cell 2's utilization surpasses 75% of its allocated slice while Cell 1 is below 20%, Cell 2 initiates an asynchronous Token Lease Request to the Global Coordinator.
3. The Coordinator transfers unutilized token slices from Cell 1 to Cell 2 via lightweight Raft consensus.
4. If the Coordinator crashes or network connectivity is severed between regions, cells continue operating autonomously using their currently assigned static minimum quota slice.

// RateLimitCellEngine represents the autonomous cell bucket
type RateLimitCellEngine struct {
    localAvailableTokens atomic.Int64
    allocatedSlice       atomic.Int64
    leakRatePerSec       int64
}

func (r *RateLimitCellEngine) Allow(cost int64) bool {
    for {
        current := r.localAvailableTokens.Load()
        if current < cost {
            return false // Rate limit exceeded within local cell slice
        }
        if r.localAvailableTokens.CompareAndSwap(current, current-cost) {
            return true
        }
    }
}

3. Fail-Open Safe Mode

If the local rate-limiting daemon becomes completely unresponsive:

• The Envoy filter's circuit breaker triggers within 5ms.
• Traffic shifts immediately to FAIL_OPEN with a warning telemetry header (X-RateLimit-Degraded: true).
• The primary mission is preserving core application availability: rate limiting should never take down the system it was engineered to defend.

The "Baigan" Take (Expert Insights)

Junior engineers love to write blog posts about "Centralized Redis Token Bucket using Lua Scripts." In a high-traffic production banking cluster, that Lua script is an incident waiting to happen. Redis is single-threaded; when 200 microservice pods start spamming complex string pattern matching in Lua across 300,000 keys, your event loop latency will crater.

Treat rate limiting like water pressure management in a municipal grid. You don't ask the central reservoir every time a resident opens a bathroom tap. You supply water towers (local cells) with bulk allocations, and balance between towers periodically.

Keep your hot request path 100% in-memory and local. Let coordination happen asynchronously in the background.

Performance Checklist

• Eliminate all synchronous external network hops from the critical HTTP request admission path.
• Implement local Unix domain socket IPC between Envoy sidecars and local rate-limit daemons.
• Enforce atomic compare-and-swap (CAS) token counters to prevent thread locking overhead.
• Configure automated fail-open circuit breakers on Envoy rate-limit filters with a 5ms timeout.
• Implement dynamic token slice rebalancing based on 500ms rolling consumption velocity.