Zero-Downtime Hash Slot Rebalancing and Key Eviction Under Heavy Write Pressure

The Context: "The Shard Rebalance Freeze"

On Tuesday, August 18, 2026, at 04:00 PM IST, our operational engineering team attempted an online scale-out of our 12-node Redis cluster. Three new master nodes were added to absorb surging session cache traffic.

The automated script initiated slot migration (redis-cli --cluster reshard) to move 4,096 hash slots from saturated nodes to the new nodes.

Within minutes, client connections began timing out. Hash slot migrations in standard Redis use synchronous MIGRATE commands, which block the single-threaded Redis engine during the transfer of large keys. One of the slots contained a 280MB sorted set containing session tracking analytics. The master node blocked for 4.2 seconds to serialize, transmit, and delete that single key.

Because the node was blocked, cluster gossip heartbeats timed out, causing the remaining cluster masters to flag the node as FAIL. An automated, uncoordinated master failover triggered mid-migration, leaving hundreds of hash slots corrupted in an MIGRATING / IMPORTING zombie state. Total baigan ka reshard (a catastrophic operational disaster).

The Problem Statement

Performing zero-downtime cluster rebalancing and hash slot live migrations on high-throughput Redis clusters under heavy continuous read/write load:

• Cluster Scale: 18 Master nodes holding 240 GB of active session state.
• Continuous Write Throughput: 180,000 writes/sec across active slots.
• SLA: Client command response times must remain under 2.0ms (P99) throughout resharding operations.
• Safety: Zero data loss and zero split-brain slot ownership states during node failures.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

1. The Dynamic Dual-Phase Migration Protocol

Redis Cluster manages data sharding across 16,384 distinct hash slots using CRC16: $\text{slot} = \text{CRC16}(\text{key}) \pmod{16384}$.

When slot $S$ is moving from Node A to Node B:

1. Node B is set to CLUSTER SETSLOT S IMPORTING A.
2. Node A is set to CLUSTER SETSLOT S MIGRATING B.
3. During this transition, Node A remains the authoritative owner of the slot.
4. If a client queries Node A for a key that has already moved, Node A replies with a redirection error: -ASK S NodeB_IP:Port.
5. The client issues an ASKING command to Node B, followed by the original query. This guarantees that traffic continues flowing with zero downtime while individual keys are mid-flight.

2. BigKey Detection and Chunked Pipeline Migration

The primary killer of live resharding is "BigKeys" (collections, sets, or hashes with hundreds of thousands of elements). A single large key blocks Redis during synchronous serialization.

Before migrating any slot:

• We scan the target slot using MEMORY USAGE <key> via non-blocking asynchronous scanner scripts.
• Keys larger than 512KB are prohibited from being transferred in monolithic MIGRATE blocks.
• We migrate keys in strictly bounded batches of 50 keys, with a maximum network socket timeout of 500ms.
• If a single key exceeds 5MB, it is decomposed into split shards prior to migration.

3. Client Driver Awareness (Lettuce / Cluster Topology Refresh)

A naive Redis client library will flood the cluster with -MOVED and -ASK redirects if its internal cluster topology cache is stale.

We configure adaptive topology refresh policies:

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
    .enableAdaptiveRefreshTrigger(
        ClusterTopologyRefreshOptions.RefreshTrigger.MOVED_REDIRECT,
        ClusterTopologyRefreshOptions.RefreshTrigger.ASK_REDIRECT,
        ClusterTopologyRefreshOptions.RefreshTrigger.PERSISTENT_RECONNECTS
    )
    .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(10))
    .enablePeriodicRefresh(Duration.ofMinutes(5))
    .build();

The "Baigan" Take (Expert Insights)

Many administrators kick off redis-cli --cluster reshard and immediately walk away for lunch. That is baigan ka confidence (reckless overconfidence).

If your cluster has maxmemory-policy set to allkeys-lru or volatile-lru, migrating slots into a newly added node can trigger a massive eviction wave if memory sizing is unbalanced. The new node hits its ceiling, begins evicting hot keys, and downstream databases crash under the resulting cache stampede.

Always pre-warm and pre-size the target nodes. Set your memory buffers conservatively, migrate slots in small batches during low-traffic windows, and monitor instantaneous_ops_per_sec on both the source and target nodes continuously.

Performance Checklist

• Pre-scan hash slots for BigKeys (>512KB) before executing live slot migration.
• Migrate keys in small bounded batches of 50 using non-blocking pipeline timeouts.
• Configure adaptive cluster topology refresh triggers (MOVED and ASK) in client libraries.
• Set migration network socket timeouts strictly below 1,000ms to prevent gossip protocol partitions.
• Verify memory headroom on receiving master nodes to prevent accidental cache eviction avalanches.