Cross-DC Distributed Locks with Apache ZooKeeper

The Context: "The 3 AM Trigger"

It was February 2024. Our logistics client, "FleetRoute" (name changed), scheduled their nightly batch computation to assign optimized delivery routes. The architecture was deployed identically across two Active-Active Kubernetes clusters (one on AWS US-East-1, the other on GCP US-East4). Both clusters spun up their Spring Batch workers at exactly 3:00 AM. We thought we were smart by using Redis SETNX for acquiring a distributed lock to ensure only one cluster actually executed the batch. Unfortunately, during a split-brain network partition between the two clouds, the Redis cluster split. Both K8s clusters successfully acquired the "exclusive" lock locally. Two separate batches ran, assigning the same delivery trucks to two different states. Total baigan logistics failure. We needed a consensus-based lock manager, not just a fast key-value store.

The Problem Statement

Our constraints for the distributed lock manager:

• Strict Consistency: Tolerate network partitions (Split-Brain) perfectly. If the network splits, only the partition holding the majority quorum may grant the lock.
• Fairness: Provide a strict First-In-First-Out (FIFO) queue for lock acquisition.
• Liveness: Automatically release the lock if the holding Java process crashes, preventing permanent deadlocks.
• Tooling: Implement natively in Java 21 without relying on polling.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We replaced the Redis locking mechanism with Apache ZooKeeper, accessed via the Apache Curator framework in Java. When the Spring Batch worker starts, it attempts to acquire an InterProcessMutex provided by Curator. ZooKeeper uses the ZAB (ZooKeeper Atomic Broadcast) protocol to ensure that writes are strictly ordered and replicated across a majority of nodes. If an AWS network partition isolates half the ZooKeeper nodes, that partition inherently loses quorum and ceases to grant locks.

Concurrency & Locking

Unlike Redis where you have to constantly while(true) poll the lock to see if it's free, ZooKeeper uses Ephemeral Sequential ZNodes and Watches. When AWS and GCP pods both request the lock, ZooKeeper creates /batch-lock/_lock-0001 (for AWS) and /batch-lock/_lock-0002 (for GCP). The AWS pod owns the lock because it has the lowest sequential number. The GCP pod does NOT poll. Instead, it places a Watch on the AWS ZNode. The GCP Java thread goes perfectly idle. As soon as AWS releases the lock (or crashes), ZooKeeper instantly fires an event to the GCP pod, waking the thread to acquire the lock. This is event-driven distributed locking.

Failure Modes

What happens if the AWS JVM hits a Fatal OOM Error and dies without releasing the lock? Because the lock is tied to an "Ephemeral" ZNode, ZooKeeper links the existence of the node to the active TCP session of the client. If ZooKeeper stops receiving heartbeats from the crashed AWS pod, it deletes the Ephemeral node automatically. The Watch triggers, and the GCP pod safely takes over.

The "Baigan" Take (Expert Insights)

Look, setting up ZooKeeper just for a lock seems like massive overkill when Redis is already in your stack. But if you are managing financial routing, fleet assignments, or database migrations, Redis is fundamentally unsafe for strict locks due to asynchronous replication.

The baigan mistake engineers make with ZooKeeper is treating the lock as an absolute guarantee forever. Even with ZooKeeper, if your AWS Java process undergoes a massive 30-second Garbage Collection pause, ZooKeeper might think the process died, delete the ephemeral node, and grant the GCP pod the lock. Suddenly, you have two workers running!

To solve this, your actual business logic (e.g., the Oracle UPDATE statement) must still use an internal fence token or a monotonic version number derived from the ZK transaction ID (zxid). If the AWS pod wakes up from GC and tries to commit its batch to Oracle using an older token, Oracle rejects the write. This is "Fencing," and it's mandatory for true distributed safety.

Performance Checklist

• Use Apache Curator's InterProcessMutex to implement the recipe flawlessly in Java.
• Deploy the ZooKeeper cluster across multiple Availability Zones to ensure split-brain quorum safety.
• Rely on Ephemeral ZNodes and Watches to avoid CPU-intensive polling loops.
• Pass the ZooKeeper zxid to your final database transaction to build a Fencing Token against GC pause zombies.