Taming Multi-Region Split-Brain: Fencing Tokens, Quorums, and Storage-Level STONITH | System Design Vault

The Context: "The 3 AM Trans-Pacific Sever"

It was a Saturday night in late July 2026. At precisely 03:14 AM IST, an undersea fiber bundle between Singapore and Tokyo suffered an anchor drag strike. Within 400 milliseconds, our primary cross-region heartbeats went silent.

Our Active-Passive Oracle Data Guard topology with automated fast-start failover (FSFO) saw the loss of primary pings. The secondary observer in Tokyo immediately presumed the Singapore primary was dead. It initiated an autonomous failover, promoting the Tokyo replica to Primary. Meanwhile, the Singapore cluster was still alive and processing thousands of regional settlement transactions per minute from local microservices.

Two primary databases. One shared sequence generator logic. Two distinct ledger states.

When the undersea links recovered 14 minutes later, we were staring directly into the abyss of split-brain state corruption: duplicate transaction IDs, diverged redo logs, and inconsistent ledger balances across regional merchant settlement ledgers. Total baigan ka tamasha (an absurd, chaotic disaster).

The Problem Statement

Preventing split-brain during unannounced network partitions requires strictly deterministic distributed coordination under brutal physical constraints:

Throughput: 42,000 Transactions Per Second (TPS) across distributed payment gateways.
Latency Budget: P99 database write latency must remain under 12ms during normal operations, and under 50ms during degradation.
Consistency vs Availability: Absolute ACID serializability is legally required for financial ledgers. Zero dual-write tolerance. Under no circumstances may two compute nodes believe they hold the write lease concurrently.

Visual Architecture

The Core Logic / Process

1. The Tri-Region Quorum Witness

A two-node cluster cannot distinguish between a dead peer and a partitioned network without a third independent observer. We deployed a lightweight Witness daemon in an independent availability zone (Region C - Mumbai).

A failover requires a strict majority vote ($N/2 + 1$). If Region A (Singapore) loses connectivity to Region B (Tokyo), it queries Region C. If Region A can still talk to Region C, it retains the quorum. If Region B also queries Region C, Region C explicitly denies failover authority to Region B.

2. Monotonic Fencing Tokens (Lease Epochs)

Network partitions often cause asynchronous delayed packets ("zombie requests"). To prevent an isolated primary node from committing pending in-flight transactions after promotion has begun on the secondary:

Every database write must be accompanied by a monotonically increasing 64-bit fencing token: $\tau$.
The witness increments the fencing token ($\tau_{new} = \tau_{old} + 1$) during any promotion ceremony.
The storage subsystem rejects any write request tagged with a token $\tau < \tau_{current}$.

public class FencedStorageDriver {
    private final AtomicLong currentEpoch = new AtomicLong(0);

    public CompletableFuture<CommitResult> executeFencedWrite(byte[] payload, long transactionEpoch) {
        if (transactionEpoch < currentEpoch.get()) {
            throw new FencingTokenStaleException("Rejected zombie write. Node epoch "
                + transactionEpoch + " superseded by " + currentEpoch.get());
        }
        return storageController.persist(payload);
    }
}

3. Storage-Level STONITH (Shoot The Other Node In The Head)

Software-level fencing is insufficient if a rogue node experiences an unannounced 20-second JVM "Stop-The-World" garbage collection pause. Upon resuming, the rogue node believes it still holds the lock and writes stale data before checking its network status.

We enforce SCSI-3 Persistent Reservations (PR). When Region B is granted promotion rights by the Quorum Witness:

Region B issues a PREEMPT_AND_ABORT service action reservation key to the underlying SAN/NVMe-oF target.
The storage controllers instantly unregister Region A's registration keys and clear all queued commands.
Any subsequent write attempt by Region A results in an immediate hardware I/O check condition error, panicking the kernel before memory buffers can corrupt disks.

The "Baigan" Take (Expert Insights)

Baigan ki theory looks great in vendor whitepapers. They will tell you "Automatic Fast-Start Failover handles network partitions seamlessly." Don't fall for that nonsense.

In production, the most dangerous failure mode is not a clean, hard cut; it is flapping packet loss (gray failures). A link dropping 12% of packets every 3 seconds will trigger repeated failover thresholds, reset election timers, and throw both clusters into half-promoted states.

You must configure asymmetric decay timers:

Require at least 5 consecutive missed heartbeats (each 1000ms apart) before declaring a quorum lost.
Enforce a mandatory minimum promotion hold of 120 seconds. Once a standby promotes, it must stay primary for at least 2 minutes even if the old primary comes screaming back. This prevents flip-flopping elections from destroying your throughput.

Performance Checklist

Configure SCSI-3 Persistent Reservation keys on all shared LUN targets.
Deploy the Quorum Witness observer in an independent cloud region with low latency to both clusters.
Attach monotonic fencing tokens to all distributed RPC payloads and DB sequence allocations.
Set TCP keepalive probe thresholds: tcp_keepalive_probes = 3, tcp_keepalive_intvl = 2, tcp_keepalive_time = 5.
Test simulated partition resilience quarterly using automated network blackhole routing (iptables -j DROP).