Five-Nines Availability: Building Active-Active Across 3 Regions

The Context: "The Region Collapse"

It was October 2024. A major cloud provider experienced a 6-hour "Networking Partition" in their US-East-1 region. Most global banks, even those with "High Availability," were either completely down or extremely unstable because their "Backup" and "Primary" regions were both affected by the same global routing issue.

Our system, however, stayed 100% online. Why? Because we were running an Active-Active-Passive (3-Region) architecture across three separate geographic clusters. This is the architecture of 99.999% Availability and how we eliminated "Single Points of Failure" from our global banking fabric.

The Problem Statement

99.9% (3 nines) allows for 8 hours of downtime a year. 99.999% (5 nines) allows for only 5 minutes of downtime a year.

Goal: Zero Downtime (RTO=0) during a regional failure.
Metric: Sub-5ms Health Checks, Sub-60s global failover.
Challenge: The "State Synchronization" across 3 regions. Latency between London, Mumbai, and New York is ~150-200ms. You cannot use synchronous mirroring at this distance.

Visual Architecture: The 3-Region Active Global Loop

The Core Logic: The Availability Engine

1. Anycast DNS & GSLB

We don't use "Static IP" addresses. We use Anycast IP or Global Server Load Balancing (GSLB). If the EU region becomes unreachable, the GSLB detects it within 5 seconds and automatically re-routes the next DNS request to the APAC or US region.

2. State Sync: The "Source of Truth" Partition

In a 3-region setup, you must decide where the "Master" database lives.

The "Anchor" Region: One region (e.g., EU) is the master for writes.
The "Local" Region: Other regions (APAC) use a local read-replica (Active Data Guard) to serve 95% of traffic.
The "Cross-Region Write": Application threads in APAC send "Write" requests via a high-speed private tunnel (e.g., Direct Connect or ExpressRoute) to the EU database.

3. Circuit Breakers for Cross-Region Calls

If the private tunnel between APAC and EU is slow, the APAC region must fail fast. Instead of blocking, it can "Degrade" – allowing read-only access to balances but stopping new outgoing transfers.

The "Baigan" Take: Insights from the Trenches

Look, people love talking about "Global Consistency" and "Spanner-like databases". But baigan ka logic hai wo (that's nonsense) when you're under a heavy DDOS attack and your Cross-Region communication is severed.

The 'Split-Brain' Nightmare. The biggest risk of "Five-Nines" is two regions thinking they are both the "Master". If EU and APAC lose contact, and both start accepting payments, you will have a reconciliation disaster.

Rule: Use a Quorum Mechanism (like Zookeeper or Etcd) in a third, separate region. A region can only be "Master" if it can reach at least one other region.
Rule: Don't just "Watch" the CPU. Watch the Error Rate. A region might have 0% CPU because its load balancer is dead, but its health check still returns "OK". Your health checks must be end-to-end (API -> DB -> API).

Performance & Implementation Checklist

Redundant Tunnels: Ensure your private mesh (e.g., AWS Peering) has at least 2 geographically separate physical routes.
Database Flashback: Keep a 24-hour Flashback window. If a sync fails, you need to "Rewind" state to find the mismatch.
Region Isolation: A failure in EU should NEVER cause a failure in APAC. Ensure no shared caches or shared service-discovery nodes exist between regions.
Zero-Trust Networking: Use mTLS between regions. If one region is compromised, the others can "Server the Connection".

Conclusion

Five-Nines Availability is not a "Configuration"; it is a Culture. You don't build it by buying more servers; you build it by assuming every single component will fail at the worst possible time. By distributing your state, isolating your clusters, and automating your failover, you turn your banking platform into a truly global, indestructible infrastructure.

Verified for Technical Nexus v1.9.0 | Part of the System Design Vault archive.