Distributed Transactions with the Choreographed Saga Pattern

The Context: "The 3 AM Trigger"

It was December 2023. Our client "TravelSync Solutions" (name changed) operated a distributed holiday booking platform. A customer booking a package required reserving a flight, a hotel, and a rental car. Initially, the orchestration API used a synchronous 2-Phase Commit (2PC) spanning the Flight, Hotel, and Car oracle databases using XA transactions. It worked fine until the Car service's database underwent a minor network partition. The XA coordinator held the row locks in the Flight and Hotel databases open for 45 seconds waiting for the Car service to vote. During this time, 10,000 other bookings backed up behind the locks. The database connections exhausted, and the entire platform experienced a cascading outage. Total baigan design. We needed to ditch 2PC for eventual consistency.

The Problem Statement

Our transaction constraints going forward:

• Consistency: The system must guarantee that we either book everything or roll back everything (Eventual Consistency).
• Latency: The initial response to the user must be under 200ms (Async processing).
• Locking: Absolutely no distributed, cross-database row locks.
• Resilience: A failure in one microservice should not block the others.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We implemented a Choreographed Saga Pattern using Kafka as our event backplane. When a user creates an order, the Order API inserts a PENDING order into its local Oracle DB and instantly returns a HTTP 202 Accepted. An outbox pattern pushes an OrderCreated event to Kafka.

The Flight, Hotel, and Car services passively listen to this topic. They independently attempt to fulfill their portion of the booking within their own local Oracle databases. Upon success, they emit a <Service>Booked event.

Concurrency & Locking

Because we don't hold global locks, there is a distinct possibility that the Hotel service books the room, but a millisecond later, the Flight service fails to secure a seat. We handle this via Compensating Transactions. A central Saga Observer listens to the success/failure events. If it receives a FlightFailed event, but already received a HotelBooked event, it emits a CancelHotel command to Kafka. The Hotel service consumes this and executes an inverse transaction (releasing the room reservation and issuing a refund ledger entry) in its own DB.

Failure Modes

What happens if the Hotel service crashes before consuming the CancelHotel command? Because Kafka topics are durable and we use explicit offset commits in our Java 21 Kafka Consumers, the command sits safely in the partition. When the Hotel service pod restarts in Kubernetes, it simply picks up where it left off, realizes it needs to cancel the booking, and executes the compensating transaction.

The "Baigan" Take (Expert Insights)

Look, everyone loves the idea of Sagas because 2PC is evil. But what tutorials don't tell you is the nightmare of "Semantic Locks".

If you use a Saga, the Hotel room is "booked" locally while the Flight transaction is still processing. What happens if the user tries to cancel their hotel room via a separate HTTP request during this 500ms window? Our initial architecture allowed this, which meant the Compensating Transaction failed because the user had manually canceled it first. Total state corruption.

We had to introduce the PENDING state explicitly into the business logic. A semantic lock. Every entity involved in a Saga has a SAGA_STATUS column. While an entity is PENDING, external REST API operations (like user cancellations) are blocked with an HTTP 409 Conflict. You cannot just use DB transactions; you have to enforce the lock in your Java code.

Performance Checklist

• Utilize the Outbox Pattern to guarantee event emission matches the local DB commit.
• Ensure every microservice implements an explicit Compensating Transaction endpoint/consumer.
• Protect in-flight sagas using Semantic Locks (status = PENDING) to prevent external state mutations.
• Use a distributed tracing system (OpenTelemetry) passing traceparent headers through Kafka to track the entire saga lifecycle.