Real-Time CDC Architecture with Debezium and Kafka

The Context: "The 3 AM Trigger"

It was November 2023. Our primary data analytics warehouse for "GlobalRetail" (name changed) was refreshed nightly via a massive Spring Batch job. The job executed a SELECT * FROM ORDERS WHERE UPDATED_AT > yesterday against the primary Oracle 19c RAC instance. As transaction volume grew, this batch job started taking 7 hours. Worse, it caused massive table locks during the most critical Asian market trading hours. One morning at 3 AM, the batch job collided with a database stats-gathering routine. The ORDERS table locked entirely. Transactions failed globally for 45 minutes until the DBA manually killed our batch worker. Total baigan extraction pattern. We had to move from batch polling to real-time streaming without adding load to the DB CPU.

The Problem Statement

Our constraints for the extraction pipeline:

• Zero Impact: Extracting data must not use the Oracle SQL execution engine or compete with transaction threads.
• Latency: Changes made in Oracle must reflect in the Analytics Warehouse within 5 seconds.
• Accuracy: We cannot miss a single update or delete. Polling UPDATED_AT misses hard deletes entirely.
• Resilience: The pipeline must survive network partitions and database failovers automatically.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We completely eliminated the Spring Batch SQL polling jobs. Instead, we shifted to Change Data Capture (CDC). We deployed Debezium running within a Kafka Connect cluster in Kubernetes. Debezium doesn't run SQL SELECT statements against the tables. Instead, it hooks into Oracle LogMiner and continuously reads the fundamental Oracle Redo Logs—the exact same binary logs Oracle uses for database replication and crash recovery.

Every time a Java service issues an INSERT, UPDATE, or DELETE, Debezium parses the binary log and instantly publishes an Avro-encoded event to a specific Kafka topic (e.g., oracle.server1.orders). The downstream analytics pipeline simply consumes these raw events and reconstructs the state.

Concurrency & Locking

The absolute beauty of this system is that reading Redo Logs requires zero row/table locks and consumes essentially none of the Database CPU meant for query execution. The transaction processing is completely decoupled from the data extraction. If the downstream pipeline slows down, Kafka buffers the CDC events. If Debezium parses logs slower than Oracle writes them, it simply falls behind in the Redo Log sequence numbers without affecting Oracle's ability to serve application traffic.

Failure Modes

What happens if the Kubernetes node hosting the Debezium connector dies? Debezium tracks the System Change Number (SCN) of every event it successfully commits to Kafka. When the Pod restarts on a new node, it simply requests Oracle LogMiner to resume parsing from the last successfully committed SCN. No duplicate data, no missed data. Complete At-Least-Once (and often Exactly-Once) delivery semantics.

The "Baigan" Take (Expert Insights)

Look, everyone turns on CDC and thinks their analytics problems are solved. But Oracle LogMiner is a tricky beast.

We initially configured Debezium to mine the active online Redo logs. This worked until Oracle recycled the log file before Debezium finished reading it (during a massive data import event). Debezium threw an exception and fundamentally crashed, requiring manual intervention to hunt down the missing Archive Logs. The baigan reality is that you must configure Debezium to read from the Archive Logs specifically, and you must ensure your DBAs retain those Archive Logs for at least 72 hours. Furthermore, capturing LOBs (Large Objects) via LogMiner is incredibly slow. We had to explicitly exclude the massive JSON_PAYLOAD columns from the Debezium capture filter to keep the latency under our 5-second SLA.

Performance Checklist

• Use Debezium over Kafka Connect to abstract away LogMiner API complexity.
• Configure Oracle to retain Archive Logs for at least 72 hours to survive connector outages.
• Filter out LOB (CLOB/BLOB) columns from the CDC capture to preserve parsing speed.
• Encode Kafka CDC payloads using Avro and a Schema Registry to minimize network payload size.