Scaling Reconciliations with Chunk Processing

The Context: "The 3 AM Trigger"

It was May 2024. Our payment processor client, "TransactPlus" (name changed), performed a nightly reconciliation of 5 million transactions against third-party bank files. Their legacy Java code used a standard List<Transaction>. It booted up at 1:00 AM, queried all 5 million rows from Oracle RAC into an ArrayList, iterated through them, and updated the statuses one by one. At 3:00 AM, the JVM simply gave up. The Garbage Collector couldn't find enough contiguous Heap space to hold 5 million massive Java objects. The infamous java.lang.OutOfMemoryError: Java heap space crashed the pod. The operations team had to manually slice the file into 20 pieces and run them sequentially, finishing well after the 8:00 AM SLA. Total baigan memory management. You cannot load the ocean into a swimming pool. We needed Spring Batch Chunk Processing.

The Problem Statement

Our constraints for the nightly batch:

• Memory Footprint: Process 5 million rows using an absolute maximum of 2GB of JVM Heap.
• Velocity: The entire reconciliation must finish in under 30 minutes.
• Transactional Integrity: If row 499,999 fails with a network timeout, we cannot roll back the previous 499,998 successful rows, nor can we leave the batch in an unknown state.
• Tooling: Implement natively using Spring Batch and Java 21.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We entirely ripped out the in-memory ArrayList processing and migrated to the Spring Batch Chunk-Oriented Processing model.

We deployed a JdbcCursorItemReader. Rather than SELECT * FROM tx, the cursor opens a persistent TCP connection to Oracle and streams rows across the network one-by-one. We defined a chunk size of 1,000. Spring Batch pulls exactly 1,000 rows through the Reader, passes them to the ItemProcessor to map the bank algorithms, and hands the List of 1,000 processed objects to the ItemWriter. The Writer uses a JdbcBatchItemWriter to execute a highly optimized PreparedStatement.addBatch() against Oracle.

Concurrency & Locking

Once the Writer finishes executing the SQL batch, Spring Batch executes a standard Database COMMIT. This is the magic: After that commit, all 1,000 Java objects within that Chunk go completely out of scope. The Garbage Collector instantly reclaims them in the Young Generation pool. The JVM Heap never exceeds a few megabytes, despite ultimately processing millions of rows. To achieve the 30-minute SLA, we configured the Step to use a TaskExecutor. Spring Batch spins up 10 Virtual Threads, each safely claiming a unique chunk of 1,000 rows concurrently, multiplexing the database updates massively without JVM contention.

Failure Modes

What happens if the bank API goes down during chunk number 500? Because each chunk is its own distinct ACID transaction, the previous 499,000 rows remain safely committed to Oracle. We configured the Step with a FaultTolerant() layer. If the processor encounters a BankTimeoutException, it is configured to skipLimit(50). It writes the ID of the failed transaction to a dead-letter table and safely continues processing the rest of the chunk.

The "Baigan" Take (Expert Insights)

Look, everyone turns on Spring Batch and assumes it solves memory problems automatically. But if you use the wrong Reader, you'll still crash.

We initially attempted to use JpaPagingItemReader because our engineers loved Hibernate. The baigan reality is that Hibernate's First-Level Cache (the EntityManager) implicitly holds strong references to every entity it loads. Even with chunking, Hibernate was artificially keeping the objects alive across chunks, defeating the Garbage Collector and leading to the exact same OOM crash.

When you process millions of rows, you must brutally bypass the ORM. You must use raw JdbcCursorItemReader and JdbcBatchItemWriter. Map your objects manually using a RowMapper. ORMs are built for transactional micro-services, not for high-velocity data pipelines. Save Hibernate for your REST APIs.

Performance Checklist

• Adopt the Spring Batch Chunk-Oriented model to process massive datasets sequentially.
• Use JdbcCursorItemReader to stream rows from Oracle without loading them all into RAM.
• Use JdbcBatchItemWriter to leverage native JDBC batching for write performance.
• Discard JPA/Hibernate entirely for batch jobs to prevent EntityManager memory leaks.