Real-Time Fraud Anomaly Detection with Apache Flink

The Context: "The 3 AM Trigger"

It was late March 2024. A coordinated syndicate launched a distributed card-testing attack against our payment gateway at "NexusBanking" (name changed). The attack involved 500 different IP addresses making $1.00 authorization attempts. Because our fraud engine evaluated isolated requests synchronously against an Oracle rules table, each $1.00 attempt looked completely legitimate. We were missing the aggregate pattern. We tried to insert every transaction into a Redis list and evaluate the aggregate count every second, but pulling massive lists from Redis back into the JVM, analyzing them, and pushing results back caused our Java Garbage Collector to buckle under the weight of allocated objects. Total baigan analytics gap. We needed stateful, temporal stream processing.

The Problem Statement

Our constraints for the new anomaly pipeline:

• Velocity: Process 50,000 transactions per second.
• Statefulness: Detect patterns across sliding 1-minute, 5-minute, and 1-hour time windows.
• Exactly-Once: Prevent duplicate counting of transactions during microservice restarts.
• Latency: Output a fraud alert to the blocking queue within 50ms of the anomaly crossing the threshold.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We decoupled the aggregate fraud evaluation from the critical path of the authorization. The Payment Gateway now fires a raw transaction event into Kafka and proceeds with standard local checks. We deployed an Apache Flink cluster running Java 21 applications. Flink consumes the Kafka tx.raw topic. Rather than storing transaction history in an external Redis, Flink utilizes its internal RocksDB state backend. It keeps a running tally of transactions per CreditCard and per IP_Address perfectly within the memory/disk of the Flink task managers.

When a 1-minute sliding window evaluates that a single IP address attempted 50 unique credit cards, Flink instantly emits an alert to the tx.fraud.alerts topic. The Auth Blocking Service reads this and blacklists the IP in Redis. The next transaction from that IP is instantly blocked at the edge.

Concurrency & Locking

Because Flink partitions the stream by a specific key (e.g., IP_Address), there are zero locks. All transactions for 192.168.1.5 are routed to the exact same Flink worker thread. The worker manipulates its local, thread-safe state without ever waiting on a distributed lock or competing with other network requests. State is kept incredibly small, evaluating billions of data points strictly in RAM/SSD via RocksDB.

Failure Modes

What happens if a Flink worker node crashes? Flink implements distributed asynchronous checkpoints. Every 10 seconds, it saves the current sliding window aggregates from RocksDB into an S3 bucket, along with the specific Kafka offsets. When the worker restarts, it downloads the last S3 checkpoint, restarts the Kafka consumer from the exact saved offset, and continues processing without double-counting transactions or missing anomalies.

The "Baigan" Take (Expert Insights)

Look, everyone tries to build their own stateful stream processor using Spring WebFlux, ConcurrentHashMaps, and a ton of @Scheduled jobs. Don't. We spent 6 months fighting OutOfMemory errors and lost messages during pod restarts before adopting Flink.

The biggest lesson we learned with Flink is understanding Event Time vs Processing Time. If your network stutters, and a transaction from 3:00 AM arrives at Flink at 3:05 AM, a Processing Time window will count it in the 3:05 AM bucket. This destroys anomaly detection accuracy. You must extract timestamps directly from the Kafka payload and configure Flink Watermarks, allowing Flink to correctly assign late-arriving events back into the 3:00 AM window. It's the only way to catch sophisticated, distributed attacks perfectly.

Performance Checklist

• Configure Apache Flink to use RocksDB for large state backends to prevent JVM Heap exhaustion.
• Enable asynchronous Checkpointing to S3 to guarantee exactly-once state recovery.
• Explicitly configure Flink to use Event Time (WatermarkStrategy) rather than Processing Time.
• Use KeyedProcessFunction in Flink to ensure zero-lock, thread-local state manipulation.