Fraud Detection Pipelines: Real-Time Decisions at Scale

The Context: "The 100ms Window"

It was mid-2024. A customer in Dubai was swiping their credit card. Between the "Swipe" and the "Approval" on the terminal, we had exactly 200ms. If we took longer, the transaction would "Time out" and the merchant would lose the sale.

But in that 200ms, we had to run 15 different checks: "Is the card reported stolen?", "Is the location unusual for this user?", "Is the amount exceeding the velocity limit?". If we were too slow, we lost users. If we were too "Loose," we lost Millions. This is the Fraud-Detection Window. This article explores the architecture of Real-Time Fraud Detection and how we used Kafka Streams to make sub-100ms decisions on 50,000 transactions a second.

The Problem Statement

Fraud detection is a race against the clock.

Goal: Detect & Block Fraud Before the Authorization.
Metric: Execution Latency < 100ms (P99).
Challenge: The "Cold Start" of context. You can't just check one transaction; you have to check it against the last 50 transactions for that card.

Visual Architecture: The Real-Time Stream Pipeline

The Core Logic: The Stream Processor

1. Kafka Streams: The Stateful Transformer

We use Kafka Streams for the heavy lifting. Unlike a standard "Batch" job, Kafka Streams processes every transaction as it arrives.

Windowing: We can "Window" the data. E.g., "Count how many transactions this card has made in the last 1 minute." If the count > 5, flag it as a "Velocity Burst".

2. Redis L1 Context (The 'Context' Bridge)

Checking a 10TB Oracle DB for transaction history takes 500ms. We can't afford that.

The Solution: We mirror the "Last 20 transactions" and "Customer Risk Score" from Oracle into Redis as a key-value store.
Latency: Fetching context from Redis takes < 2ms.

3. The "Hybrid" Rule Engine

We use a two-step evaluation:

Fast Rules: Simple static checks (e.g., "Is the amount > $10,000?").
ML Scoring: A pre-computed Machine Learning model (e.g., XGBoost) running in a Sidecar container that returns a "Risk Score" of 0.1 to 1.0.

The "Baigan" Take: Insights from the Trenches

Look, people love talking about "Complex Neural Networks" and "AI-first Fraud Detection". But baigan ka logic hai wo (that's nonsense) if you can't even tell if the same card was swiped in London and Tokyo in the same 5 minutes.

The 'False Positive' Killer. The biggest mistake I've seen is "Over-Blocking". If you block every "Unusual" transaction, your customer support team will be under a DDoS attack of angry users.

Rule: Use "Soft Failure". If the risk score is medium (e.g., 0.6), don't block. Trigger a Push Notification to the user: "Did you just spend $50 at Store X? Reply YES or NO."
Rule: Your "Feature Store" is more important than your "Model". If your Kafka stream can't accurately calculate "User's Average Spend" in real-time, your ML model is just guessing.

Performance & Implementation Checklist

KSQL / Kafka Streams: Use KSQL for simple "Windowed Counts" and Java-based Kafka Streams for complex "Cross-Entity" logic.
SerDe Optimization: Use Protobuf or Avro for your Kafka messages. JSON is too slow to parse at 50k TPS.
Sidecar Deployment: Put your ML model in a local Sidecar (C++) or use ONNX Runtime in Java to avoid a remote API network call.
A/B Testing: Run your "New Rules" in Shadow Mode for a week. Replay historical data and check: "Would this have blocked legitimate transactions?"

Conclusion

Real-time fraud detection is the ultimate test of "Architecture under Pressure". In a 100ms window, you must balance security with user experience. By leveraging the power of Kafka Streams and Redis-based context, you transform a slow, brittle auditing task into a hyper-fast defense system. Block the fraud, approve the honest user, and never miss a swipe.

Verified for Technical Nexus v1.9.0 | Part of the System Design Vault archive.