Multi-Stage Event Deduplication via Scalable Bloom Filters & Sliding Windows

The Context: "The Duplicate Billing Nightmare"

On August 27, 2026, during a brief cellular network flapping episode across southern India, our payment notification ingest pipeline processed 140,000 duplicate payment capture webhooks within a 3-minute window.

Because our ingestion microservice relied on a centralized SQL lookup (SELECT id FROM processed_events WHERE id = ?) to detect duplicates, the sudden surge of duplicate checks saturated the database connection pool. The queries began timing out.

To survive the timeout storm, a fallback circuit breaker was tripped which bypassed the duplicate check! As a result, 4,200 customers received double debit notifications, and several merchant payout ledgers were credited twice. Total baigan ka deduplication (a deduplication pipeline that fails the moment it is actually tested).

The Problem Statement

Building an ultra-fast, memory-efficient event deduplication filter capable of rejecting millions of duplicate events in real time without querying relational disks:

• Stream Throughput: 150,000 events per second.
• Deduplication Horizon: 24-hour sliding window.
• Accuracy Invariant: Zero false negatives (every duplicate event must be stopped).
• Latency Budget: Deduplication decision latency < 100 microseconds.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

1. The Multi-Stage Deduplication Architecture

A naive Bloom filter suffers from two critical flaws in streaming pipelines:

1. Saturation: As millions of items are added, the false positive rate ($p$) steadily degrades towards 100%.
2. Inability to Expire: You cannot delete an item from a standard Bloom filter without corrupting other bit vectors.

To overcome this, we designed a Sliding-Window Rotating Bloom Filter Array coupled with a distributed fallback:

2. The 24-Hour Rotating Bloom Filter Array

Instead of one giant static Bloom filter:

• We maintain 24 distinct hourly Bloom filters in shared application memory.
• Current hour $H$: Writes always populate the active filter for hour $H$.
• Lookups query all active hourly slices spanning the 24-hour window ($H_0 \dots H_{23}$).
• When an hour finishes, the oldest slice ($H - 24$) is simply dropped from memory and garbage collected, completely resetting its bit array with zero algorithmic complexity ($O(1)$).

public class SlidingBloomDeduplicator {
    private final RingBuffer<BloomFilter<String>> hourlyFilters;
    private final int windowHours = 24;

    public boolean isDuplicateCandidate(String eventId) {
        // Check across all active hourly slices
        for (BloomFilter<String> slice : hourlyFilters) {
            if (slice.mightContain(eventId)) {
                return true; // Possible duplicate, requires verification
            }
        }
        return false; // Guaranteed fresh unique event
    }

    public void registerEvent(String eventId) {
        hourlyFilters.getActiveSlice().put(eventId);
    }
}

3. Resolving the False Positive (The Redis Gate)

Bloom filters mathematically guarantee zero false negatives (if it says an item was NOT seen, it is 100% guaranteed fresh). However, they have a small false positive probability ($p \approx 0.01%$).

• If the Bloom filter returns false: The event is immediately passed to the core payment engine with zero external network hops. This handles 99.99% of normal traffic entirely in RAM in under 5 microseconds!
• If the Bloom filter returns true: It could be a real duplicate, or one of the rare 0.01% false positives. Only in this tiny fraction of cases does the service query the secondary Redis cluster (SET key value NX EX 86400) for absolute confirmation.

The "Baigan" Take (Expert Insights)

Many software architects love to say: "Just do a unique constraint in the SQL database, problem solved."

That is baigan ki scalability. When 150,000 events per second hit a relational database, running INSERT ... ON CONFLICT DO NOTHING will shred your database write locks and serialize your I/O threads.

By placing an In-Memory Sliding Bloom Filter in front of your pipeline, you filter out 99.99% of disk and network hits before they can even touch a database socket. You get absolute data safety without melting your storage infrastructure.

Performance Checklist

• Deploy an hourly rotating array of 24 Bloom filter slices in application memory.
• Size bit arrays using $m = -\frac{n \ln p}{(\ln 2)^2}$ targeting $p \le 0.0001$.
• Protect against false positives using an atomic Redis SET NX secondary verification gate.
• Route unique events directly to processing without synchronous relational database lookups.
• Alert when Bloom filter saturation crosses 80% of projected capacity.