The Context: "The 3 AM Trigger"
It was April 2024. A major merchant partner on our gateway, "GlobalShopify" (name changed), updated their SSL certificates incorrectly. For four straight hours, every webhook notification we fired to their endpoints to confirm a completed payment failed with a 525 Handshake error. Our legacy webhook dispatcher was a simple Spring Boot @Scheduled cron job that basically SELECTed from a WEBHOOK_QUEUE table, fired the HTTP request, and updated the status. Because their endpoints were down, our dispatcher retried 30,000 times a second, completely saturating the Oracle connection pool and starving our core ingestion APIs. Total baigan design. We were DDoSing ourselves because a client broke their own infrastructure. We needed an isolated, resilient webhook architecture.
The Problem Statement
Our new webhook dispatcher had to respect these constraints:
- Isolation: Failed webhooks must not consume resources meant for primary transactional components.
- Fairness: A single broken client must not block webhooks going to healthy clients.
- Delivery Guarantees: Provide "At Least Once" delivery mechanics.
- Backoff Mechanics: Implement intelligent exponential backoffs up to 72 hours.
Visual Architecture
The Core Logic / Process
Data Flow
We decoupled the event generation entirely from the delivery. When a payment completes, the Core Engine simply inserts a record into an Oracle WEBHOOK_EVENTS table as part of its primary database transaction (the Outbox Pattern). A Debezium Change Data Capture (CDC) connector reads the Oracle Redo Logs and streams these events into a Kafka topic.
A central Webhook Dispatch Router reads the primary topic and partitions the events into separate merchant-specific topics based on the merchant_id. This is crucial: it prevents "Head of Line" blocking.
Concurrency & Locking
By partitioning webhooks at the Kafka level, we no longer need to execute distributed database locks or SELECT FOR UPDATE SKIP LOCKED queries on the Oracle DB, which was killing the CPU. The Webhook Worker JVM uses Java 21 Virtual Threads to fire thousands of parallel outbound HTTP requests. If a request fails, we do NOT block the Kafka partition. Instead, the failing event is published to a Dead Letter Queue (DLQ) with a custom retry header, and the worker commits the current offsets.
Failure Modes
The DLQ is processed by a highly-tuned Retry Service that reads the retry count from the Kafka headers. It uses Resilience4j to apply an exponential backoff algorithm. If the client is down, the Retry Service sleeps the virtual thread. If the client remains completely dead for 72 hours, the webhook is eventually moved to an ARCHIVED_FAILURES table in Oracle for manual support intervention.
The "Baigan" Take (Expert Insights)
Look, everyone builds outbox patterns and thinks their webhooks are instantly reliable. But if a single enterprise client goes offline, and your workers spend 30 seconds blocking on HTTP Socket Timeouts for every single message, your entire queue will freeze.
The trick is Time Limiting and Circuit Breaking at the client level. In our Webhook Worker JVM, we use Resilience4j Circuit Breakers configured per merchant_id. If "GlobalShopify" returns 50 consecutive 5xx errors or Socket Timeouts, their specific circuit breaker instantly trips to OPEN. Subsequent virtual threads trying to send to "GlobalShopify" will instantly fail-fast—freeing up the thread in 1ms instead of 30,000ms. We immediately route the event to the DLQ. This keeps the worker fleet perfectly healthy, even when upstream clients are burning down.
Performance Checklist
- Use the Transactional Outbox pattern paired with Debezium CDC.
- Partition Kafka topics by
merchant_idto prevent Head of Line blocking. - Wrap HTTP dispatchers in Virtual Threads (
java.net.http.HttpClientsupports this natively). - Implement Per-Client Circuit Breakers to fail-fast on severely degraded endpoints.