Operating Kafka Tiered Storage with Low-Latency Compacted Topics

The Context: "The $40,000 NVMe EBS Disk Wall"

On September 2, 2026, our financial data platform hit a hard operational constraint: our 18-broker Apache Kafka cluster was storing 180 Terabytes of historical transaction events on AWS io2 block storage to support a 90-day regulatory audit replay SLA.

The cloud bill was eye-watering: over $42,000 every month purely for attached EBS volumes. More dangerously, Kafka broker recovery times after a node restart had stretched to 3.5 hours because the broker had to scan and verify terabytes of local segment indexes during startup.

When we attempted to enable Kafka Tiered Storage (offloading cold log segments to Amazon S3), a subtle disaster struck: several of our most critical topics were configured with Log Compaction (cleanup.policy=compact). The background tiering cleaner clashed with the log compactor, resulting in duplicate tombstone retention, missing intermediate account states, and consumer fetch timeouts on cold S3 reads. Total baigan ka tiering (tiered storage configured without understanding segment boundaries).

The Problem Statement

Architecting a cost-effective, multi-petabyte Kafka streaming platform that offloads cold data to cloud object stores while maintaining sub-10ms fetch latencies for real-time consumers and deterministic compaction:

• Cluster Throughput: 650 MB/s continuous ingestion across 1,200 partitions.
• Storage Tiering Ratio: 95% of data stored on object storage (S3/GCS); 5% on high-speed local NVMe.
• Latency SLA: Real-time tailing consumers must experience 0ms object storage read latency.
• Cost Reduction: Slashing broker block storage infrastructure costs by over 70%.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

1. The Two-Tier Storage Hierarchy

Kafka Tiered Storage decouples the compute/ingest layer from the long-term storage layer:

• Hot Tier (Local NVMe): Keeps the active segment and the last 2 to 4 hours of closed segments. Real-time consumers read directly from memory (Linux OS PageCache) with zero disk reads and zero object store calls.
• Cold Tier (Object Storage): Closed, sealed segments are asynchronously copied to Amazon S3 in chunks of 512MB. Once uploaded and verified, the local copy on the broker's NVMe drive is purged.

2. Resolving the Compacted Topic Conundrum

Compacted topics present a unique architectural challenge for tiered storage:

• In a compacted topic, Kafka periodically removes older records when a newer record with the same key arrives.
• If you offload a segment to S3 before it is compacted, you end up storing obsolete, bloated records forever in S3, wasting money and breaking tombstone guarantees.

We enforce a strict lifecycle policy:

# Topic-level configuration for compacted tiered streams
cleanup.policy=compact,delete
remote.storage.enable=true
local.retention.bytes=10737418240 # 10 GB local retention
segment.bytes=536870912          # 512 MB segment roll
min.compaction.lag.ms=86400000    # 24 Hours minimum wait before compacting
max.compaction.lag.ms=172800000   # 48 Hours maximum compaction deadline

By ensuring that segments are only tiered after passing the min.compaction.lag.ms window, the local broker compacts the data first on high-speed NVMe, and uploads only the deduplicated, finalized segment state to S3.

3. Asymmetric Fetching for Cold Replays

When an audit replay consumer requests offsets from 60 days ago:

• The broker does not download the entire 512MB segment back onto its local disk.
• It leverages HTTP Range Requests against S3, streaming only the requested offset byte slices directly to the client socket via zero-copy Netty byte buffers.

The "Baigan" Take (Expert Insights)

Do you know what happens when you treat Kafka like a database dump?

Many teams store 2 years of raw, un-compacted logs on expensive EBS volumes because: "Who knows, compliance might ask for it." That is baigan ka kharch (senseless expenditure).

Tiered storage gives you the best of both worlds: your Kafka brokers run lean with small, ultra-fast 200GB local drives that restart in 15 seconds, while your multi-petabyte historical archives sit safely in S3 at 1/10th the cost. But never tier uncompacted logs on topics that require key-based deduplication; always let local compaction finalize the segments before letting them cross the wire to object storage.

Performance Checklist

• Configure remote.storage.enable = true on high-retention transactional topics.
• Size local broker storage to hold at least 24 hours of hot log segments.
• Enforce segment compaction prior to tiered cloud offloading on compacted topics.
• Utilize S3 HTTP Range Requests for cold historical reads to prevent broker disk bloat.
• Monitor RemoteCopyLagBytes and RemoteLogManager upload metrics across broker clusters.