Ingesting Line-Rate Modem Telemetry: Ring Buffers, Lossless Downsampling, and Surviving a Noisy Link

The Context: "SNR Plots That Lied"

Fresnel is my modem telemetry suite — it collects signal-quality samples from a broadband link: SNR, attenuation, sync rate, RTT, and negotiated PHY rate. The goal was a live dashboard and, more importantly, a usable historical record so I could actually answer "was the line bad last Tuesday, or am I imagining it?"

The first version stored every sample in SQLite. One sample every second, a handful of metrics, several lines. That is nothing — roughly 300KB per day. It worked for three weeks.

Then I plugged in a second modem and started sampling at 10Hz during a speed test. The database grew to about 2.1GB in a fortnight, the dashboard's range selector took nine seconds to respond, and my own tooling could not answer a question about Tuesday because it had to scan the whole table to find Tuesday.

The data itself was fine. The architecture was wrong — I had built a database where a purpose-built time-series store belonged, and I had stored samples at a rate I never actually needed.

The Problem Statement

Capture and retain high-resolution link telemetry for long-horizon analysis, on a machine with a browser and a disk:

Ingest: Sustain 10Hz sampling across multiple modems with bounded memory, even when the consumer is slow or offline.
Retention: Keep 10Hz detail for 48 hours, 1Hz for 30 days, 1-minute rollups for two years — without a column store.
Query: Any metric, any time range, answer in under 100ms.
Honesty: A dropped sample must be recorded as dropped, never interpolated into looking like real data.

That last requirement turned out to be the one that shaped the entire design.

Visual Architecture

The Core Logic / Process

1. Segments, Not a Database Table

Storage is a directory of immutable, compressed segments rather than rows in a table. A segment is a time window of one metric on one line, written as append-only tsv.zst:

# seg: 20260914T093000Z modem=eth0 metric=snr
1757850000	41.2	GOOD
1757850001	41.0	GOOD
1757850002	NA	NO_SYNC

Segments are the single most important decision in this system. They make writes purely sequential (the fastest thing a disk does), make retention a file deletion rather than a DELETE that reclaims rows one at a time, and make compaction a local operation instead of a global one.

The NA row on line three is the whole point, and I will return to it.

2. The Ring Buffer Between Producer and Consumer

Polling runs on a fixed schedule. Writing to disk does not. If the disk stalls for 200ms during an fsync, samples must not be dropped on the floor.

pub struct Ring {
    slots: [AtomicSample; RING_SIZE],  // 65536
    head: AtomicU64,                  // monotonic write cursor
}

impl Ring {
    pub fn push(&self, s: Sample) {
        let idx = (self.head.fetch_add(1, Relaxed) & (RING_SIZE - 1)) as usize;
        // write the slot, then publish the cursor — never the reverse
        self.slots[idx].store(s, Release);
    }
}

Single-writer, single-consumer, power-of-two sizing with a bitmask instead of modulo, Release/Acquire ordering so a consumer never reads a half-written slot. At 10Hz the 65,536-slot ring holds roughly 1.8 hours of headroom — far beyond any plausible disk stall, and bounded at a few megabytes.

The dashboard reads the ring tail directly, so the live view never touches disk at all. It stays smooth even while a segment is being fsynced.

3. Tiered Downsampling

Retention at every original rate is unaffordable, so I roll up progressively. Each tier preserves a strictly smaller window than the one before:

Tier	Resolution	Retention	Approx. size
Raw segments	10Hz	48 hours	~90 MB
Rollup 1	1Hz	30 days	~110 MB
Rollup 2	1min (min/max/mean/p99)	2 years	~35 MB

Two years of one-minute data costs about the same as two days of raw. The rollups are append-only too, and the sweep deletes whole tiers on schedule.

The subtlety is the aggregation choice. Mean alone hides every excursion, and SNR spikes are precisely what you are hunting. Every one-minute bucket keeps min, max, mean, and p99 — a bucket where min and max diverge sharply is the signal that something happened, and a mean-only store would have silently smoothed it away.

4. Gaps Are Data

A modem that loses sync does not report low SNR. It reports nothing. The polling layer distinguishes three states, and this is non-negotiable:

enum Quality {
    Good,     // modem responded, value is real
    NoSync,   // modem reachable, no bearer — record as NA
    Unreachable, // modem did not respond — record as NA
}

Every sample is written, including the ones with no value. A missing row is ambiguous — was the modem fine and the writer asleep, or was the modem down? An explicit NA row with a reason answers that question permanently.

This matters enormously the first time you debug a speed test. I lost an afternoon chasing a phantom "sudden SNR drop" that was actually a 4-second logging gap during a firmware update. Interpolating across the gap would have produced a smooth, confident, completely fictional dip on the chart.

The "Baigan" Take (Expert Insights)

The honest observation from this rebuild: the first version's only real sin was not the storage engine. It was storing data at a resolution I never used and dropping the information I did.

baigan ka telemetry is the belief that "just log everything, you can aggregate later." Aggregation is lossy and frequently irreversible. If you do not store the p99 at write time, no amount of querying recovers it. Decide your retention tiers before you have a million samples and an opinion about what matters.

Three lessons from the rebuild:

Append-only segments beat table rows for any time-series workload, and not mainly for speed. Retention becomes rm. Compaction becomes local. A corrupt write damages one file, not one row, and the other files remain readable. When my disk filled up mid-segment, the partial file was discarded and 48 hours of telemetry stayed intact.
Explicit gap markers are worth more than perfect uptime. Every "smooth" telemetry chart I have built hid a real outage behind an interpolation. Now every chart has visible gaps, and I trust the charts.
Retention tiers should be derived from the question, not the sample rate. I originally kept 10Hz "because we might want detail." We never wanted detail beyond 48 hours. Fourteen gigabytes and nine-second queries later, the answer was obvious.

The meta-lesson: telemetry systems fail at the schema level, not the ingestion level. Getting 10Hz samples written reliably took an afternoon. Deciding that one-minute buckets need p99 and that gaps must be first-class records took a rewrite.

Configuration