An alert nobody saw
The platform stored production logs, traces and metrics in SigNoz, which runs on ClickHouse with ZooKeeper for coordination. Data was spread by hash across two shards.
A volume-almost-full alert had been firing for days, but it reached nobody. It only surfaced because, in another piece of work, I had rewired alert delivery for that environment. When I looked, the problem was much bigger than a full disk: one of the two shards hadn't written anything for more than 50 days. Since distribution was by hash, about half of all production logs, traces and metrics were being silently thrown away.
Measure before deleting
The reflex here is to raise retention or delete old data. Before anything, I inventoried the disk per ClickHouse database. Customer telemetry took about 241 MiB, or 0.12% of the disk. The other 186 GiB were ZooKeeper's internal operations log, a table inside the database itself growing out of control.
With authorization, I expanded the shard's volume from 200 to 400 GiB with the pod running, no restart. Usage dropped from 100% to 50% and the shard started writing again immediately. I recorded what couldn't be undone: the 50 days of data on that shard were gone.
The cause of the cause
Shortly after came another critical incident. The NOC cleared the table, and in less than 10 hours it was back to 85 GiB. That wasn't accumulation; it was an open tap. I rebuilt the whole chain.
The telemetry collector inserted in micro-batches, flushing every 200 ms and never by size. That created about 65 thousand new parts per hour in ClickHouse, and each part generated dozens of ZooKeeper operations: around 15 million operations per hour per pod. Each operation wrote two rows to the internal log, close to 9 GiB per hour. The table's TTL was 30 days, but the disk filled in two.
To make it worse, ZooKeeper ran with its heap equal to the container's memory limit, with no headroom at all. Under pressure, the liveness probe failed and the pod was killed, hundreds of times. On every crash, ClickHouse replayed operations in bulk and fed the cycle again.
I proposed the fixes in order of impact and risk. First, widen the flush window, the strongest and safest lever. Then shorten the internal log's TTL in the configuration file that was already versioned, give ZooKeeper memory headroom and loosen the probe. Finally, as the definitive fix, migrate to ClickHouse Keeper, which was already running stable in the same cluster.
Watching the watcher
ClickHouse was the biggest CPU consumer in the cluster and the only large component with no metrics at all. I found the exporters were already there and nobody had scraped them for 160 days. Creating the ServiceMonitors was enough.
I built monitoring with the USE method (utilization, saturation and errors) for each ClickHouse resource, with thresholds taken from a baseline measured in the environment itself. On the first read of the errors side, 57 entries stuck in the replication queue showed up, some more than 100 days old, causing about 142 thousand fetch failures a day. They were parts lost in the disk-full incidents that ClickHouse kept trying to recover in a loop.
I also documented a trap: the replication delay metric read 124 days on every table, but that was only the age of the oldest stuck entry. An alert on it would fire forever with nothing anyone could do.
What changed
The shard started writing again and the loss of half the telemetry stopped, after 50.6 days. The full causal chain was documented, with an ordered fix plan. ClickHouse and ZooKeeper went from no metrics to monitoring with measured thresholds, which on its first read already exposed a problem hidden for months.
What I take from it
When a database disk fills up, the first question is what's taking the space, not what to delete. Here, "not enough retention" was a symptom: the cause was in ZooKeeper and in how data was being inserted.
And observability needs to be observed too. The tool that should warn about problems was losing half its data, and the only alert about it reached nobody.