The night
In the middle of the night, the ZFS storage pool on one of the Proxmox hypervisors reached 99.8% usage and took more than 14 nodes of the production Kubernetes cluster down with it. The incident lasted about 14 hours, until the last new nodes, built on another hypervisor, became available.
The obvious explanation was that the data had grown. It was right, but incomplete, and a post-mortem that stops at the obvious explanation doesn't prevent the next outage.
The pool was promised four times over
I started with the Proxmox API, read-only, listing the nodes, the storage and each VM's disks. There were 62 TB provisioned on a 14 TB physical pool, a 4.3-to-1 overcommit. The four database VMs alone had 6.5 TB each, thin provisioned. And the hypervisor had no backup, no replication and no capacity monitoring.
I rebuilt the timeline of the outage and recovery from the Kubernetes nodes' own transition events, and cross-checked it with the other four incidents of the day. An API availability alert was legitimate, because the API really was degraded. Another might be linked to a service running on a node that went down. A third had nothing to do with it: it was a permissions issue.
Zombie blocks
The actual data didn't explain how fast the pool filled, so I dug deeper. TRIM was off on all 47 volumes of the Kubernetes distributed storage, Longhorn. MongoDB frees blocks at every checkpoint and tells the disk, but Longhorn ignored the notice, and ZFS never got the space back.
That had been piling up for more than 500 days. On each MongoDB replica, the filesystem showed about 437 GB while the volume took up 1.1 TB. In MongoDB alone there were some 4.8 TB of dead space, multiplied by Longhorn's three replicas. Several volumes were already above their provisioned size, up to 135%. And MongoDB had no TTL: date fields were stored as integers, so the native TTL index would never work without changing the application.
A mistake of mine, corrected out loud
In the middle of the incident, looking at one of the day's alerts, my first read was that the rule was misconfigured. When I opened the full rule, I saw it was right and the API really was down. I corrected the record on the spot. The real problem was something else: the public status page said "all systems operational" while the API was returning 503.
The post-mortem
The root cause was recorded as the combination of three decisions: a 4.3-to-1 overcommit, TRIM off with three replicas per volume, and data with no expiration. I listed nine process failures, five of them P0 or P1, such as no capacity alert on the hypervisor and no backup or replication.
The highest-impact fix didn't depend on the development team: enable TRIM on the volumes and run fstrim, with an estimated 3 to 5 TB recovered, plus fixing the over-limit volumes and creating the ZFS capacity alert.
What changed
The root cause went past "the data grew" and reached the mechanism that made the space disappear. The action plan was prioritized, with the most important fix within the infrastructure team's own reach, and the nine process failures were recorded so they wouldn't repeat.
What I take from it
The obvious explanation is rarely the whole root cause. "The disk filled up because the data grew" was true, but 4.8 TB of dead space wasn't data. It came from three decisions that were reasonable on their own and that together made a time bomb.
And in the middle of an incident, correcting a wrong read out loud is worth more than defending it.