Monitored on paper
The products had Prometheus, Grafana and alert rules. On any checklist, they'd be ticked as monitored. What made me suspicious was a public API that was down for ten minutes, and the one who noticed was an external monitor, not ours.
I decided to follow the full path of an alert, from the metric to a person receiving the message. In several clusters, Alertmanager's root route pointed to a null receiver: the alert was evaluated and thrown away. In one of them, 36 active alerts were going into the void. A message router silently dropped everything from one product because its name prefix had a dot, and a channel sat empty for 113 days without anyone noticing. On top of that, the rules were almost all about saturation, like CPU, disk and connections. A database, a queue or a cache could go down without opening any incident.
Measuring what the customer uses
I started with what the customer feels. I deployed the Blackbox Exporter to probe the public API every 30 seconds, with a critical alert after two minutes down and a route straight to the NOC. The manifests were versioned and the webhook token kept in a Secret, out of Git.
On a product where the critical service is the instances' connection to the messaging channel, there was no ready-made metric. I built the signal from logs: a Grafana Alloy pipeline that collects only connection and disconnection events, dropping about 99% of the volume at the source, and a rule that detects mass disconnection.
Down rules, carefully
I wrote unavailability rules for PostgreSQL, Redis, Dragonfly, RabbitMQ, Elasticsearch, MongoDB and Kafka across the products, set up so that a metric disappearing also counts as an outage. Before turning any of them on, I measured each signal's 24-hour coverage and the flapping over the previous five days, so the new rules wouldn't become noise.
In one of them, the math didn't add up. The Kafka broker alert waited for three brokers down, but the cluster had five and an internal topic with replication factor 1. A single broker down was already an incident. I fixed the threshold and versioned the rules in the product's Git repository.
Proving it arrives
I rewrote Alertmanager routing with separate destinations for the NOC, the product channel and the team chat, with critical alerts going to the NOC and known noise contained on purpose. Before applying it, I tested the routing matrix with amtool across 12 cases, and afterwards checked the operator log.
Along the way, I fixed details that killed delivery without any error: the dotted prefix, a grouping that left the alert name empty for the router, and a route that required a label with a trailing space.
The final proof was synthetic alerts: 38 firings across four products, 38 delivered to the chat and the product channel, and the count matching one to one at the NOC. In the next audit I still found a regression: a webhook I had validated days earlier no longer existed in the NOC gateway. It was documented and fixed.
Finally, I removed a recurring false positive (a rule that, with no data, kept its last state and stayed stuck firing after the problem had passed) and standardized every rule name with the product prefix across four clusters, by script, keeping the routing labels intact.
What changed
Any core database, queue or cache going down now opens an incident, which didn't happen before on several products. Alerts that were being discarded now reach three destinations, with delivery proven by testing. The public API got its own availability monitoring.
What I take from it
An alert only exists when someone receives it. Writing the rule is the easy part; monitoring is the whole path to a person, tested. That's why none of my routing changes were validated with "the config looks right".
And a saturation rule doesn't replace a down rule. A database at 50% CPU is fine. A database that disappeared isn't saturated either.