3 a.m., one alert and no clues
When an alert fired at night, the on-call engineer got a message with the rule name and nothing else. To investigate, they had to work out alone which dashboard to open, which panel to look at and whether anyone had deployed around that time.
I audited the six production environments and counted 871 active alert rules. Only 40 of them, 4.6%, led to any panel. And when I opened those few links, most were wrong. They pointed to gauge panels, which show the current value with no curve and no event markers. In one case, the database disk alert led to the database size panel, which is a different quantity. Whoever clicked would investigate the wrong number, and confidently.
No product had a RED view of its services (rate, errors and duration), and no Grafana showed when a deploy had happened.
Every alert points to its own condition
I mapped rule by rule by the metric it evaluates, not by the panel name. I created the missing time series panels and a full dashboard for object storage, with 18 panels on health, capacity, traffic and latency. I linked each rule to the exact panel and repeated the link in the alert description, because the description is what shows up in the on-call message.
So this wouldn't fall apart over time, I treated panel numbering as a contract, so editing a dashboard doesn't break links already published, and wrote a script that checks that every alert link points to a panel that exists. The rules went into Git and through promtool before being applied.
RED, fast
I built RED dashboards for four products. Where there was application instrumentation, I used SigNoz, from traces. Where only the edge view existed, I used ingress-nginx metrics in Prometheus.
The ingress's raw latency histogram had 238,800 series, and Prometheus refused the query over a six-hour window. I created recording rules that pre-aggregate per service and bring it down to 228 series. The dashboard's 12 queries now answer in 0.05 seconds.
On every panel I wrote a description of what it measures and how to read the number, with non-specialists in mind. On their very first read, the dashboards showed things nobody was seeing, such as a service with almost half its responses returning 5xx errors. On one product I also built the queues panel and checked the metrics against the broker's real offsets: a 1% difference.
The deploy's vertical line
The first question in any incident is "did anything change?". I configured ArgoCD Notifications to publish every deploy as an annotation in Grafana, and every chart started showing a vertical line at the moment a version went out.
Two traps showed up before applying it, and I measured both. The Grafana token had to live in a Secret outside the Helm release, because a helm upgrade would rewrite the Secret and wipe the token without any error. And in multi-source applications, the revision field from the official example came back empty, which would make the notification fire once and never again. I used the deploy's completion time as the key. The backfill published old deploys at the exact time, checked to the second.
What changed
Alerts now take the on-call engineer straight to the chart of the condition that fired, for example 39 of 39 rules in one environment and 12 of 12 in another, with links validated by script. The products gained a RED view per service, fast even over long windows, and charts now show when a deploy happened.
What I take from it
A dashboard only helps if it answers the question the alert asked. Linking an alert to the "closest-looking" panel is worse than not linking it, because it sends the person to the wrong number. And a good panel teaches: the description on each chart is documentation for whoever is on call at 3 a.m.