An alert nobody can act on is a subscription to worry
Alerting estates grow by addition and are almost never pruned. One question decides what to keep, and everything that fails it is a report, not an alert.
Somewhere in most companies there is a channel called something like alerts-prod. It carries a few hundred messages a day. Almost everyone with access has muted it, and the mute is not laziness. It is the only rational response available to a person who was being interrupted forty times a week by a condition they could not change.
The failure that follows is always the same shape. The one alert that genuinely needed a human arrives at two in the morning and lands between a disk usage warning that has been firing continuously since somebody set the threshold at eighty percent two years ago, and a job duration alert for a pipeline that was retired in the spring. Nobody reads it, because nobody reads any of them. The coverage was there. The attention was not.
This is not really a monitoring problem. Monitoring is the easy half, and every tool on the market does it competently. It is an inventory problem, and the specific defect is that no step in the process ever removes anything.
Alerts accumulate because nothing in the process removes them
Watch where alerts come from. Almost all of them are born in the hour after something went wrong. The incident review produces an action item that reads: add an alert for this. The threshold gets picked from the one bad week everyone still remembers, which is the least representative week in the history of the system. The alert ships, the incident closes, and the alert outlives the architecture that produced it.
Now look for the opposite action item. Nobody writes: remove the alert for that. Adding an alert is visible diligence, and it is free at the moment of writing. Removing one is a decision that a named person has to defend if it ever turns out to have been useful. So the estate only ever grows, and the cost of the growth is paid quietly, by degrading everyone's response to everything.
Adding an alert is free at the moment you write it. The whole cost arrives later, spread across every alert you already had.
The pattern, stated plainly
The test that decides what stays
There is one question worth applying to every rule in the estate, and it has to be asked in a specific form. Not: would somebody want to know this. Almost everything passes that. The form that works is: would a named person do something different in the next hour as a result of receiving this. Name the person. Name the something.
- No named owner It goes to a channel, a distribution list, or a rota nobody has updated. A group is not an owner. If the alert fires and two people each assume the other is looking at it, the alert has an owner of zero.
- No action inside the window It fires at three in the morning and the only available action is a call to a supplier whose office opens at nine. That is a real fact worth knowing at nine, not an interruption worth having at three.
- No change of state The condition persists, so the rule re-fires every fifteen minutes and re-states something already known. After the third repeat it stops being information and becomes weather.
- No reasoning behind the threshold Eighty percent, because it is a round number. Nobody can say what happens at seventy-nine, and nobody has looked at what the number actually does across a normal quarter.
Everything that fails the test is a report. That is not a demotion. Reports are genuinely useful, and a great deal of what people currently receive as alerts is good analysis arriving in a format that makes it impossible to think about. A report tolerates ambiguity, invites comparison, and can be read with a coffee. An alert has to survive being read on a phone by somebody walking to a car. Those are different products and they should not share a channel.
Making the survivors worth waking for
- 01Give it an owner and a fallbackOne name for the primary, one name for when the primary does not acknowledge inside an agreed window. Both names are people, and both are reviewed when either changes role.
- 02Give it a duration clauseFire on a condition sustained for ten or fifteen minutes rather than on a single sample. This removes most of the spikes that resolve themselves before anyone opens the laptop, and it costs you almost nothing in real detection time.
- 03Fire on transition, and send the resolveOne message when the condition starts, one when it ends. The resolve message matters more than people expect, because without it every alert stays psychologically open.
- 04Silence the windows you already know aboutDeploys, the month-end batch, the annual stock count. If you know the condition will be true on the first Monday of the month, an alert on the first Monday of the month is not evidence of anything.
- 05Give it an expiry dateEvery rule carries a review date at creation. On that date it is deleted unless somebody argues for keeping it. Defaulting to deletion is the only mechanism that has ever worked.
The body of the message deserves as much design attention as the rule. Assume the reader has been asleep. Say what fired, what the value is against what threshold, how long it has been true, what else changed nearby such as a deploy or a batch run, and where the runbook is. Without that, the first ten minutes of every response are spent working out whether the thing is real, and those ten minutes are how a person learns to ignore the channel.
Freshness and value are two different alerts
This one catches good teams. A retailer sets an alert for stock falling below the reorder point, defined as a condition on a chart. It works for a year. Then the pipeline that loads inventory movements fails one night, the table holds yesterday's snapshot, and the alert stays quiet because from the metric's point of view nothing has crossed a line. The alert did not fail. It answered the question it was asked, on the data it was given.
So you need two rules with two owners. The orchestrator knows whether the load landed, on time, with a plausible row count, and that alert belongs to whoever runs the platform. The metric layer knows whether the number looks wrong, and that alert belongs to whoever owns the business decision. Keep in mind that a rule defined on a dashboard chart is a rule defined on a query, and a query that silently returns no rows reads to most alerting engines as no breach. Absence has to be alerted on explicitly, because nothing else will do it for you.
When the answer is not another alert
Some of this we build. Mobile-first KPI alerting on Apache Superset is what Glimvia does, and reducing the count of rules is genuinely the first useful thing it does on an existing estate. But there are situations where more alerting, ours included, is the wrong recommendation and we would rather say so early. If the person receiving the alert has no authority to act without approval from somebody two levels up, the alert is a way of making one person feel the stress of a decision they cannot make. If the underlying problem is a supplier who is late every month, an alert converts a commercial problem into a nightly interruption. And if the decision is honestly a weekly one, a weekly report is the better instrument and the alerting project should be cancelled.
For everything that survives, run a review with numbers rather than opinions. For each rule over the last quarter: how many times it fired, how many times somebody acknowledged it, and how many times anything at all changed as a result. That third number is the one that matters. Where it is zero, the rule is deleted or demoted to a report, and the burden of proof sits with keeping rather than removing. This is uncomfortable in the room and it is the only part of the exercise that does any work.
The measure of an alerting estate is not how much it covers. It is whether anyone has the channel unmuted, and whether the person who does still looks up when it makes a sound.