A data test nobody acts on is worse than no test at all
Most test suites fail quietly, by being read less each week. The useful cut is not which tests are correct but which ones have a named owner and a first move.
The pipeline alerts channel is the honest record of a data team's relationship with its own tests. Open one that has been running for two years and you can usually date the moment it stopped working. There is a stretch near the beginning where every message has a reply under it, somebody saying they are looking, somebody else saying it was a bad file from the source and it has been reloaded. Then the replies thin out. Then they stop. The messages keep arriving.
By that point the channel usually carries three or four familiar failures on every run. A not_null test on a column that is genuinely null for one class of customer, written before that class of customer existed. A freshness check on a source that lands late every Monday because of a batch window in a system nobody on the team controls. An accepted_values test that fires whenever sales operations adds an option to a pick list. Every one of those tests is correct. None of them is a bug. And together they have taught everyone who reads the channel that the right response to a red message is to scroll past it.
That is the cost people underrate. A suite that fires and is ignored does not leave you where you were before you wrote it. It leaves you worse off, because you now have a documented control that an auditor, a regulator or your own finance director will reasonably assume somebody is watching, and because the one genuine failure will arrive in exactly the same colour as the four you have trained yourself to dismiss.
Correct and actionable are different properties
A test encodes an assertion about the data. A failure is a claim that the world has moved away from that assertion. Teams argue about whether the assertion is true, which is the easy question, and skip the one that actually decides where the result should go: if this fires at three in the morning, what does a named person do about it before breakfast.
There are only three honest answers. Someone intervenes immediately, because a downstream consumer is about to read the table and a wrong number will reach a person who acts on it. Someone intervenes during the working day, because the problem is real but the blast radius is a day of slightly stale reporting. Or nobody intervenes at all, because the failure describes a known property of the source that the business has quietly decided to live with. The third answer is not shameful. It is common and it is often correct. It just means the check belongs in a weekly record, not in a channel that pings phones.
A test earns its place in an alert channel by having an owner and a first move, not by being true.
The distinction, stated plainly
Three questions to run over an existing suite
- Who is named A person, not a team, with somebody behind them for the days they are away. A test owned by "data engineering" is owned by nobody at eight in the morning.
- What is the first move Written down before the failure happens, in one line. Rerun the model. Hold the load and tell the controller the dashboard is frozen. Call the operations lead at the supplier. If nobody can write that line, the test is not ready to page anyone.
- What breaks if nobody looks until Thursday This is the question that does the pruning. If the honest answer is that nothing breaks, you have a number you would like to watch, not an incident you need to catch.
Severity is a routing decision, not a grade
Most frameworks already give you the machinery. dbt has warn and error on every test. Orchestrators such as Dagster let you decide whether a failed asset check stops the run, marks the asset degraded, or simply records a value against the materialisation. Teams set everything to error because warn feels like an admission that you are not going to fix something. That instinct is the whole problem. Warn is not a weaker error. It is a different destination.
Blocking checks stop the load and leave the last good version of the table in place, so the consumer sees yesterday's number instead of today's wrong one. Notifying checks let the data through and put a message in front of a named owner. Recording checks write their result to a table and never speak. Most suites should be weighted heavily towards the third lane, and almost every suite that has gone quiet is weighted towards the first two.
Put the loud tests at the boundaries
The place where an assertion is worth waking somebody for is where data crosses a line of ownership. Inside your own project, a not_null on a staging model is largely a test of your own SQL, and its right home is continuous integration, where it blocks a merge and pages no one. The same test on a column arriving from another team's application is a different animal. It is a check on a relationship, and there is a person on the other side of it who can change the behaviour. Boundary tests also tend to outlive refactoring, which is one reason internal suites accumulate dead assertions that no longer describe anything anyone intended.
- 01Pull the failure historyTest results should be stored, not just printed to a log. dbt writes run results you can load; a single table in Postgres or your warehouse with test name, timestamp and status is enough. Without a history you are negotiating from memory, and the loudest opinion in the room wins.
- 02Rank by how often each test firesThe tests at the top of that list are the ones teaching people to ignore the channel. Deal with them first, and deal with them by making a decision rather than by nudging a threshold until the noise stops.
- 03Assign or deleteEvery test that stays in an alerting lane gets a named owner and a written first move. Anything nobody will claim is demoted to the recording lane or removed. Deleting a test is a legitimate outcome, and the reason belongs in the pull request so the next person does not reinstate it.
- 04Give the quiet lane somewhere to be readIf recorded results go nowhere, you have only moved the problem. A short weekly review of trends, read by one person who has the standing to escalate, does more for data quality than a channel of red messages nobody opens.
Where this advice is wrong
Two situations. The first is a genuinely regulated pipeline, where the fact that a check ran and was evidenced matters on its own, whether or not anyone reacted on the day. There you keep everything, but you still route it, because an auditor reading a channel of ignored alerts draws precisely the conclusion you were hoping to avoid.
The second is more awkward. Sometimes an alert channel is dead not because the tests are badly designed but because nobody owns the data at all. Every failure is somebody else's problem and there is no forum where that gets settled. No tool repairs that. A data quality platform bought into that situation gives you a better-rendered version of the same wallpaper, and the money would have gone further on one afternoon a fortnight where the people who own the source systems and the people who own the reports sit in the same room. It is the same reason we push governance onto the query rather than onto copies of the data: a control only works when it is attached to somebody's actual job.
The test you can delete without an argument was never doing anything. The test somebody defends is worth keeping, and the person defending it has just told you who owns it. That conversation, run once a quarter over the failure history, is most of data quality practice. The rest is choosing the right lane.