Most real-time requirements are a deadline in disguise

A logistics operations lead asked us for real-time visibility on shipment exceptions. We asked what would happen if a shipment went into exception at 09:14 and the dashboard showed it at 09:20. Nothing, he said. The exceptions desk works a queue. What actually mattered was that the 11am carrier call started with a list that was complete as of 10:30, because a shipment missing from that list did not get chased that day.

That is not a real-time requirement. That is a deadline. The number has to be right and present before a particular moment, and the moment repeats on a known schedule. Those two problems look similar on a whiteboard and cost wildly different amounts to run.

We have had this conversation often enough that it is now the first thing we test in a Week 1 audit. Someone writes down real-time, we go and find the meeting, the cut-off, the SLA clock or the customer-facing screen that the phrase is standing in for. Most of the time there is one. When there is, the honest answer is a well-scheduled batch, and we say so even though a streaming build is a larger piece of work for us.

Ask when the number is looked at, not how fresh it should be

Ask a stakeholder how fresh they want the data and the answer is always as fresh as possible, because nobody has ever been punished for wanting more. It is the wrong question. Freshness is free in the asking and expensive in the running, so the request carries no information.

The useful questions are about the moment of use. Who looks at this number, and when. What decision changes because of it. If the number were fifteen minutes stale at that moment, what would go wrong, concretely, to whom. If the answer is that a person would make a different call, you may have found a genuine latency requirement. If the answer is that it would feel wrong, or that finance would query it, you have found a correctness or trust problem wearing a latency costume.

Freshness is free in the asking and expensive in the running, which is why asking how fresh people want the data tells you nothing.

The pattern, stated plainly

There is a second question worth asking, and it is the one that most often ends the discussion. How long does it take the human to act once they see the number. If an exception takes four hours to resolve through a carrier, sub-minute detection buys you nothing. The data pipeline is rarely the slowest link in the chain, and speeding up the fastest link is a strange place to spend a quarter.

What a stream actually costs after go-live

The build cost of streaming is not the problem. Standing up CDC off a production database, pushing changes into Kafka and landing them in a warehouse is well-trodden work with good tooling. Any competent team can get a demo of it running quickly. The cost is everything that comes after.

A batch job that fails at 03:00 is a job you rerun at 08:00. The blast radius is one run and the fix is idempotent because you designed it that way. A stream that breaks at 03:00 is a position in a log, a consumer group that is now behind, and a set of downstream tables holding partial state that nobody wants to explain. Recovery is a judgement call made by whoever is awake.

• Ordering and duplicates become your problem Delivery guarantees are usually at-least-once, so every consumer needs to be idempotent and every join needs to tolerate arriving out of order. In batch, ordering is a property of the run.
• Late and out-of-order events need a policy A record arrives an hour late. Does it update yesterday's figure, land in today's, or get dropped. Someone has to decide, write it down, and defend it to finance later.
• Schema changes stop being polite An upstream column type change breaks a batch model at the next run, in daylight. It breaks a stream immediately, quietly, in the middle of the night.
• Reconciliation becomes permanent work Once numbers move continuously, you need a scheduled check that the streaming totals agree with the source of truth, plus a process for when they do not. That is a batch job you now run alongside your stream.
• On-call is real Streams need somebody who can read consumer lag at midnight. If the team cannot staff that rota honestly, the stream will degrade into a batch pipeline with worse failure modes.

None of that is an argument that streaming is bad. It is an argument that streaming has a standing operational cost that has to be paid every week, forever, and that cost should be weighed against a benefit someone can name. When the named benefit is that a dashboard feels live, the trade is a poor one.

How to make batch meet the deadline

If the requirement is a deadline, then the engineering goal is not speed, it is a completion time you can promise. That is a different design conversation and a more productive one.

Done properly this covers a surprising amount of ground. An hourly incremental model with a hard completion guarantee, alerting tied to the business deadline and a visible freshness stamp will satisfy most of what gets requested as real-time, and a junior engineer can debug it at nine in the morning with a coffee.

When streaming is the honest answer

There are cases where batch is the wrong tool and we will say so just as directly. The test is whether the latency is inside a machine loop rather than a human one.

• The decision is automated and time-boxed Fraud scoring at checkout, dynamic routing, holding or releasing an order. Software acts within seconds and no human is in the path. A schedule cannot serve this.
• The freshness is the product A customer-facing tracking page, a live status board sold as part of the service. Here the perceived liveness is what the customer is paying for, and that is a legitimate requirement even though it is a felt one.
• The event is the only record Clickstream, sensor telemetry, application events with no queryable system of record behind them. If you do not capture the event when it happens, it is gone, and a log-based transport is the right shape regardless of how often anyone queries it.
• Volume makes full extraction impractical Some sources cannot be re-read on a schedule without hurting the production system. Change data capture is often the gentlest option available, and it is worth noting that CDC into an hourly-loaded warehouse is a perfectly reasonable design. Streaming ingestion does not oblige you to stream everything downstream.

That last point is the one that gets missed most often. The transport and the consumption cadence are separable decisions. You can capture changes continuously, land them cheaply, and still model and publish on a schedule that matches how the business actually works. Teams talk themselves into end-to-end streaming when they only needed the first hop.

What we do about it

On a Week 1 audit, when a latency requirement appears in the brief, we go looking for the deadline behind it before we cost anything. Usually we find a meeting, a cut-off or a customer screen, and the one-pager comes back recommending a schedule with a completion guarantee rather than the streaming build that was asked for. We have talked ourselves out of larger pieces of work this way more than once.

Sometimes we find the machine loop instead, and then we build the stream and say plainly what it will cost to run and who will need to be reachable at night. The difference is that by then it is a decision somebody made with the operational bill in front of them, rather than a phrase that survived from a kick-off slide into an architecture.