Data latency, the time between a data point being emitted and it becoming available for analytics, has long been a vanity metric for data platforms. It has pushed data platform teams toward streaming and near-real-time architectures in design meetings, justifying the choice by citing streaming as best practice. With the rise of AI, this rhetoric has grown stronger, arguing that agents shouldn’t rely on day-old data to make decisions. While compelling in principle, such an architecture proposal should prompt a harder question: “Are you sure the use case actually needs real-time?”
When streaming is the right answer
Streaming genuinely enables valuable use cases.
Customer engagement improves when systems react to events quickly. For example, when a customer performs actions on a mobile app, it triggers events. These events can be fed into a personalization engine to update views or to trigger notifications to increase conversion. Running the pipeline once a day would be too late. For example, it would be poor timing for your recommendation engine to suggest “item A” to a customer a few hours after this customer already purchased it. And when this recommendation is made, there is no guarantee that the customer is still online or that they will come back.
Resource constraints can also motivate building streaming pipelines. For example, when building a data lake, you may want to ingest data from a database with a continuous high load or strict latency requirements, making bulk extraction too expensive or disruptive. That’s when Change Data Capture (CDC) pipelines become an interesting alternative.
Streaming does solve problems, but it is not a bulletproof answer. In the two examples I’ve just mentioned, batch-based recommendation engines are often acceptable depending on latency requirements and business context. And not many companies face such a high load that bulk extraction can’t be implemented. So the question is whether the problem you’re solving requires streaming.
Where the myth comes from
Big Tech companies provide impressive content on their engineering blogs. The architectures that Netflix or Uber built are real and impressive, but they are built for a specific scale and specific engineering organizations. Before replicating them, engineers should carefully consider their own context.
Saying that business analysts need real-time data sounds compelling. It becomes harder to defend when someone asks: what decision requires this, and when would 30-minute-old data actually be too late? This also applies to AI Agents. We often hear they need real-time data, but for what decisions? For many use cases, nobody can answer clearly. The real-time requirement is often assumed rather than derived. Engineers bake it into the architecture before anyone has validated the business need.
Another common thought is that starting with a streaming architecture is necessary to handle future requirements without having to redesign the entire system later. In reality, these requirements rarely come. By the time a genuine real-time requirement emerges, the system will likely need refactoring regardless. I’ve also never seen a team ditch their batch data platform because of a new streaming requirement. Instead, they build a new module on top of what already exists.
The cost of getting it wrong
Streaming pipelines solve real problems. They also introduce operational complexity that most teams underestimate.
First, the cost of building streaming infrastructure is often higher than expected, especially if the stack isn’t already in place. Kafka, Flink, Spark, Kafka Connect, Schema registry, and other tools take time to learn and to configure correctly. All these clusters require compute and tight networking. It’s not as simple as taking the Docker image or the Helm Chart from the open-source repository. You also need to integrate them with the monitoring stack, build deployment pipelines, integrate with a secret manager, and consider security, for example. In reality, data platform engineers deploy them quickly to enable the platform, but then spend months optimizing and strengthening them.
After a few months, the operational complexity surfaces. The running costs quickly rise as more pipelines are added. The first incidents related to data schema changes, CDC issues, and other pipeline errors appear. The observability tooling isn't yet mature enough to diagnose the incidents they're now facing. And you can observe the first source of friction with the source system owners who are reluctant to configure their systems to support streams, or with application teams that modify or deprecate streams without knowing the analytics platform depends on them. Each of these problems is solvable, but it's harder than using a batch pipeline when that’s suitable.
In the long term, knowledge of the real-time data platform is concentrated within the data platform team. The expertise accumulates in a small group of experts who are always called upon for every streaming incident. Because other engineers learn to avoid touching the streaming infrastructure, nobody else knows what is failing, the platform or the pipeline. Over time, this pool of people shrinks, making hiring for these specific skills more difficult and expensive. That creates a bottleneck, which becomes a real organizational problem if all the data pipelines are streaming pipelines.
These hidden reliability taxes compound quietly but are never reflected in the original architectural decisions.
When to choose streaming or batch
Before choosing streaming, ask how the data will be used in real-time. Not in theory, or a potential use case in six months. Be specific. What decision, by whom, at what maximum latency, and with what consequence if it’s 30 minutes stale? Also, ask about the return on investment. Compared to a batch pipeline that refreshes every 30 minutes, what does streaming add vs. what it costs?
As mentioned earlier, be careful if you end up answering “maybe we’ll need it in the future”. And if the decision is being driven from above rather than from the use case, make the costs explicit: visible and invisible.
Batch pipelines are operationally safer when well designed, and the operational stress on engineers is lower. I’ve always been much more worried when I see a streaming pipeline alert than when I see an Airflow alert. With batch, failures are recoverable, and re-triggering is straightforward. With streaming, it's always more challenging, and the fear of missing data is always present. Most BI requirements are batch in principle, even when they feel urgent. For example, a streaming data platform I worked with streamed data into the warehouse, but all the data transformations for the dashboards ran in batch mode every 30 minutes. Because 30 minutes satisfies most business users who think they need real-time, and is much cheaper.
About your next design decision
When designing your next data platform, data pipelines, or AI system, think about the business needs first. Don't jump to a solution before clearly understanding the requirement. Streaming can give your organization a real edge, but make sure that edge is real before accepting the operational cost. Batch is more boring. It's also easier to build reliability into and to maintain. And if you end up picking the streaming path, document the trade-off you've made. So that next time someone asks, “Are you sure?” you have an answer.
