Batch or streaming: how fresh does your data need to be?
“Real-time” sounds better than “overnight”, so it ends up in many requirements without anyone asking what it costs. Batch and streaming are two ways of processing data, and the right one depends on how quickly someone, or something, acts on the result.
By the FocalID team in Colombo, , 2 min read

The two approaches
Batch processing collects data and handles it in groups on a schedule: every night, every hour. Streaming processes each event as it arrives, within seconds.
| Batch | Streaming | |
|---|---|---|
| Data is ready | After the next scheduled run | Seconds after the event |
| Cost and effort | Lower, simpler to build and run | Higher, more moving parts to monitor |
| Fixing mistakes | Rerun the batch | Harder: events must be replayed in order |
| Typical uses | Financial reports, payroll, daily sales summaries | Fraud checks, live vehicle tracking, stock shown to online shoppers |
Ask how fast anyone acts
The useful question is not “how fresh can the data be?” but “how quickly does anyone act on it?” If a manager reads the sales report once each morning, data that updates every second changes nothing. If a card payment must be blocked before it completes, a nightly batch is useless.
Go through each use of the data and write down the slowest update that would still be acceptable. Many turn out to be daily, some hourly and a few truly immediate.
The middle ground
Between nightly and instant sits a practical option: small batches every few minutes. They are built and repaired like ordinary batches, yet feel live to most users. For dashboards and operational screens this is often enough.
Use both where needed
Most companies end up with a mix. Streaming handles the few flows where seconds matter, and batch does everything else. Start with batch, measure whether anyone is waiting on the data, and move a flow to streaming only when the delay has a real cost.
Common questions
Is streaming always better than batch?
No. It costs more to build and run. It is better only where acting within seconds has real value.
Can we change from batch to streaming later?
Yes. A well-built batch pipeline defines the sources, rules and checks, and that work carries over.
Do AI agents need streaming data?
An agent needs data as fresh as the decision it makes. Confirming stock for an order needs current figures; a weekly summary does not.

