Skip to content
FocalIDFocalID

Batch or streaming: how fresh does your data need to be?

“Real-time” sounds better than “overnight”, so it ends up in many requirements without anyone asking what it costs. Batch and streaming are two ways of processing data, and the right one depends on how quickly someone, or something, acts on the result.

By the FocalID team in Colombo, , 2 min read

Water flowing over mossy rocks in a forest stream.

The two approaches

Batch processing collects data and handles it in groups on a schedule: every night, every hour. Streaming processes each event as it arrives, within seconds.

BatchStreaming
Data is readyAfter the next scheduled runSeconds after the event
Cost and effortLower, simpler to build and runHigher, more moving parts to monitor
Fixing mistakesRerun the batchHarder: events must be replayed in order
Typical usesFinancial reports, payroll, daily sales summariesFraud checks, live vehicle tracking, stock shown to online shoppers

Ask how fast anyone acts

The useful question is not “how fresh can the data be?” but “how quickly does anyone act on it?” If a manager reads the sales report once each morning, data that updates every second changes nothing. If a card payment must be blocked before it completes, a nightly batch is useless.

Go through each use of the data and write down the slowest update that would still be acceptable. Many turn out to be daily, some hourly and a few truly immediate.

The middle ground

Between nightly and instant sits a practical option: small batches every few minutes. They are built and repaired like ordinary batches, yet feel live to most users. For dashboards and operational screens this is often enough.

Use both where needed

Most companies end up with a mix. Streaming handles the few flows where seconds matter, and batch does everything else. Start with batch, measure whether anyone is waiting on the data, and move a flow to streaming only when the delay has a real cost.

Common questions

Is streaming always better than batch?

No. It costs more to build and run. It is better only where acting within seconds has real value.

Can we change from batch to streaming later?

Yes. A well-built batch pipeline defines the sources, rules and checks, and that work carries over.

Do AI agents need streaming data?

An agent needs data as fresh as the decision it makes. Confirming stock for an order needs current figures; a weekly summary does not.

Related