Comparisons and explanations · Intermediate

Compare batch and stream processing for the same sales

Compare bounded input, first and final output, late events, correction and recovery effects under one shared requirement.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Compare when the same sales answer is needed
  2. Compare bounded inputs with retained streaming state
  3. Align late-data and correction requirements
  4. Prevent duplicate effects during either recovery style
  5. Compare operating cost under the same service objective
  6. Solve the one-second and next-morning requirements
  7. Ask Logeia about this articleCheck your understanding

Compare when the same sales answer is needed

Batch versus stream is not simply large versus small data. Here, batch processes an explicitly bounded input, while streaming updates state from continuing arrivals. Read stream-time material first. Use the same sales facts and compare first output, correction and finality. Large inputs can stream and small ones can be batched; latency and completeness requirements matter more than volume labels.

Use sales one hundred at 10:01, two hundred at 10:04, forty at 10:02 and four at 10:04. All four total 344, but the last two arrive late. A live display might move from 300 to 340 and later 344. A daily batch can produce one result from its cutoff set, yet still needs a policy for arrivals after that cutoff. Batch does not know every fact in the world automatically. Define the bounded input and its completeness claim explicitly.

Compare bounded inputs with retained streaming state

A batch input can be an identified file set or snapshot. Fixed input and code support reproducible recomputation, but mutable external reads can undermine that premise. A stream retains positions and state across arrivals and needs a restart boundary. Both approaches need code version and input identity; neither automatically creates reproducibility.

Micro-batching repeatedly processes finite slices while presenting continuous updates. The categories are therefore not always an exclusive binary choice. Inspect where input is bounded and output finalized. A few seconds of allowed latency may justify a simpler repeated finite job. A control requiring less delay needs verification that the actual scheduling and processing path meets it; shortening a nominal window alone does not prove that. Match the unit of execution to the required observation and recovery behavior.

Align late-data and correction requirements

Streaming can use event-time windows, progress and retained lateness state to revise 300 to 340. The still later four may require a side output and later reconciliation after cleanup. Batch must also decide whether post-cutoff data belongs in another run or a correction of an earlier day. Do not compare a fast incomplete pipeline with a complete one and attribute the whole difference to execution style.

Agree whether outputs replace a total or carry deltas. A receiver summing successive totals 300 and 340 can miscount under either approach. Versioned totals, retractions and idempotent sink writes are possible contracts, depending on storage. If the display distinguishes provisional and final, define the cutoff and input set behind finality. Business corrections after that point may still create a revised version. Execution mode does not decide these product semantics.

Equal-requirement axisBatchStream
Input boundaryIdentified files or snapshotContinuing arrivals and positions
First outputAfter bounded execution or stagesIncremental or window firing
Late dataAnother run or historical correctionRetained update or side output
RecoveryRecompute fixed inputRestore state and positions
Shared obligationsInput identity, versions, idempotent outputInput identity, versions, idempotent output

Prevent duplicate effects during either recovery style

Restarting a batch can repeat already written output; restoring a stream position can replay output too. Verify state/position consistency and sink transactions or stable-ID replacement. If exactly-once is documented, inspect its boundary rather than extending it to arbitrary emails or payments. Aggregation state and external effects need not share atomicity.

A checkpoint or snapshot must correspond to its claimed input position. Mismatched state and position can double-count or skip work. Without retained original input, a later full recomputation may be impossible. Compare retained-state cost with input-retention and recomputation cost. Include recovery time and permitted display during recovery, not only steady-state throughput. Both execution styles need an explicit answer to what can be restarted without changing the required result.

Compare operating cost under the same service objective

A daily billing aggregate prioritizing completion by morning and reproducible reruns may fit bounded batch. A monitor following second-scale changes may need ongoing updates. Either can require eventual completeness. Compare throughput, first and final latency, corrections, retained state, recomputation and operational complexity under equal requirements.

A live provisional stream can be reconciled with a daily batch. Their event meaning, cutoff, rounding, time and duplicate rules must align or discrepancies can persist. Define authority and correction flow when they differ. The sales sequence is a worked example, not a benchmark of a selected product or production pipeline. A concrete choice should lead to tests using fixed input, late arrival and controlled stop positions, matching the output contract discussed here.

Before comparing results, identify the admitted input IDs and cutoff for each run. If one accepts the late four and the other intentionally excludes it, explain the changed input contract before reporting a mismatch. Only after that alignment should throughput and latency be used to compare execution approaches. This preserves an equal-requirement comparison instead of rewarding a pipeline merely for doing less required work. Preserve the shared comparison contract.

Solve the one-second and next-morning requirements

A first answer is needed within one second, while complete 344 is required by the next morning. One candidate combines provisional continuous output with later reconciliation under shared event rules and versioned results. Daily 344 alone misses the first-latency objective; immediate 300 alone misses completeness. Evaluate the combined contract and its cost rather than naming one universal winner.

Appending the same window total on every rerun can double-count downstream. Use window identity, version and replacement or delta semantics. The learning outcome explains input boundary, time, correction, recovery and finality rather than choosing solely from data volume.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 17

Choose a learning path