Stream vs Batch Processing

Batch processing computes over a bounded, complete dataset and produces results when the job finishes. Stream processing computes continuously over an unbounded sequence of events and produces results incrementally as data arrives. The choice determines latency, complexity, and how late or out-of-order data is handled.

Batch processing

A batch job reads a fixed input (files in distributed storage, a table snapshot), processes all of it, and writes an output. It runs on a schedule (hourly, daily) or on demand.

MapReduce is the classic model. The map phase runs a function on each input record and emits key-value pairs. The framework shuffles the pairs so all values for one key reach the same reducer. The reduce phase combines the values per key.

Word count example: the map function turns each line into (word, 1) pairs. The shuffle groups pairs by word, so all ("error", 1) pairs reach one reducer. That reducer sums the values and emits ("error", 3) if three pairs arrived.

graph LR
    I["Input splits"] --> M1["Map"]
    I --> M2["Map"]
    I --> M3["Map"]
    M1 -->|"shuffle by key"| R1["Reduce"]
    M2 -->|"shuffle by key"| R1
    M2 -->|"shuffle by key"| R2["Reduce"]
    M3 -->|"shuffle by key"| R2
    R1 --> O["Output files"]
    R2 --> O

Apache Spark generalizes this to a graph of transformations over distributed datasets, keeping intermediate data in memory where possible instead of writing it to disk between every stage as classic Hadoop MapReduce does. Jobs are fault tolerant: a failed task is re-run from its input, and a deterministic computation gives the same result on re-execution.

Properties of batch:

• High throughput and efficient use of resources over large data.
• Simple correctness reasoning: the input is fixed and complete, so a rerun gives the same output.
• Latency is at least the job duration plus the schedule interval: minutes to hours.
• Suited to reports, model training, large joins, backfills, and periodic aggregation.

Stream processing

A stream processor consumes events continuously from a log such as Kafka, applies operators (filter, map, join, aggregate), and emits results to another stream or a store. Apache Flink and Kafka Streams are common engines. Latency is milliseconds to seconds.

Stream processing brings problems batch does not have.

Event time and processing time

• Event time is when the event occurred, recorded in the event.
• Processing time is when the processor sees it.

Networks delay and reorder events, so processing order differs from event order. Results that should reflect when things happened (for example, clicks per minute) must use event time.

Windowing

An aggregate over an unbounded stream needs a boundary. A window groups events into finite sets.

Watermarks and late data

When can a window for 10:00 to 10:01 be finalized, given events might still arrive late? A watermark is the processor's estimate of event-time progress: a watermark with time T asserts that events with event time at or before T have (almost certainly) all arrived. A window is emitted when the watermark passes its end.

sequenceDiagram
    participant S as Source
    participant W as Window operator
    S->>W: event t=10:00:20
    S->>W: event t=10:00:50
    S->>W: event t=10:00:40 (out of order)
    S->>W: event t=10:01:10
    Note over W: max event time 10:01:10 minus 5 s allowed disorder gives watermark 10:01:05
    Note over W: watermark passes 10:01:00, emit window 10:00-10:01
    S->>W: event t=10:00:55 (late)
    Note over W: late: drop, update the emitted result, or send to a side output

In the diagram, events at 10:00:20, 10:00:50, and 10:00:40 belong to the 10:00 to 10:01 window. The event at 10:01:10 raises the maximum event time seen, and with a 5-second allowance the watermark becomes 10:01:05, which is past the window end, so the window is emitted with three events. The event stamped 10:00:55 arrives after that and is late.

A common watermark strategy is the maximum event time seen minus a fixed allowed out-of-orderness. A bigger allowance handles more disorder but increases latency. Events that arrive after the watermark are late. Policies: drop them, keep the window state for an allowed lateness period and emit an updated result, or route them to a side output.

State and fault tolerance

Aggregations, joins, and deduplication need state per key. Engines keep state locally (Flink uses embedded stores such as RocksDB, Kafka Streams uses local stores backed by changelog topics) and take periodic checkpoints or replicate state changes so a failed task restores state and resumes from the matching input offset. With a replayable source and transactional or idempotent output, this yields exactly-once effects on state. See delivery semantics and idempotency.

Comparison

The boundary is blurred: Spark Structured Streaming by default processes streams as a series of small batches (micro-batches), and Flink treats batch as a bounded stream.

Lambda and Kappa architectures

The Lambda architecture runs two paths over the same data:

• Batch layer: periodically recomputes accurate results from the full immutable dataset.
• Speed layer: a stream job computes low-latency approximate results over recent data.
• Serving layer: merges both views for queries.

graph LR
    S["Event source"] --> BL["Batch layer: recompute all data"]
    S --> SL["Speed layer: recent data"]
    BL --> SV["Serving layer"]
    SL --> SV
    SV --> Q["Queries"]

Cost: two codebases implementing the same logic, which must produce equivalent results, and two systems to operate.

The Kappa architecture uses only stream processing. A retained log (Kafka) is the source of truth. To recompute after a code change or bug fix, start a new stream job that replays the log from the beginning into a new output, then switch consumers to it. One codebase, one system. It requires a log that retains all needed history and a stream engine capable of processing history at high speed.

Choosing

• Daily reports, model training, large historical joins, and cases where results within hours are enough: batch.
• Fraud detection, monitoring, live dashboards, real-time personalization, and event-driven reactions: stream.
• Need both fresh and corrected results: Kappa-style streaming with replay, or Lambda where batch and streaming tools have different strengths.

Interview questions

• Why do stream processors need event time? Events arrive late and out of order, so processing time misassigns them to windows. Event time places each in the window where it occurred.
• What does a watermark do? It states how far event time has progressed, so the engine knows when a window can be emitted. A strategy that allows more out-of-orderness produces more complete results with higher latency.
• What is the main drawback of the Lambda architecture? The same logic is implemented twice, in a batch and a stream system, and the two outputs must stay consistent.
• How does Kappa reprocess data? It replays the retained log through a new version of the stream job into a new output, then switches readers to it.