Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsSpark Structured Streaming can track input progress, checkpoint work and recover a query—but those capabilities do not make every pipeline correct, bounded or restartable. End-to-end behavior still depends on choices such as whether inputs can be replayed, whether sinks tolerate retries, how state is limited, what lateness a watermark permits and whether a changed query can use its existing checkpoint.
What Structured Streaming does—and what it does not
Structured Streaming represents a stream as a DataFrame or Dataset computation that Spark incrementally updates as new data arrives. To recover progress, Spark records source offsets and per-trigger offset ranges in checkpointing and write-ahead logs. If a query fails, Spark can use that recorded progress to restart and, where necessary, reprocess input.
As an Amazon Associate I earn from qualifying purchases.
That is meaningful fault tolerance, but it is not an unconditional promise that every external effect happens once. End-to-end exactly-once behavior depends on the whole path: a source must support replay, and the sink must handle reprocessing safely. The Spark 3.5.8 Structured Streaming Programming Guide says, “The streaming sinks are designed to be idempotent for handling reprocessing.” That describes the intended behavior of streaming sinks; it does not make an arbitrary database call, API request or other side effect idempotent.
Exactly once depends on the source and sink
Consider a trigger that reads a range of source offsets and then writes results to an external system. If the job fails after the external system accepts the write but before Spark records successful progress, recovery may process that input again. The source’s replay capability lets Spark recover the computation; it does not, by itself, undo or deduplicate a write that already took effect.
#1 Best Overall
For every output, identify what happens if the same logical record or batch is submitted again. A sink that can safely recognize a repeat is different from one that performs an irreversible action each time it receives a request. Check the guarantees of the particular source and sink implementation, and test failure points around writes and progress commits. Do not infer end-to-end exactly-once behavior merely from the fact that the query has a checkpoint.
State needs a size and retention plan
Aggregations, deduplication, joins and other stateful operations retain intermediate information across triggers. The amount retained depends on the operation, the keys and their distribution, and the rules that let Spark discard old state. A query can be logically correct yet operationally unhealthy if its state grows beyond what the workload and chosen state store can sustain.
Spark 3.5.7 documentation warns that large state in the HDFS-backed state store can cause long garbage-collection pauses when state is held in JVM memory. It also documents a RocksDB state-store provider, which manages state using native memory and local disk while continuing to checkpoint it. RocksDB is an available design option, not a universal performance fix: it does not remove the need to understand state growth, local storage, recovery and workload behavior.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchBefore deploying a stateful query, estimate which keys and time ranges must remain available, observe state growth under realistic key cardinality, and decide how state will be cleaned up. If the business logic has no safe point at which old state can be discarded, changing the state store alone does not solve the unbounded-retention problem.
Rank #3
Watermarks make the late-data trade-off explicit
A watermark encodes how late the query is prepared to accept event-time data and helps determine when state can be cleaned up. It is a policy, not a guarantee that every event arriving later will be retained. The right setting depends on how much lateness the application must accommodate and what it can tolerate losing or delaying.
For queries with multiple inputs, Spark 3.5.6 documentation describes a global watermark policy with two distinct trade-offs:
| Policy | How it behaves | Practical consequence |
|---|---|---|
| Minimum watermark (documented default) | Follows the slowest input stream. | Can preserve more opportunity for late data from a slower stream, while delaying state cleanup or finalization. |
| Maximum watermark | Advances with the faster input. | Can move cleanup and finalization sooner, but aggressively drops data from slower streams. |
Choose based on the cost of incompleteness versus the cost of waiting. A dashboard that values quick results may make a different choice from a reconciliation pipeline that must tolerate slow arrivals. Document what happens to records outside the accepted lateness window so that downstream users understand whether a result is provisional or final.
A checkpoint is not a promise that every query change can restart
Checkpoints preserve recovery information for a query, but stateful query evolution can make that information incompatible with new code. Spark 3.5.6 documentation says stateful operator schemas must remain compatible across restarts when recovering state; changes such as changing grouping keys or aggregates are not allowed under that constraint.
Treat checkpoint continuity as part of deployment design, not as an automatic property of changing application code. Before altering a live stateful query, consult the documentation for the Spark version actually deployed and determine whether the change is compatible with its existing state. If it is not, plan an explicit migration or a new query and checkpoint rather than assuming a restart will work.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Set and measure latency targets on the real workload
The Spark 3.5.6 guide states that default micro-batch execution can achieve end-to-end latency “as low as 100 milliseconds.” This is a version-specific capability statement in the documentation, not a universal benchmark or a service-level guarantee. It does not establish that a particular query—with its source, state, sink, trigger cadence and resource constraints—will meet that latency.
Define the target that matters to users, then measure it with representative input volume, event-time skew, state size and sink behavior. Measure throughput and recovery behavior as well as the time for an individual batch: a configuration that looks fast at low volume may fall behind when arrivals exceed processing capacity, and a low steady-state latency says little about how long recovery takes after failure.
Review the whole pipeline, not just the streaming API
Before calling a design fault tolerant, answer these operational questions for the deployed Spark version and each source and sink:
- Recovery: Can the source replay the required offsets, and can operators tell what progress was checkpointed when a failure occurred?
- Writes: What happens if the same output is delivered again, especially when the sink triggers an external side effect?
- State: Which operations retain data, how large can that state become, and what policy permits cleanup?
- Event time: How late can events arrive, and in a multi-stream query, is slower-stream completeness or faster finalization more important?
- Evolution: Is the planned code or schema change compatible with the state already stored in the checkpoint?
- Performance: What latency and throughput must hold under realistic load, and how will lag, state growth and recovery be observed?
Structured Streaming provides mechanisms for incremental computation and recovery. Architecture determines whether those mechanisms fit the pipeline’s correctness, retention, change-management and performance requirements.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
