Halloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PCHalloween DealsAmazon USOctober picks are moving - compare before checkoutAmazon US: current deals, useful picks and tech finds.See Picks×
FREEDOM251

Freedom251 › Blog › backpressure

Node.js write() vs pipe(): Choose Stream Flow Control That Saves Memory

By Freedom251 Team · · 9 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Backpressure is Node.js stream flow control. A producer can generate chunks faster than a writable destination can process them. When writable.write(chunk) returns false, stop producing and wait for the writable’s drain event before sending more. If you ignore that signal, the internal queue can grow, garbage collection can become less effective, and process RSS can remain high.

The safest design is usually demand-driven: expose your producer as a Readable and connect it with pipe(), or use pipeline() when coordinated completion and error handling matter. Manual write()/drain control is appropriate when your application must schedule each chunk itself.

As an Amazon Associate I earn from qualifying purchases.

What backpressure means in Node.js

Backpressure is a speed-matching mechanism between two stages of a data flow. The upstream stage might read files, fetch records, compress data, or generate messages rapidly. The downstream stage might be a file, socket, parser, database client, or another transform that can accept data only at a slower rate.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Node.js streams place a buffer between those stages. The writable side accepts a chunk, queues it when necessary, and eventually hands it to the destination. Once the queued amount reaches a configured threshold, the writable tells the producer to pause. That signal is the return value from write():

  • true means the chunk was accepted and the writable remains below its signaling threshold.
  • false means the producer should stop writing until the writable emits 'drain'.

This is flow control, not a promise that the process can never use more memory. The stream limits how quickly it asks for more data, but your code can defeat that protection by continuing to enqueue work or by retaining chunks elsewhere.

Why ignoring write() fills memory

Consider a loop that creates thousands of buffers and calls destination.write(buffer) without checking the result. The writable accepts each call by placing data in its internal queue or passing it to the underlying destination. If the destination is slower, queued chunks accumulate faster than they are completed.

As the queue grows, Node.js must retain more Buffer objects, strings, or application objects. Allocation and garbage collection consume more CPU, pauses become less predictable, and resident set size (RSS) can stay elevated even after the producer stops. The Node.js writable documentation specifically warns that continuing to write while a stream is not draining can lead to excessive memory use and degraded garbage-collector performance. A TCP socket can remain in this state indefinitely if the remote peer stops reading, so a network producer that ignores backpressure may grow without a natural recovery point.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

There is no universally safe queue length. The right amount depends on chunk size, destination latency, concurrency, object size, and the amount of memory available to the process. Measure your workload rather than treating a convenient number of chunks as a guarantee.

How the false/drain protocol works

The protocol is deliberately simple:

  1. Call write(chunk).
  2. If it returns true, continue producing according to your normal scheduling logic.
  3. If it returns false, stop calling write().
  4. Resume only after the writable emits 'drain'.

Waiting for 'drain' means the queued data has fallen enough for the writable to accept more. It does not mean every byte has necessarily reached its final destination; completion still depends on the stream’s normal finish and close behavior.

A minimal manually controlled producer

function writeChunk(stream, chunk, continueProducing) {
  if (!stream.write(chunk)) {
    stream.once('drain', continueProducing);
  } else {
    continueProducing();
  }
}

This pattern illustrates the essential pause and resume behavior. Production code must also handle 'error', 'close', abort signals, and producer cancellation. If the destination fails while the producer is paused, waiting forever for 'drain' is a bug; an error or abort path must release the producer and its retained data.

Do not add a second unbounded queue

Pausing stream writes does not help if your application continues to collect every pending item in an array, promise list, or message buffer. Keep the number of in-flight items bounded at the same point where you apply backpressure. A bounded producer should stop fetching or generating new work, not merely stop calling write() after it has already materialized the next million chunks.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

highWaterMark: threshold, not a memory cap

highWaterMark controls when a stream signals that the producer should slow down. The Node.js documentation describes it as a threshold, not a limit, and states that it does not enforce a strict memory limitation in general. Data already being processed, queued in other stages, retained by application code, or held by the operating system is outside that one threshold.

Stream configuration Documented default What the value counts
Writable, non-object mode 65,536 bytes (64 KiB) Bytes for binary data such as Buffers and typed arrays
Writable, object mode 16 objects Object count, regardless of each object’s size
String writes without decoding Uses the stream’s configured threshold UTF-16 code units

These defaults come from the Node.js v22.23.3 documentation. The v24.20.0 documentation records the same documented defaults and notes that the default was increased in Node.js v22.0.0. Verify the runtime actually deployed by your application and any custom stream constructor before relying on a value from an older tutorial.

In object mode, 16 small records and 16 multi-megabyte records both count as 16 objects. Therefore, object-count backpressure can still represent a large memory footprint. If object sizes vary widely, impose an additional byte or cost budget in your application rather than assuming the object-mode threshold is a memory limit.

Choose the right stream composition

Use the highest-level composition that matches your data source. It is easier to preserve backpressure when the stream machinery controls when the producer is asked for another chunk.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Approach Best fit Backpressure and error behavior Main caution
Readable.pipe(writable) A demand-driven Readable feeding one destination Limits how much the source pushes while the destination is saturated An error on the Readable does not automatically close the writable; add explicit cleanup when needed
pipeline() Multiple stages or async-iterator workflows that need coordinated completion Handles backpressure-related flow and coordinates errors and teardown Use the callback, promise, or abort form appropriate to your Node.js version
Manual write() and 'drain' A scheduler must decide exactly when to create and send each chunk Gives explicit pause/resume control You must implement error, close, cancellation, and termination handling

Prefer a Readable for on-demand generation

If records can be fetched or generated as the destination requests them, represent that source as a Readable. With pipe(), downstream pressure can prevent the Readable from continuing to pull data at full speed. This avoids writing a separate queue coordinator and makes the data path visible as a stream topology.

Use pipeline() for coordinated lifecycles

For a chain such as source, decompressor, parser, and destination, pipeline() is generally safer than wiring every event yourself. Node.js documents pipeline support for async iterators and describes it as abstracting backpressure handling and backpressure-related errors. It also gives you a single completion or failure path on which to release resources.

A plain pipe() connection remains useful for simple flows, but remember its error asymmetry: a Readable error does not automatically close the writable destination. If the destination holds a file descriptor or socket, explicitly destroy or otherwise clean it up when the source fails.

Duplex and Transform streams have two pressure points

A Duplex or Transform stream is not one queue with one threshold. It has a writable side that receives input and a readable side that exposes output. Those sides maintain separate buffering and can become saturated independently.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For example, a Transform may accept input quickly while its consumer reads output slowly. Its readable buffer can fill even though the writable side is still receiving calls. Conversely, a slow upstream source can leave the Transform’s writable side mostly empty while a downstream consumer remains ready. Diagnose input and output pressure separately and make sure the output is actually consumed.

Node.js specifically identifies an unconsumed Transform as a problematic write target. Writing into a Transform that nobody reads can keep its output queued and prevent the expected flow-control cycle from completing.

A practical debugging checklist

When a queue appears to “eat all the memory,” determine whether the growth is in a stream buffer, an application queue, or both.

  1. Find every write site. Log or inspect calls to write(), including wrappers and libraries. Confirm that each call’s Boolean result is observed.
  2. Record pauses and resumes. Count how often writes return false, how long the producer waits for 'drain', and whether a drain ever arrives.
  3. Measure the destination. For a socket, check whether the peer is reading and whether network latency or a stalled connection explains the sustained pressure. For files or transforms, inspect downstream throughput and consumption.
  4. Track process memory. Compare heap usage, external memory, array-buffer usage, and RSS over time while reproducing the same workload. A growing RSS with stable JavaScript heap can indicate retained Buffers or native allocations.
  5. Inspect your own queues. Count pending promises, arrays, retry lists, and fetched records. A stream may be applying backpressure correctly while an outer queue continues growing.
  6. Test failure paths. Force a destination error, remote disconnect, abort, and slow consumer. The producer should stop, release retained chunks, and finish with an observable error instead of waiting forever.

Do not build production diagnostics around undocumented fields such as writable.writableBuffer or readable.readableBuffer; the Node.js v24 documentation discourages relying on them. Prefer supported stream events and state, application-level counters, runtime memory metrics, and a reproducible workload.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Common failure modes and fixes

The loop keeps writing after false

Symptom: memory rises rapidly during a large export or upload. Fix: return from the producer loop as soon as write() returns false, then resume from a one-time 'drain' listener. Do not schedule another batch before that callback runs.

The producer fetches ahead of the stream

Symptom: the writable queue is modest, but thousands of records remain in application memory. Fix: move the fetch operation into a demand-driven Readable or enforce a separate maximum number of fetched-but-unsent records. Backpressure must control acquisition, not only transmission.

Drain never fires

Symptom: the producer pauses permanently. Fix: listen for 'error', 'close', and abort conditions alongside 'drain'. A remote socket may never drain if its peer has stopped reading; reconnect, time out, or cancel according to the protocol rather than waiting indefinitely.

Object mode hides the real cost

Symptom: the queue reports only a few pending objects but RSS is large. Fix: measure object sizes or impose a byte-weighted budget. The object-mode threshold counts objects, not their serialized or in-memory size.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A Transform appears to accept data but the pipeline stalls

Symptom: writes continue into a Transform while output memory grows. Fix: ensure the readable side is consumed, connect the stages with pipeline() where appropriate, and inspect both input and output rates.

Setting a custom highWaterMark

Raising highWaterMark can reduce how often a producer pauses, but it also allows more data to be buffered before the signal arrives. Lowering it can reduce per-stream buffering while increasing scheduling overhead and the number of pauses. Neither choice makes an unbounded producer safe.

Choose a value from measurements of your actual chunk sizes, destination latency, throughput, and concurrency. Document whether the value is bytes or objects, and include it in capacity calculations for every simultaneous stream. Recheck the choice after changing Node.js versions, because documented defaults changed in Node.js v22.0.0.

Design rules for a memory-safe queue

  • Make the producer stop acquiring work when downstream is saturated.
  • Treat a false return as a required pause, not an advisory warning.
  • Resume on 'drain' only after confirming the stream has not errored or closed.
  • Keep retries, promises, and prefetched records bounded independently of stream buffers.
  • Use Readable composition for demand-driven sources and pipeline() for coordinated multi-stage flows.
  • Account for separate readable and writable buffers in Duplex and Transform streams.
  • Measure bytes for binary streams and object sizes for object-mode streams.
  • Validate behavior on the exact Node.js runtime and stream constructors deployed in production.

Backpressure works when every layer respects the same pause signal. A stream’s threshold tells you when to stop asking for more; your producer, queue, and error paths must make that pause real.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

backpressureJavaScriptMemory ManagementNode.jsstreams

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.