Node.js interview questions: Streams and backpressure

So when the writable side is slower, its internal buffer starts filling up. Every write() call adds the chunk to that buffer, and once the buffered amount hits the highWaterMark, write() returns false, which is basically telling the producer to stop for a bit. Then the producer is supposed to wait for the 'drain' event, that fires when the buffer has emptied out, and then it can start writing again. If you connect things with pipe() this is all handled for you, it pauses the readable when the destination returns false and resumes it on 'drain'. And that's backpressure, really. The consumer's speed propagates back to the producer so data doesn't pile up in memory. If you're writing manually and you ignore the false return, Node just keeps buffering and memory grows, so you kind of have to respect that false.

So pipe() moves the data and handles backpressure fine, but the error handling is where it falls down. Errors don't propagate through the chain, so each stream needs its own 'error' listener, and if one thing fails the other streams in the chain can just stay open. stream.pipeline() is the fix for that. You pass it the streams plus a callback, or there's a promise version in stream/promises, and it wires up the pipes, forwards the first error to your callback, and cleans up every stream when the pipeline finishes or fails. That cleanup is what stops the resource leaks bare pipe chains are kind of famous for, like file descriptors left open when a destination errors. The rule I've picked up is basically use pipeline for anything production-facing, and treat pipe as the lower-level primitive it wraps.

highWaterMark is basically the buffering threshold for a stream. On a writable, once the buffered data reaches it, write() starts returning false and the producer is supposed to wait for 'drain'. On a readable it's how much the stream pre-reads into its internal buffer before it pauses the underlying source. The default is 64KiB for byte streams, I believe. And it's a soft threshold, you can keep writing past it and the stream just keeps buffering. With objectMode: true the units change from bytes to objects, and the default becomes 16, so sixteen objects buffered no matter how big each one is. That's the classic gotcha, sixteen large objects can be a ton of memory even though sixteen sounds small. Tuning it is a memory versus throughput thing, bigger buffers mean fewer pauses and larger reads, smaller ones keep memory tight.

A Transform stream is a duplex stream where you implement _transform(chunk, encoding, callback). Data written to the writable side comes into your function, you do your processing, push() the results out to the readable side, and call the callback to say you're ready for the next chunk. Compression streams and parsers are the textbook examples, I think. The other way to consume a readable is async iteration, so for await (const chunk of stream) pulls chunks one at a time. And because you only pull after your loop body finishes, backpressure kind of just works, the stream won't run ahead of your processing. It reads like normal sequential code, error handling is a regular try/catch, and from what I've seen it's the recommended style for consuming streams when you don't actually need a full pipeline of stages.

So there are two consumption modes on a readable. In flowing mode the stream pushes chunks at your 'data' listeners as fast as they come in. In paused mode you pull explicitly with read(), usually inside a 'readable' handler. Streams start out paused, I'm pretty sure. Then attaching a 'data' listener, calling resume(), or piping to a destination flips it to flowing, and pause() flips it back. Flowing is just the plain push style, and paused gives you control over when and how much you consume. Honestly modern code doesn't really manage this by hand, pipe, pipeline, and for await handle the modes internally. But the modes explain some classic surprises, like a stream that just sits there doing nothing until a 'data' listener shows up and then suddenly fires chunks all at once.

So in Node the incoming request is a readable stream and the response you're sending is a writable, and a proxy is basically just connecting the two. You make the upstream request, then pipe the upstream response into your server response, and you have to write the status code and headers before the body starts. With stream.pipeline(upstreamRes, res) the data flows through in chunks and memory stays flat no matter how big the body is. If your memory scales with response size, you're buffering somewhere, collecting the whole upstream body before sending it on, like with await response.json() or concatenating chunks by hand or something. Backpressure comes along for free, if the client is slow the pipe slows the upstream read down too. And you need error handling for either side dying mid-transfer, which pipeline covers by cleaning up both streams.

I'd build a pipeline with three stages, basically. fs.createReadStream reads the input, a transform splits the chunks into lines and applies the per-line change, and a writable stage writes the output file. You wire them with stream.pipeline so if anything errors the whole chain gets torn down. The read stream emits chunks of around 64KiB, so memory only ever holds a chunk or two instead of the whole file. The tricky bit is chunks don't line up with line endings, so the splitter keeps the trailing partial line in a small buffer and sticks it in front of the next chunk. readline.createInterface or a split transform handles that for you. Backpressure stops the reader outrunning the writer, so the footprint stays constant from the first byte to the ten-billionth. To verify I'd just watch process.memoryUsage() during a run, it should plateau early and stay flat regardless of file size.

destroy() is the forceful teardown, basically. It releases the underlying resource, the socket or the file descriptor, emits an 'error' if you passed one in, and then emits 'close'. After that the stream is unusable and further writes fail. The events break down roughly like, 'end' fires on a readable once all the data has been consumed, 'finish' fires on a writable once everything you wrote has been flushed to the underlying system, and 'close' fires when the stream and its resource are actually released. When a socket dies mid-transfer, the streams attached to it need to get destroyed too or they leak. Which is why the advice is stream.pipeline, or stream.finished for a single stream. They watch every terminal path, error, premature close, completion, and destroy the chain instead of leaving pieces open. ERR_STREAM_PREMATURE_CLOSE is what you get when a stream closed before it signaled completion, I believe.

Node ships two stream families, so there's the classic Node streams, Readable, Writable, events, pipe, and then the WHATWG web streams, ReadableStream, WritableStream, TransformStream, which came over from the browser standard. A fetch body is a web stream. The main difference is style, I think. Node streams are event-based and push-oriented, web streams are promise-based and pull-oriented, with readers and controllers instead of events. For interop Node gives you official converters, Readable.toWeb() and Readable.fromWeb(), and the same methods exist on Writable and Duplex. So for the fetch-to-SDK case you'd write Readable.fromWeb(response.body) and then pipe or pipeline into the SDK's stream like normal. Backpressure works across the conversion, from what I understand. Web streams are the more portable choice since they run in browsers, edge runtimes, and Node alike, and Node streams stay the native fit for Node's own APIs like fs and http.