Node Essentials
Streams & buffers
Lesson 4 of 5
What you'll learn
- Understand why streaming beats loading a whole file
- See chunked, transform-style processing
- Grasp backpressure as the slow side telling the fast side to wait
A buffer is a fixed chunk of raw bytes. A stream is a sequence of buffers arriving over time. The point of streaming is memory: instead of loading a 2 GB file into RAM and then processing it, you handle it one small chunk at a time, so memory stays flat no matter how big the source is.
import { createReadStream, createWriteStream } from "node:fs";
const source = createReadStream("huge.log");
const dest = createWriteStream("upper.log");
source.on("data", (chunk) => {
dest.write(chunk.toString().toUpperCase());
});
source.on("end", () => dest.end());
Transforms and backpressure
A transform sits between a source and a destination: each chunk flows in, gets changed, and flows out. The standard library expresses this with pipe, which also handles backpressure — if the writable side is slower than the readable side, it signals "pause" so chunks don't pile up in memory.
source.pipe(transform).pipe(dest); // pipe manages backpressure for you
Order is preserved
Chunks come through in order, but boundaries are arbitrary — a chunk can split a line or even a multi-byte character in half. Transforms that care about structure must buffer leftovers between chunks.
The challenge is a transform model: it pulls chunks one at a time, runs each through a transform function, and accumulates the result — the essence of stream processing without ever holding the whole input.
Run it. The transform consumes chunks one at a time, never holding the full input, and accumulates the output.
Next: putting it together into an HTTP server.
Sign in to save your progress across devices.