Node.js Streams

Why Streams Exist

Reading an entire 2GB video file into memory before doing anything with it would be slow and could exhaust available RAM. Streams solve this by processing data in small chunks as it arrives, rather than waiting for the whole thing to be available. Node defines four fundamental stream types: Readable (a source of data, like a file or an incoming HTTP request), Writable (a destination for data, like a file or an outgoing HTTP response), Duplex (both readable and writable, like a TCP socket), and Transform (a duplex stream that modifies data as it passes through, like a gzip compressor). Every stream in Node is also an EventEmitter.

Readable and Writable Streams

A readable stream emits a 'data' event for each chunk it produces and an 'end' event once there's nothing left to read. A writable stream exposes .write(chunk) to send data to it and .end() to signal that no more data is coming, after which it emits 'finish'. Both kinds respect backpressure: if a writable destination can't keep up with an incoming readable source, Node automatically pauses the flow rather than piling up unbounded data in memory.

• fs.createReadStream(path) — a readable stream over a file's contents
• fs.createWriteStream(path) — a writable stream that saves data to a file
• process.stdin / process.stdout — readable and writable streams for the terminal
• 'data' and 'end' — core readable stream events
• 'finish' — emitted by a writable stream once .end() has fully flushed
• 'error' — emitted by either kind when something goes wrong; always handle it

const fs = require('node:fs');

const readStream = fs.createReadStream('large-file.txt', { encoding: 'utf8' });

let chunkCount = 0;

readStream.on('data', (chunk) => {
  chunkCount++;
  console.log(`Received chunk ${chunkCount}, size: ${chunk.length} characters`);
});

readStream.on('end', () => {
  console.log(`Done. Read ${chunkCount} chunks in total.`);
});

readStream.on('error', (err) => {
  console.error('Read stream failed:', err.message);
});

Piping Streams Together

Manually forwarding every 'data' chunk from a readable stream into a writable one, while correctly handling backpressure and errors, is tedious and easy to get wrong. The .pipe() method does all of that for you: readableStream.pipe(writableStream) connects the two, automatically pausing the source when the destination is busy, and resuming it once the destination is ready for more. Transform streams can be chained in the middle, so a readable file stream can be piped through a compressor and then into a writable stream in a single line.

const fs = require('node:fs');
const zlib = require('node:zlib');

const source = fs.createReadStream('report.txt');
const gzip = zlib.createGzip();
const destination = fs.createWriteStream('report.txt.gz');

source.pipe(gzip).pipe(destination);

destination.on('finish', () => {
  console.log('report.txt.gz created successfully');
});

source.on('error', (err) => console.error('Source error:', err.message));
destination.on('error', (err) => console.error('Destination error:', err.message));

const fs = require('node:fs');

const writeStream = fs.createWriteStream('numbers.txt');

for (let i = 1; i <= 5; i++) {
  writeStream.write(`Line ${i}\n`);
}

writeStream.end(); // signal that no more data is coming

writeStream.on('finish', () => {
  console.log('All lines flushed to numbers.txt');
});