Node.js Streams

Loading a whole file into memory before doing anything with it works fine for small files. For a 10GB video file, it doesn't — you'd need 10GB of free memory just to start. Streams solve this by processing data in small, manageable pieces as it arrives, rather than waiting for everything at once.

The four kinds of streams

• Readable — a source you read data from (a file being read, an incoming HTTP request).
• Writable — a destination you write data to (a file being written, an outgoing HTTP response).
• Duplex — both readable and writable at once (a TCP socket).
• Transform — a duplex stream that modifies data as it passes through (compression, encryption).

Reading a file as a stream

const fs = require('fs');

const readStream = fs.createReadStream('large-video.mp4');

readStream.on('data', (chunk) => {
  console.log(`Received a chunk of ${chunk.length} bytes`);
});

readStream.on('end', () => {
  console.log('Finished reading the file');
});

readStream.on('error', (err) => {
  console.error('Something went wrong:', err.message);
});

Each 'data' event delivers one small piece — typically 64KB by default — instead of the whole file arriving at once.

Writing a file as a stream

const writeStream = fs.createWriteStream('output.txt');

writeStream.write('First line\n');
writeStream.write('Second line\n');
writeStream.end('Final line\n'); // writes this last piece, then closes the stream

Piping — connecting a readable stream directly to a writable one

Manually listening for 'data' events and writing each chunk yourself works, but .pipe() does it more simply and handles an important detail automatically: backpressure (pausing the readable side if the writable side can't keep up, so memory doesn't pile up).

const fs = require('fs');

const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output-copy.txt');

readStream.pipe(writeStream); // reads input.txt, writes it to output-copy.txt, chunk by chunk

A realistic example — serving a large file over HTTP without loading it into memory

const http = require('http');
const fs = require('fs');

http.createServer((req, res) => {
  const fileStream = fs.createReadStream('large-report.pdf');
  res.writeHead(200, { 'Content-Type': 'application/pdf' });
  fileStream.pipe(res); // streams the file directly into the HTTP response
}).listen(3000);

Even a multi-gigabyte file works fine here — the server never holds the whole thing in memory at once, just one chunk at a time as it flows from disk to network.

A transform stream example — compressing a file on the fly

const fs = require('fs');
const zlib = require('zlib');

fs.createReadStream('report.txt')
  .pipe(zlib.createGzip())          // transform stream: compresses each chunk as it passes through
  .pipe(fs.createWriteStream('report.txt.gz'));

Three streams chained together — read, transform, write — each handing off small chunks to the next, none of them needing the whole file in memory at once.

Why this matters

Streams are the reason Node.js handles large files and high-throughput network traffic efficiently. Anywhere you see "streaming video," "large file upload," or "real-time data processing," streams are almost certainly doing the actual work behind the scenes.