"Streams in node are one of the rare occasions when doing something the fast way is actually easier. SO USE THEM. not since bash has streaming been introduced into a high level language as nicely as it is in node."
– @dominictarr in his high level node style guide.
Streams emits Events, the native observer pattern of NodeJS.
At this moment exists 3 iterations of the Stream implementation that depend of your version of node/iojs.
Instead of use the native API (that depend of your node version) better use readable-stream or through2. Both are backward compatibility and works fine in browser build. (this last is more lightweight because just expose a Duplex Stream).
.pipe() is just a function that takes a readable source stream and hooks the output to a destination writable stream (as UNIX commands):
tweetStream.pipe(process.stdout)
Using .pipe() has other benefits too, like handling backpressure automatically so that node won't buffer chunks into memory needlessly when the remote client is on a really slow or high-latency connection.
We have 4 types of Streams: Duplex, Readable, Transform and Writable.
A good library that collect stream utilities are mississippi.
You can implement a Stream using inheritance or composition.
For emit chunks of data you need to create a object that implement the ._read method.
It emits data events each time they get a chunk of data. From the implementation this is synonymous of this.push(data).
It emits end when it has no more data this.push(null).
When you are using a Readable Stream you can use resume() and pause() methods to control the data flow of the stream.
For emit chunks of data you need to create a object that implement the ._write method.
.end to close the stream and also you can pass the last chunk to .write.
Just provide the callback if you want to wait, but the order of the successive calls is guaranteed.
finish event is emitted when the end() method has been called, and all data has been flushed to the underlying system.
A duplex stream is one that is both Readable and Writable, such as a TCP socket connection.
It was implemented in the most recent node version but you can use through2.
It's a special type of streams because interact with the filesystem.
uses open event to control the file state of the fs.ReadStream/fs.WriteStream streams.
Also it's an especial kind of streams. They particularry fire exit event that is different from close.
It uses stdio to setup stream communication between the child_process and where the output have to be write/read (by default stdin, stdout and stderr that are align with UNIX standard streams).
You can convert whatever stream interface into a callback. See my stream-callback library that makes easy this conversion.
It's also possible transform an async callback function into a stream interface. You need to be sure to handle correctly the backpressure of the stream. In my experience in this area I use from2. Check fetch-timeline or totalwind-api as examples.
Interested libraries to use with streams are: