What are the purposes of vinyl-buffer and gulp-streamify in gulp?

Question

As the documentation says, they both deal with transforming non-stream plugins to stream.

What I try to understand is, if I can use the .pipe() method on something, doesn't it mean it's a stream?

If so, what do I convert to what here?

vinyl-source-stream example:

(from: https://www.npmjs.com/package/vinyl-buffer)

var browserify = require('browserify')
var source = require('vinyl-source-stream')
var buffer = require('vinyl-buffer')
var uglify = require('gulp-uglify')
var size = require('gulp-size')
var gulp = require('gulp')

gulp.task('build', function() {
  var bundler = browserify('./index.js')

  return bundler.pipe()
    .pipe(source('index.js'))
    .pipe(buffer()) // <---------------------- why?
    .pipe(uglify())
    .pipe(size())
    .pipe(gulp.dest('dist/'))
})

gulp-streamify example:

(from: https://www.npmjs.com/package/vinyl-source-stream)

var source = require('vinyl-source-stream')
var streamify = require('gulp-streamify')
var browserify = require('browserify')
var uglify = require('gulp-uglify')
var gulp = require('gulp')

gulp.task('browserify', function() {
  var bundleStream = browserify('index.js').bundle()

  bundleStream
    .pipe(source('index.js'))
    .pipe(streamify(uglify())) // <----------- why?
    .pipe(gulp.dest('./bundle.js'))
})

Answer 1

One semi-useful example is to think about putting out a campfire with a bucket of water. To put out the fire you would want to completely fill up the bucket before dumping it on the fire rather putting a few drops in the bucket and then dumping lots of little drops over time on the fire. This metaphor doesn't capture everything but the big idea is this: you need a FULL bucket of water before you can put out the fire.

That "uglify" plugin works the same way. Imagine some enormous JS file you'd want to compress/uglify.

It will take a little bit of time to load the whole codebase & you definitely wouldn't want to try minifying each line as it comes in, right? Imagine you load a single line, minify it, load another line, minify it, etc etc-- it'd be a mess. You can't stream it (you need a full "bucket" of code before you can uglify it.) To uglify that file properly you'd need to load all that code first before attempting to uglify it.

Since Gulp is a "streaming" build system, you can't use uglify unless you have some mechanism to turn the stream into a buffer (& when it's done emit a stream.) Both tools you mention make this possible.

Here's the flow: STREAM > (BUFFER) > {perform some work on the whole "buffered" file} > STREAM > {other gulp work, etc }

To your specific question, you can use .pipe() because vinyl-buffer/gulp-streamify help "convert" streams to buffers then buffers to streams. They're different approaches to accomplish essentially the same thing.

Answer 2

As said, most plugins work with buffers (although some of them also support streams). Examples include gulp-uglify and gulp-traceur. You can do the conversion to buffers using gulp-buffer.

via https://medium.com/@webprolific/getting-gulpy-a2010c13d3d5

• gulp-uglify dosen't support stream, so you should convert stream to buffer (example uses vinyl-buffer)

• gulp-streamify can wrap old plugins to support streams(example uses gulp-uglify)

Different approaches but equally satisfactory results.

Answer 3

What I try to understand is if I can use the .pipe() method on something, doesn't it mean that it's a stream?

No, .pipe() can also pass buffers. this blog post explains it well:

https://medium.com/@sogko/gulp-browserify-the-gulp-y-way-bb359b3f9623

Some gulp-* plugins works by taking in buffered vinyl files objects as input.
But vinyl-source-stream emits a streaming vinyl file object.

That’s where vinyl-buffer comes in. So we simply need to convert that to a buffered vinyl by using vinyl-buffer, like so

Answer 4

What I try to understand is if I can use the .pipe() method on something, doesn't it mean that it's a stream?

Yes! It is a stream. But it's a an object stream!

Instead of streaming a series characters, it streams a series of objects, which are the files that you sourced.

Each 'data' event in a gulp stream emits a Vinyl file object, which looks something like this:

{
  cwd: '/',              //<string>
  base: '/test/',        //<string>
  path: '/test/file.js', //<string>
  contents: contents     //<string> | <Buffer> | <stream.Readable>
}

---

So gulp-buffer plugin is a Transform stream that converts the file contents from stream.Readable to Buffer.

You can see this in the source, where it saves the original content stream on line 24 and assigns a Buffer as the new file contents on line 35.

Streamify does the same thing, on line 35 and line 48.

It's ok to leave the file contents as a Buffer after Uglify is done processing it. It's always ok for the contents to be a Buffer, gulp just doesn't do this when sourcing because it's too costly.