Piping stream from busboy to request post

Question

I have multipart/form-data that I am posting to an express endpoint /data/upload, form markup below:

form(enctype=\"multipart/form-data         


        

Answer 1

The problem here is that you need to provide 'Content-Length' for the multipart upload manually, because request (and underlying form-data) can't figure it out by themselves. So request sends invalid Content-Length: 199 (the same for any incoming file size), which breaks the java multipart parser.

There are multiple workarounds:

1) Use incoming request 'Content-Length'

request.post({
  url: server.baseURL + 'api/data',
  formData: {
    file: {
      value: fileStream,
      options: {
        knownLength: req.headers['content-length']
      }
    }
  }
}, function (err, r, body) {
  // Do rendering stuff, handle callback
})

This will produce a bit incorrect request though, because incoming length includes other upload fields and boundaries, but busboy was able to parse it w/o any complaints

2) Wait until file is completely buffered by the node app then send it to java

var concat = require('concat-stream')
req.busboy.on('file', function (fieldName, fileStream, fileName, encoding, mimeType) {
  fileStream.pipe(concat(function (fileBuffer) {
    request.post({
      url: server.baseURL + 'api/data',
      formData: {
        file: fileBuffer
      }
    }, function (err, r, body) {
      // Do rendering stuff, handle callback
    })
  }))
})

This will increase app memory consumption, so you needed to be careful and consider using busboy limits

3) Buffer file to disk before uploading (just for the reference)

• express + multer - I recommend using express for webservers, it makes things more manageable, and multer is based on the busboy
• formidable

Answer 2

If possible, send a custom header with exactly size file (bytes). The header always can be read before handle payload stream. Use this instead of content-length header of previous answer, because that sometimes doesn't works (with small files, i guess, but i cannot ensure that works with large files).

Answer 3

For the answer of Afanasii Kurakin

request.post({
  url: server.baseURL + 'api/data',
  formData: {
    file: {
      value: fileStream,
      options: {
        knownLength: req.headers['content-length']
      }
    }
  }
}, function (err, r, body) {
  // Do rendering stuff, handle callback
})

You should change from req.headers['content-length'] to the real file size, normally the content-length from header is bigger than the file size. I got pain because of the content-length and after using the file size, everything worked perfectly.