Large file upload with WebSocket

Question

I\'m trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can\'t figure out how to write \"send this slice

Answer 1

EDIT : The web world, browsers, firewalls, proxies, changed a lot since this answer was made. Right now, sending files using websockets can be done efficiently, especially on local area networks.

Websockets are very efficient for bidirectional communication, especially when you're interested in pushing information (preferably small) from the server. They act as bidirectional sockets (hence their name).

Websockets don't look like the right technology to use in this situation. Especially given that using them adds incompatibilities with some proxies, browsers (IE) or even firewalls.

On the other end, uploading a file is simply sending a POST request to a server with the file in the body. Browsers are very good at that and the overhead for a big file is really near nothing. Don't use websockets for that task.

Answer 2

In order to serialize this operation you need the server to send you a signal every time a slice is received & written (or an error occurs), this way you could send the next slice in response to the onmessage event, pretty much like this:

function Uploader(url, file) {
    var fs = new FileSlicer(file);
    var socket = new WebSocket(url);

    socket.onopen = function() {
       socket.send(fs.getNextSlice());
    }
    socket.onmessage = function(ms){
        if(ms.data=="ok"){
           fs.slices--;
           if(fs.slices>0) socket.send(fs.getNextSlice());
        }else{
           // handle the error code here.
        }
    }
}

Answer 3

I think this socket.io project has a lot of potential:

https://github.com/sffc/socketio-file-upload

It supports chunked upload, progress tracking and seems fairly easy to use.

Answer 4

Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice().

This article helps you to handle large files in workers. change XHR send to Websocket in main thread.

//Messages from worker
function onmessage(blobOrFile) {
 ws.send(blobOrFile);
}

//construct file on server side based on blob or chunk information.

Answer 5

I believe the send() method is asynchronous which is why it will return immediately. To make it queue, you'd need the server to send a message back to the client after each slice is uploaded; the client can then decide whether it needs to send the next slice or a "upload complete" message back to the server.

This sort of thing would probably be easier using XMLHttpRequest(2); it has callback support built-in and is also more widely supported than the WebSocket API.

Answer 6

You could use https://github.com/binaryjs/binaryjs or https://github.com/liamks/Delivery.js if you can run node.js on the server.