Websocket frame size limitation

Question

I'm sending huge chunks of JSON data through websockets. The JSON may have over 1000 entries. Due to the frame size limitation, the Websocket protocol automatically splits the JSON into frames, which cannot be helped. As we cannot change the frame size of websockets.

The problem:

When I try to evaluate my JSON using JSON.parse it gives me a parsing error which is obvious because the frames are not complete JSON objects. All this happens in the Websocket onmessage event callback. How can I recieve the huge JSON in differnt frames and still be able to parse it? I have tried to concat the frames in onmessage still the error persists.

Side question:

How to concatinate a broken JSON properly?

Answer 1

A single WebSocket frame, per RFC-6455 base framing, has a maximum size limit of 2^63 bytes (9,223,372,036,854,775,807 bytes ~= 9.22 exabytes) (correction by @Sebastian)

However, a WebSocket message, made up of 1 or more frames, has no limit imposed on it from the protocol level.

Each WebSocket implementation will handle message and frame limits differently. Such as setting maximum messages sizes for whole message (usually for memory consumption reasons), or offering streaming options for large messages to better utilize memory.

But in your case, it is likely that your chosen WebSocket implementation has a bug and is improperly splitting up the JSON message into multiple messages, instead of multiple frames. You can use the network inspection tooling in Chrome or an external tool like Wireshark to confirm this behavior.

Answer 2

Since you are dealing with WS, which is low-level, you need to create an application protocol that deals with data that is sent over multiple WS frames. It is up to you to concatenate the data that is in each WS frame (btw, don't concat the frames... concat the data in each frame).

Basically you are reinventing a file transfer protocol.

Answer 3

var wsServer = new websocket.server({
            httpServer: server,
            maxReceivedFrameSize: 131072,
            maxReceivedMessageSize: 10 * 1024 * 1024,
            autoAcceptConnections: false
        });

Change the default maxFrameSize & MessageSize