How to parse and validate a WebSocket frame in Java?

Question

I wrote a WebSocket frame decoder in Java:

private byte[] decodeFrame(byte[] _rawIn) {
        int maskIndex = 2;
        byte[] maskBytes = new byte[4];

           


        

Answer 1

The first safeguard against an application connecting to a websocket server it wasn't designed for is the HTTP websocket handshake. When it doesn't include an Upgrade: websocket, Sec-WebSocket-Key or Sec-WebSocket-Version: 13 it isn't even a RFC6455 websocket client and must be rejected.

The second safeguard works against clients which are speaking websocket, but were designed for a different application. This is the Sec-WebSocket-Protocol: something header. This header is optional, but should be a string which identifies the application the client wants to use. When the value doesn't match the application(s) the server expects, the client should be rejected.

The last saveguard against clients which think that they speak websocket and connect to the right server but actually have a bug in their websocket protocol implementation are the reserved bits.

There are no illegal values for the masking key or length. A wrong length will cause the next frame to begin after interpreting not enough or too much data as payload, but this can be hard to detect. The only sign that this has happened, is when the first byte of an alleged frame doesn't make sense.

The 2nd, 3rd and 4th bit of a frame are reserved and, according to the RFC "MUST be 0 unless an extension is negotiated [...] If a nonzero value is received [...] the receiving endpoint MUST Fail the WebSocket Connection.". There are no extensions yet which use these bits, and when there will ever be one, you will have to do something to switch it on. So when one of these bits is non-zero, something is wrong.

When you want, you can add further safeguards on your protocol level, like a specific magical byte-value every message has to start and/or end with (keep in mind that there are multi-fragment messages and a browser can use this when it feels like doing so). The application I develop at the moment uses JSON payloads, so when a message isn't a valid JSON string starting with { and ending with }, I know the client is broken (or my servers frame decoding method is, which is far more likely).