Convert array of bytes to base128 valid JSON string

Question

I want to send big array of bytes using JSON (I was inspired by this question), to have small overhead I wana to use base128 encoding (which can in fact produce valid json s

Answer 1

ES6:

Encode

let bytesToBase128 = (bytesArr) => {
    // 128 characters to encode as json-string
    let c= "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz¼½ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ" 
    let fbits=[]; 
    let bits = (n,b=8) => [...Array(b)].map((x,i)=>n>>i&1); 
    bytesArr.map(x=> fbits.push(...bits(x)));

    let fout=[]; 
    for(let i =0; i<fbits.length/7; i++) { 
        fout.push(parseInt(fbits.slice(i*7, i*7+7).reverse().join(''),2))  
    }; 

    return (fout.map(x => c[x])).join('');
}

// Example
// bytesToBase128([23, 45, 65, 129, 254, 42, 1, 255]) => "NÚ4AèßÊ0ÿ1"

Decode

let base128ToBytes = (base128str) => {
    // 128 characters to encode as json-string
    let c= "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz¼½ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ" 

    dfout = base128str.split('').map(x=>c.indexOf(x));
    let dfbits = [];
    let bits = (n,b=8) => [...Array(b)].map((x,i)=>n>>i&1);
    dfout.map(x=> dfbits.push(...bits(x,7) ));

    let dfbytes=[]; 
    let m1 = dfbits.length%8 ? 1 : 0;
    for(let i =0; i<dfbits.length/8-m1; i++) { 
        dfbytes.push(parseInt(dfbits.slice(i*8, i*8+8).reverse().join(''),2))  
    }; 

    return dfbytes;
}

// Example
// base128ToBytes("NÚ4AèßÊ0ÿ1") => [23, 45, 65, 129, 254, 42, 1, 255]

I embeded here bits function - here. The coversion idea here is to convert bytes array to bit array and then take each 7 bits (the value is from 0 to 127) as character number i charakter list c. In decoding we change each character number 7-bit number and create array, and then take each 8-bit packages of this array and interpret them as byte.

To view characters from ASCI and choose 128 from them (which is arbitrary) I type in console

[...Array(256)].map((x,i) => String.fromCharCode(i)).join('');

I try to avoid characters that has "special meaning" in different contexts like ! @ # $ % ' & ...

And here is working example (which convert Float32Array to json).

Tested on Chrome, Firefox and Safari

Conclusion

After conversion bytes array to base128 string (which is valid json) the output string is less than 15% bigger than input array.

Update

A dig a little bit more, and expose that when we send characters which codes are bigger than 128 (¼½ÀÁÂÃÄ...) then chrome in fact send TWO characters (bytes) instead one :( - I made test in this way: type in url bar chrome://net-internals/#events (and send POST request) and in URL_REQUEST> HTTP_STREAM_REQUEST > UPLOAD_DATA_STREAM_INIT > total_size we see that request are two times bigger when body contains charaters witch codes bigger than 128. So in fact we don't have profit with sending this characters :( . For base64 strings we not observe such negative behaviour - However I left this procedures because they may bye used in other purposes than sending (like better alternative to storage binary data in localstorage than base64 - however probably there exists even better ways...?). UPDATE 2019 here.