Lua, dealing with non-ascii byte streams, byteorder change

Question

Need to encode & decode byte-stream (containing non-ascii characters possibly), from/into uint16, uint32, uint64 (their typical C/C++ meaning), taking care of endianness

Answer 1

my suggestion for an "Int16ToByte"-function without checking of parameters:

function Int16ToBytes(num, endian)
  if num < 0 then 
      num = num & 0xFFFF
  end

  highByte = (num & 0xFF00) >> 8
  lowByte  = num & 0xFF

  if endian == "little" then
      lowByte, highByte = highByte, lowByte
  end

  return string.char(highByte,lowByte)
end

Answer 2

for converting from bytes to int (taking care of endianness at byte level, and signedness):

function bytes_to_int(str,endian,signed) -- use length of string to determine 8,16,32,64 bits
    local t={str:byte(1,-1)}
    if endian=="big" then --reverse bytes
        local tt={}
        for k=1,#t do
            tt[#t-k+1]=t[k]
        end
        t=tt
    end
    local n=0
    for k=1,#t do
        n=n+t[k]*2^((k-1)*8)
    end
    if signed then
        n = (n > 2^(#t*8-1) -1) and (n - 2^(#t*8)) or n -- if last bit set, negative.
    end
    return n
end

And while we're at it also the other direction:

function int_to_bytes(num,endian,signed)
    if num<0 and not signed then num=-num print"warning, dropping sign from number converting to unsigned" end
    local res={}
    local n = math.ceil(select(2,math.frexp(num))/8) -- number of bytes to be used.
    if signed and num < 0 then
        num = num + 2^n
    end
    for k=n,1,-1 do -- 256 = 2^8 bits per char.
        local mul=2^(8*(k-1))
        res[k]=math.floor(num/mul)
        num=num-res[k]*mul
    end
    assert(num==0)
    if endian == "big" then
        local t={}
        for k=1,n do
            t[k]=res[n-k+1]
        end
        res=t
    end
    return string.char(unpack(res))
end

Any remarks are welcome, it's tested, but not too thoroughly...

Answer 3

Take a look at the struct and lpack libraries.

In this example, I use the struct.unpack to decode a Lua string into two integers with forced little-endian encoding:

require 'struct'
-- convert character codes to a Lua string - this may come from your source
local str = string.char(0x00, 0x1d, 0xff, 0x23, 0x44, 0x32)
-- format string: < = little endian, In = unsigned int (n bytes)
local u16, u32 = struct.unpack('<I2I4', str)
print(u16, u32) --> 7424    843326463