Speed up python's struct.unpack

Question

I am trying to speed up my script. It basically reads a pcap file with Velodyne\'s Lidar HDL-32 information and allows me to get X, Y, Z, and Intensity values. I have profiled m

Answer 1

Numpy lets you do this very quickly. In this case I think the easiest way is to use the ndarray constructor directly:

import numpy as np

def with_numpy(buffer):
    # Construct ndarray with: shape, dtype, buffer, offset, strides.
    rotational = np.ndarray((firingBlocks,), '

This returns separate arrays instead of the nested list, which should be much easier to process further. As input it takes a buffer object (in Python 2) or anything that exposes the buffer interface. Unfortunately, it depends on your Python version (2/3) what objects you can use exactly. But this method is very fast:

import numpy as np

firingBlocks = 10**4
lasers = 32
packet_raw = np.random.bytes(42 + firingBlocks*100)

%timeit readDataPacket(memoryview(packet_raw))
# 1 loop, best of 3: 807 ms per loop
%timeit with_numpy(packet_raw)
# 100 loops, best of 3: 10.8 ms per loop