I am trying to read 12-bit binary files containing images (a video) using Python 3.
To read a similar file but encoded in 16 bits, the following works very well:
import numpy as np
images = np.memmap(filename_video, dtype=np.uint16, mode='r', shape=(nb_frames, height, width))
where filename_video is the file and nb_frames, height, and width characteristics of the video that can be read from another file. By 'working very well' I mean fast: reading a 640x256 video that has 140 frames takes about 1 ms on my computer.
As far as I know I cannot use this when the file is encoded in 12 bits because there is no uint12 type. So what I am trying to do is to read a 12-bit file and store it in a 16-bit uint array. The following, taken from (Python: reading 12 bit packed binary image), works:
with open(filename_video, 'rb') as f:
data=f.read()
images=np.zeros(int(2*len(data)/3),dtype=np.uint16)
ii=0
for jj in range(0,int(len(data))-2,3):
a=bitstring.Bits(bytes=data[jj:jj+3],length=24)
images[ii],images[ii+1] = a.unpack('uint:12,uint:12')
ii=ii+2
images = np.reshape(images,(nb_frames,height,width))
However, this is very slow: reading a 640x256 video thas has only 5 frames takes about 11.5 s with my machine. Ideally I would like to be able to read 12-bit files as efficiently as I can read 8 or 16-bit files using memmap. Or at least not 10^5 times slower. How could I speed things up ?
Here is a file example: http://s000.tinyupload.com/index.php?file_id=26973488795334213426 (nb_frames=5, height=256, width=640).
I have a slightly different implementation from the one proposed by @max9111 that doesn't require a call to unpackbits
.
It creates two uint12
values from three consecutive uint8
directly by cutting the middle byte in half and using numpy's binary operations. In the following, data_chunks
is assumed to be a binary string containing the information for an arbitrary number number of 12-bit integers (hence its length must be a multiple of 3).
def read_uint12(data_chunk):
data = np.frombuffer(data_chunk, dtype=np.uint8)
fst_uint8, mid_uint8, lst_uint8 = np.reshape(data, (data.shape[0] // 3, 3)).astype(np.uint16).T
fst_uint12 = (fst_uint8 << 4) + (mid_uint8 >> 4)
snd_uint12 = ((mid_uint8 % 16) << 8) + lst_uint8
return np.reshape(np.concatenate((fst_uint12[:, None], snd_uint12[:, None]), axis=1), 2 * fst_uint12.shape[0])
I benchmarked with the other implementation and this approach proved to be ~4x faster on a ~5 Mb input:read_uint12_unpackbits
65.5 ms ± 1.11 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
read_uint12
14 ms ± 513 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Edit
With the help of @Cyril Gaudefroy answer, I created a compiled solution using Numba
.
import numba as nb
import numpy as np
@nb.njit(nb.uint16[::1](nb.uint8[::1]),fastmath=True,parallel=True)
def nb_read_uint12(data_chunk):
"""data_chunk is a contigous 1D array of uint8 data)
eg.data_chunk = np.frombuffer(data_chunk, dtype=np.uint8)"""
#ensure that the data_chunk has the right length
assert np.mod(data_chunk.shape[0],3)==0
out=np.empty(data_chunk.shape[0]//3*2,dtype=np.uint16)
for i in nb.prange(data_chunk.shape[0]//3):
fst_uint8=np.uint16(data_chunk[i*3])
mid_uint8=np.uint16(data_chunk[i*3+1])
lst_uint8=np.uint16(data_chunk[i*3+2])
out[i*2] = (fst_uint8 << 4) + (mid_uint8 >> 4)
out[i*2+1] = ((mid_uint8 % 16) << 8) + lst_uint8
return out
Timings
num_Frames=10
data_chunk=np.random.randint(low=0,high=255,size=np.int(640*256*1.5*num_Frames),dtype=np.uint8)
Cyril Gaudefroy(numpy only): 11ms ->225MB/s
Numba version: 1.1ms ->2.25GB/s
Previous version (not recommended)
If Numba is not an option see @Cyril Gaudefroys answer.
When i read the question, i thought there must be a simple answer, but I failed. Nevertheless i wrote a simple (but ugly) code that is about 300 times faster than your example and achieves about 25 MB/s on my Notebook (Core i5 3210M).
def read_uint12(filename_video,nb_frames,height,width):
data=np.fromfile(filename_video, dtype=np.uint8)
data=np.unpackbits(data)
data=data.reshape((data.shape[0]/12,12))
images=np.zeros(data_2.shape[0],dtype=np.uint16)
for i in xrange(0,12):
images+=2**i*data[:,11-i]
images = np.reshape(images,(nb_frames,height,width))
return images
Found @cyrilgaudefroy answer useful. However, initially, it did not work on my 12-bit packed binary image data. Found out that the packing is a bit different in this particular case. The "middle" byte contained the least significant nibbles. Bytes 1 and 3 of the triplet are the most significant 8 bits of the twelve. Hence modified @cyrilgaudefroy answer to:
def read_uint12(data_chunk):
data = np.frombuffer(data_chunk, dtype=np.uint8)
fst_uint8, mid_uint8, lst_uint8 = np.reshape(data, (data.shape[0] // 3, 3)).astype(np.uint16).T
fst_uint12 = (fst_uint8 << 4) + (mid_uint8 >> 4)
snd_uint12 = (lst_uint8 << 4) + (np.bitwise_and(15, mid_uint8))
return np.reshape(np.concatenate((fst_uint12[:, None], snd_uint12[:, None]), axis=1), 2 * fst_uint12.shape[0])
Here's yet another variation. My data format is:
first uint12: most significant 4 bits from least significant 4 bits of second uint8 + least significant 8 bits from first uint8
second uint12: most significant 8 bits from third uint8 + least significant 4 bits from most significant 4 bits from second uint8
The corresponding code is:
def read_uint12(data_chunk):
data = np.frombuffer(data_chunk, dtype=np.uint8)
fst_uint8, mid_uint8, lst_uint8 = numpy.reshape(data, (data.shape[0] // 3, 3)).astype(numpy.uint16).T
fst_uint12 = ((mid_uint8 & 0x0F) << 8) | fst_uint8
snd_uint12 = (lst_uint8 << 4) | ((mid_uint8 & 0xF0) >> 4)
return numpy.reshape(numpy.concatenate((fst_uint12[:, None], snd_uint12[:, None]), axis=1), 2 * fst_uint12.shape[0])
来源:https://stackoverflow.com/questions/44735756/python-reading-12-bit-binary-files