Embedding binary data in a script efficiently

Question

I have seen some installation files (huge ones, install.sh for Matlab or Mathematica, for example) for Unix-like systems, they must have embedded quite a lot of binary data,

Answer 1

You can use base64 + compression (using bz2 for instance) if that suits your data (e.g., if you're not embedding already compressed data).

For instance, to create your data (say your data consist of 100 null bytes followed by 200 bytes with value 0x01):

>>> import bz2
>>> bz2.compress(b'\x00' * 100 + b'\x01' * 200).encode('base64').replace('\n', '')
'QlpoOTFBWSZTWcl9Q1UAAABBBGAAQAAEACAAIZpoM00SrccXckU4UJDJfUNV'

And to use it (in your script) to write the data to a file:

import bz2
data = 'QlpoOTFBWSZTWcl9Q1UAAABBBGAAQAAEACAAIZpoM00SrccXckU4UJDJfUNV'
with open('/tmp/testfile', 'w') as fdesc:
    fdesc.write(bz2.decompress(data.decode('base64')))