Packing data of different sizes into a list of unsigned ints

Question

I have a set of data that represents a hardware structure that I need to manipulate in python. The real structure is 4 KB in size...I'll just whip up a quick example:

Byte(s)  Value  
0-1      0x0102
2-3      0x0304
4-23     "AStrWith20Characters"
24-63    "WoahThisStringHas40CharactersItIsHuge!!!"
64-71    "Only8Chr"
72-74    0x050607
74       0x08
75-127   0x00 (padding)

The idea is I pack this structure up into a list of 32 bit Ints, pass the list off to another function, and that function then writes the whole shebang out to memory. The Memory Writing function accepts 64 Bytes at a time, so I'd have to make two calls.

So, for the example above, I'd need to make these calls (I'll use big endian for readability):

WrToMemory([0x01020304, 0x41537472, 0x57697468, 0x3230436, 
            0x61726163, 0x74657273, 0x576F6168, 0x5468697, 
            0x53747269, 0x6E674861, 0x73343043, 0x6861726, 
            0x63746572, 0x73497449, 0x73487567, 0x65212121])

WrToMemory([0x4F6E6C79, 0x38436872, 0x05060708, 0x00000000,  
            0x00000000, 0x00000000, 0x00000000, 0x00000000,  
            0x00000000, 0x00000000, 0x00000000, 0x00000000,  
            0x00000000, 0x00000000, 0x00000000, 0x00000000])    

The problem is that I don't know how to pack this data in any kind of efficient way. I have been struggling with using struct and array, but either I am missing something conceptually or I am simply using the wrong tool for the job.

Ultimately, I know I can write a big hairy function that checks data types, does a bunch of shifting and masking, converts ASCII to hex, and the constructs data packets I need. But that seems...inelegant.

I figure there's got to be a way to tell python "This is a number that is two bytes long. Pack it up. Another two byte number, append it to the last one. Now this is a string, with 20 bytes. Append it. Go on until we have 64 bytes. Then convert the list of 64 bytes we just made into a lsit of 16 unsigned integers.

It seems like the pack and unpack functions are what I should be using, but like I said above...I must be missing something. How does pack handle the 3 byte number, for example? If you use a format string like HB, then pack expects two arguments:

>>> pack('HB', 0x050607)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: pack requires exactly 2 arguments

And I can't use unpack, because that expects a string, not a number.

Hope I've made my problem clear. Any help is much appreciated!

Answer 1

You are using the right tool, but you told pack to encode 2 arguments, but pass in only one.

The string HB means: pack two values, one as an unsigned short, the other as an unsigned byte. Pass in two values and it works:

>>> pack('HB', 0x0506, 0x07)
'\x06\x05\x07'

Just keep experimenting, you'll get the hang of it. Here are the first set of values from your example, in Big Endian notation:

>>> pack('>2H20c40c', 0x0102, 0x0304)
'\x01\x02\x03\x04AStrWith20CharactersWoahThisStringHas40CharactersItIsHuge!!!'

Note the '>' at the start, this signals the endianess. The strings are converted to sequences and applied as individual variables. Generally, it's easier to just append them though:

>>> pack('>2H20c40c', 0x0102, 0x0304) + "AStrWith20Characters" + "WoahThisStringHas40CharactersItIsHuge!!!"
'\x01\x02\x03\x04AStrWith20CharactersWoahThisStringHas40CharactersItIsHuge!!!'

Answer 2

You can pack the whole thing using your struct format, and unpack it using the e.g. 32-bit words format:

>>> format="cchi"
>>> p=pack(format,"a","b",1,2)
>>> p
'ab\x01\x00\x02\x00\x00\x00'
>>> (z1,z2)=unpack("ii",p)
>>> (z1,z2)
(90721, 2)