How to unpack CHIME/FRB data files?

问题

Depiction of a Fast Radio Burst

I have an issue with the msgpack.py which has to be used to read and uncompress the msgpack data. Here is the location of the files which I'm trying to uncompress CHIME/FRB data files. Here is additional information if you can help me. PyPi msgpack.

Does anyone here have experience in trying to read the CHIME/FRB data files and if so can you explain the process?

 import msgpack
 from io import BytesIO

 buf = BytesIO()
 for i in range(100):
   buf.write(msgpack.packb(i, use_bin_type=True))

 buf.seek(0)

 unpacker = msgpack.Unpacker(buf, raw=False)
 for unpacked in unpacker:
    print(unpacked)

回答1:

Actually, there is no difficulties to unpack the "astro" data. The only difficulty is that the data is huge... and contains other (unspecified) binary data.

Here is how I successfully unpack the file "astro_10889573_20180814144950703901_beam1180_00161339_01.msgpack":

import pprint
import msgpack

data_msgpack = "astro_10889573_20180814144950703901_beam1180_00161339_01.msgpack"
with open(data_msgpack, mode="rb") as fd:
    data = msgpack.unpack(fd)

pprint.pprint(data)

The starting of the output is:

['assembled_chunk in msgpack format',
 1,
 0,
 16777216,
 1180,
 16,
 16,
 384,
 64,
 65536,
 16777216,
 63441076224,
 393216,
 1,
 b'T\xb2\xc1?z\xd1\xd4?bs\xd1?\xed\xf5\xc4?\xaf#\xc7?`\x92\xd4?\xc7L\xd5?'
 b'\x00\x15\xcb?\x14\xb0\xbf?M\xec\xd2?6\xea\xd1?\xe1N\xcb?*\x12\xca?'
 b'\x9e\xf1\xd2?\xa39\xd3?\xab\x9d\xe4?z\xb2\xd3?\xcdg\xdd?\xedc\xdc?:\xd4\xd4?'
 b'"\xbf\xdb?A$\xd1?))\xd0?EE\xc1?o \xcb?g$\xca?\xd7\x7f\xce?\xe3\x9e\xde?'
 b'J\xa0\xdc?O\xef\xd7?\x07P\xd9?(a\xe0?\xa8V\xdb?;\xb9\xe0?\xdbY\xd6?'
 b'\x8a\xb8\xe4?eI\xe5?\x0eu\xea?\xca\xac\xec?\xb8\xe1\xcf?\xa0\x97\xd2?'
 b'\xf1\xd9\xdb?\xa7I\xd8?\xde\xb5\xc1?\xb6\x02\xc2?$\x08\xc6?1\x84\xcb?'
 b'\x1d\x84\xcb?\x94\xb2\xd3?M=\xd9?\xd5\x87\xdb?\xcc\xea\xce?M\xd7\xcb?|9\xda?'
 b'\xc6\x02\xda?\x04-\xdb?\x99\xf1\xcf?\\\xa3\xdb?\xf39\xde?\xce\x9a\xd3?'
 b'm\xec\xe2?j\xac\xdd?\xdf\xa8\xd3?Gj\xd0?H\xe6\x0b?\xd2I\n?\x7f\x01\x00?'
 b'\xa9\xfb\xf4>\xdb\xf1\xf1>\xe0\xff\x03?\x9c\xb9\x04?\xf7\x08\xef>'
 b'\xc4\x84\x05?TZ\xeb>\xd0\x1c\x01?\xb0.\xfb>\xfd \x08?\x8f~\xf8>\xef\x19\xff>'
 b'<\x9e\t?0\xd0\xdb>\x89+\xdd>R\xad\x05?\x03*\x00?Z\xd0\x08?M\x82\x17?Ba\xf1>'
 b'\xd4i\xf1>\xa7\xeb\xef>\x83\x13\x0c?Hq\xf4>\xf51\x1a?\x815\xf8>\x94\xd7\x01?'
 b'E\xb8\x07?/\xfa\t?\x83\x9a\xee>\xd3\xce\xf3>\x94Q\xfe>9\t\xd9>\xf9n\x02?'
 b"\x81'\xf3>}O\xef>\n\xd0\x0c?\x04\n\x08?5\xff\xdf>\xb1\xb1\x0c?\x9d\xda\x06?"
 b'\x8fO\xf5>>\xa7\xf3>\x13\xc9\x0c?\xc1B\x06?t\xbd\xea>{\xc2\x01?\x00\x0c\x08?'
 b'\x1e\xf3\x03?\x98\xf3\xfa>\xcfn\n?c\x08\x04?pV\x04?3Y\x17?\xf5\x86\xf4>'
 b'\xc6\xd2\xec>\x9b\x9c\x04?Y\xed\xf2>\x08\x95\x00?\xbe\x15\t?\xc2H\xf0>'
 b'%&\x1b?$\x0e\x0b?\x03$\x07?\xd3R(?\x1a\xc4\xfe>O!\x0f?\x93](?p\x07\x13?'
 b"\xe2U\x06?\t\x89'?\x95\xf6\x1f?\x06\xed\r?\xbag\x04?w\xd2\x07?7\xe9\x1a?"
[...]

来源：https://stackoverflow.com/questions/61169429/how-to-unpack-chime-frb-data-files

标签

python

msgpack