raw decoder for protobufs format

Question

I\'d like to find a way to convert a binary protobuf message into a human readable description of the contained data, without using the .proto files.

The background

Answer 1

You could try forcing it through the wireshark plugin, or you could probably borrow the "reader" part of some of the implementations (I know how I would do this in C#, for example, but I doubt that is what you meant).

However, be cautious - string in protocol buffers doesn't really mean "string" - it could be:

• a UTF-8 string
• a raw BLOB of arbitrary data
• a sub-message
• a "packed" array
• (probably something else I'm forgetting)

Answer 2

If you happen to have a binary file containing (multiple?) length-prefixed protobuf messages, protoc ‒‒decode_raw < file cannot parse it because of the length prefixes. A simple way around that is to split the file into its consecutive messages and then convert each with protoc.

My take:

var fs = File.OpenRead(filename));
var buffer = new byte[4096];
int size;
for (int part = 1; Serializer.TryReadLengthPrefix(fs, PrefixStyle.Base128, out size); part++) {
  long startPosition = fs.Position;
  using (var writer = File.OpenWrite(string.Format("{0}[{1}].pb", filename, part))) {
    for (int bytesToRead = size; bytesToRead > 0; ) {
      int bytesRead = fs.Read(buffer, 0, Math.Min(bytesToRead, buffer.Length));
      bytesToRead -= bytesRead;
      if (bytesRead <= 0) // End of file.
        break;
      writer.Write(buffer, 0, bytesRead);
    }
  }
}

Answer 3

For Posterity: Google's protocol buffer tools have the ability to decode raw buffers.

Just send the unknown buffer to it and pass the--decode_raw flag

$ cat has_no_proto.buff | protoc --decode_raw
2 {
  2: "Error retrieving information from server. [RH-02]"
}

So here's a message with field 2 set to an embedded message which in turn has its second field set to a string telling me I pissed off Google Play.

Type information isn't definite (it looks like it will try to display all binary data as strings -- but your requirement for varint/string/submessage distinction is met).

Answer 4

As noted in Michel de Ruiter's answer, it's possible that your protobuf message has a length-prefix. Assuming it does, this answer should help.

(NOTE: For most of the commands below, I'm assuming your protobuf message is stored in a file called input.)

protoc --decode_raw + dd for a single message:

If it's simply a single message, then you can indeed leverage protoc --decode_raw, but you need to strip off the length-prefix header first. Assuming the header is 4 bytes long you can use dd to strip the header off of input and then feed the output into protoc.

dd bs=1 skip=4 if=input 2>/dev/null | protoc --decode_raw

protoc-decode-lenprefix --decode_raw for a single message:

I also wrote a script that handles the header stripping automatically:

protoc-decode-lenprefix --decode_raw < input

This script is simply a wrapper on top of protoc --decode_raw, but handles parsing out the length-prefix header and then invoking protoc.

Now, this script isn't terribly useful in this case, because we can just use the dd trick above to strip the header off. However, say we have a data stream (e.g., a file or TCP stream) containing multiple messages that are framed with length-prefix headers....

protoc-decode-lenprefix --decode_raw for a stream of messages:

Instead of a single protobuf message in the input file, lets say input contained multiple protobuf messages which are framed by length-prefix headers. In this case its not possible to just use the dd trick, because you need to actually read the contents of the length-prefix header to determine how long the subsequent message in the stream is, and thus how many bytes ahead the next header+message lies. So instead of worrying about all of that, you can simply use protoc-decode-lenprefix again:

protoc-decode-lenprefix --decode_raw < input

protoc-decode-lenprefix --decode ... foo.proto for a stream of messages

This script can also be used to fully decode length-prefixed messages (instead of just "raw decode" them). It assumes you have access to the .proto files that define the protobuf message, just like the wrapped protoc command. The invocation syntax is identical to protoc --decode. For example, using the dd trick with protoc --decode, along with input being a Mesos task.info file, the syntax looks like this:

dd bs=1 skip=4 if=task.info 2>/dev/null | \
protoc --decode mesos.internal.Task \
                      -I MESOS_CODE/src -I MESOS_CODE/include \
                      MESOS_CODE/src/messages/messages.proto

And the parameters are identical when using protoc-decode-lenprefix

cat task.info | \
protoc-decode-lenprefix --decode mesos.internal.Task \
                      -I MESOS_CODE/src -I MESOS_CODE/include \
                      MESOS_CODE/src/messages/messages.proto