问题
I have been using protobuf-net to send some objects over the wire and everything has been working nicely until now. However, I have come across a particular instantiation of my class that fails to deserialize when running under mono. The exact same object deserializes correctly running under .net. I have verified that it is the exact same byte[] that I receive over the wire when I run under mono and under .net by checking the md5 sum. This would indicate that the problem must be with protobuf-net deserialization. Here is the code I am using to deserialize the byte[]:
using (MemoryStream ms = new MemoryStream(serializedByteArray))
{
return (MyProtoBufDto)Serializer.Deserialize<MyProtoBufDto>(ms);
}
and here is the exception I am getting:
System.IO.EndOfStreamException: Failed to read past end of stream.
at ProtoBuf.ProtoReader.Ensure (int,bool) <0x00167>
at ProtoBuf.ProtoReader.ReadString () <0x0005b>
at (wrapper dynamic-method) System.Collections.Generic.KeyValuePair`2<string, string>.proto_18 (object,ProtoBuf.ProtoReader) <0x000c8>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.ProtoReader.ReadTypedObject (object,int,ProtoBuf.ProtoReader,System.Type) <0x00056>
at ProtoBuf.ProtoReader.ReadObject (object,int,ProtoBuf.ProtoReader) <0x0001b>
at (wrapper dynamic-method) System.Collections.Generic.KeyValuePair`2<string, System.Collections.Generic.List`1<System.Collections.Generic.KeyValuePair`2<string, string>>>.proto_16 (object,ProtoBuf.ProtoReader) <0x00220>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.ProtoReader.ReadTypedObject (object,int,ProtoBuf.ProtoReader,System.Type) <0x00056>
at ProtoBuf.ProtoReader.ReadObject (object,int,ProtoBuf.ProtoReader) <0x0001b>
at (wrapper dynamic-method) MyGroupDto.proto_14 (object,ProtoBuf.ProtoReader) <0x001bc>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.ProtoReader.ReadTypedObject (object,int,ProtoBuf.ProtoReader,System.Type) <0x00056>
at ProtoBuf.ProtoReader.ReadObject (object,int,ProtoBuf.ProtoReader) <0x0001b>
at (wrapper dynamic-method) System.Collections.Generic.KeyValuePair`2<MyGroupNameDto, MyGroupDto>.proto_12 (object,ProtoBuf.ProtoReader) <0x00197>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.ProtoReader.ReadTypedObject (object,int,ProtoBuf.ProtoReader,System.Type) <0x00056>
at ProtoBuf.ProtoReader.ReadObject (object,int,ProtoBuf.ProtoReader) <0x0001b>
at (wrapper dynamic-method) MyResultProtoBufDto.proto_8 (object,ProtoBuf.ProtoReader) <0x002b9>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.ProtoReader.ReadTypedObject (object,int,ProtoBuf.ProtoReader,System.Type) <0x00056>
at ProtoBuf.ProtoReader.ReadObject (object,int,ProtoBuf.ProtoReader) <0x0001b>
at (wrapper dynamic-method) MyProtoBufDto.proto_6 (object,ProtoBuf.ProtoReader) <0x00116>
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) <0x0002d>
at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) <0x00112>
at ProtoBuf.Meta.TypeModel.DeserializeCore (ProtoBuf.ProtoReader,System.Type,object,bool) <0x0005b>
at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type,ProtoBuf.SerializationContext) <0x00097>
at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type) <0x0001f>
at ProtoBuf.Serializer.Deserialize<MyProtoBufDto> (System.IO.Stream) <0x00033>
at MyClient.Serialization.ProtoBufSerialization.DecompressAndDeserialize<MyProtoBufDto> (byte[]) <0x0008b>
I have tried mono 2.10.9 and the latest mono 3.2.3 and have received the same exception. I am using the latest protobuf-net dll version (2.0.0.666). I tried the dll from protobuf-net-mono directory (compiled with mono compiler?) and the regular .net compiled version but keep getting this exception when running under mono.
Details of the MyProtoBufDto class:
[DataContract]
public class MyProtoBufDto
{
[DataMember(Order = 1)]
public List<MyResultProtoBufDto> Result { get; set; }
}
[DataContract]
public class MyResultProtoBufDto
{
[DataMember(Order = 1)]
public Dictionary<MyGroupNameDto, MyGroupDto> Groups { get; set; }
}
[DataContract]
public class MyGroupDto
{
[DataMember(Order = 1)]
public Dictionary<string, List<KeyValuePair<string, string>>> Group { get; set; }
}
MyGroupNameDto is just an enum
More Info: This is using mono on linux. I have not tested mono on windows yet. I verified both the byte[] off the wire and the byte[] after decompression is identical under mono and on windows so we should be feeding protobuf-net the same exact data.
Update We made a change on the server side to always remove empty collections from the objects that are serialized and since then have not encountered this error. I understand that protobuf does not make a distinction between empty collections and null collections, but it is still peculiar that somehow the behavior was different on mono vs .net and it triggered this error.
回答1:
That is intriguing. Can I ask: mono on what? mono on windows? mono on linux? I can try to repro, but anything you can give me to make sure that we're looking at the same thing would really help me. Indeed, if possible what would really help is "and here's the base-64 of the data I am trying to deserialize" (i.e. serializedByteArray) - which would allow me to very quickly determine whether it is the data vs the deserializer at fault. Alternatively, maybe some "here's some code that generates some invented data that then doesn't deserialize" sample?
To explain what the error means: something has declared "I need {n} bytes" - in this case ReadString. It tries to load this into an internal buffer, and the Stream doesn't give it enough data.
I'm also slightly concerned by DecompressAndDeserialize in the call stack: there is of course nothing whatsoever wrong with compression - but when you are doing your "did we get the right data" checks, is that before you decompress? or after? One possibility is that it is the decompression layer that is producing different results.
Happy to investigate; but it would save a lot of time if you can supply more context / sample - if that can't be done on a public site, then maybe by email etc?
回答2:
Are both endpoints running on a .NET environment with protobuf-net, or is one of them different (e.g. using the C++ library)?
I've hit the same problem once when deserializing a message using protobuf-net (Windows) that was sent from a C++ application (Linux) via sockets.
My message looked like the following:
message Envelope
{
required int32 type = 1;
required bytes data = 2;
}
On the C++ side I was setting 'data' (an embedded message) without passing in the size, i.e.
envelope->set_data(buf);
instead of
envelope->set_data(buf, bufSz);
Which meant not only the size of the serialized message was wrong (envelope->ByteSize()) but the buffer also contained rubbish at the end, which was probably why protobuf-net interpreted that as extra fields and kept on parsing beyond what it should.
Worth checking if it's something along the same lines.
回答3:
I was able to figure out the cause of this weird problem. Apparently I made an error when debugging this problem originally. I had thought the byte[] I was sending for deserialization was identical when running in .net and under mono. However, this was not the case and the error was with the use of DeflateStream in conjunction with deserialization. When running under mono the DeflateStream was not return all the bytes that protobuf-net was expecting.
The broken code looked like this:
using (MemoryStream ms = new MemoryStream(compressedByteArray))
using (DeflateStream ds = new DeflateStream(ms, CompressionMode.Decompress, true))
return (MyProtoBufDto)Serializer.Deserialize<MyProtoBufDto>(ds);
The working code looks like this:
using (MemoryStream msDecompressed = new MemoryStream())
{
using (MemoryStream ms = new MemoryStream(compressedByteArray))
using (DeflateStream ds = new DeflateStream(ms , CompressionMode.Decompress, true))
ds.CopyTo(msDecompressed);
msDecompressed.Seek(0, SeekOrigin.Begin);
return (MyProtoBufDto)Serializer.Deserialize<MyProtoBufDto>(msDecompressed);
}
It's too bad that this difference between mono and .net exists. I understand that DeflateStream uses underlying buffers. Mono is probably using the lazy implementation of only returning the decompressed bytes in the buffer even though more bytes are requested. As long as DeflateStream.ReadBytes returns at least 1 byte, this technically might not break any .net specifications. A change to protobuf-net that continues reading bytes from the stream when it does not get the expected number back would probably fix this issue. Only if ReadBytes returns zero bytes should protobuf-net throw the exception we are seeing.
来源：https://stackoverflow.com/questions/19018627/protobuf-net-deserialization-system-io-endofstreamexception-under-mono