How to serialize/deserialize large list of items with protobuf-net

前端 未结 3 1120
心在旅途
心在旅途 2021-02-14 13:17

I have a list of about 500 million items. I am able to serialize this into a file with protobuf-net file if I serialize individual items, not a list -- I cannot collect the item

3条回答
  •  北恋
    北恋 (楼主)
    2021-02-14 13:44

    The API apprently has changed since Marc's answer.
    It seems there's no SerializeItems method any more.

    Here's some more up to date info that should help:

    ProtoBuf.Serializer.Serialize(stream, items);
    

    can take an IEnumerable as seen above and it does the job when it comes to serialization.
    However there's a DeserializeItems(...) method and the devil is in the details :)
    If you serialize IEnumerable like above, then you need to call DeserializeItems passing PrefixStyle.Base128 and 1 as fieldNumber cause apprently those are the defaults.
    Here's an example:

    ProtoBuf.Serializer.DeserializeItems(stream, ProtoBuf.PrefixStyle.Base128, 1));
    

    Also as pointed out by Marc and Vic you can serialize/deserialize on a per item basis like this (using custom values for PrefixStyle and fieldNumber):

    ProtoBuf.Serializer.SerializeWithLengthPrefix(stream, item, ProtoBuf.PrefixStyle.Base128, fieldNumber: 1);
    

    and

    T item;
    while ((item = ProtoBuf.Serializer.DeserializeWithLengthPrefix(stream, ProtoBuf.PrefixStyle.Base128, fieldNumber: 1)) != null)
    {
        // do stuff here
    }
    

提交回复
热议问题