SerializationException when serializing lots of objects in .NET

Question

I\'m running into problems serializing lots of objects in .NET. The object graph is pretty big with some of the new data sets being used, so I\'m getting:

Sy         


        

Answer 1

Dude, you have reached the end of .net!

I haven't hit this limit, but here are a few pointers:

1. use [XmlIgnore] to skip some of the objects - maybe you don't need to serialize everything

2. you could use the serializer manually (i.e. not with attributes, but by implementing Serialize() ) and partition the models into more files.

Answer 2

I'm guessing... serialize less objects at a time?

2 main questions:

• what objects are they?
  • POCO?
  • DataTable?
• what type of serialization is it?
  • xml?
    • XmlSerializer?
    • DataContractSerializer?
  • binary?
    • BinaryFormatter?
    • SoapFormatter?
  • other?
    • json?
    • bespoke?

Serialization needs to have some consideration of what the data volume is; for example, some serialization frameworks support streaming of both the objects and the serialized data, rather than relying on a complete object graph or temporary storage.

Another option is to serialize homogeneous sets of data rather than full graphs - i.e. serialize all the "customers" separately the "orders"; this would usually reduce volumes, at the expense of having more complexity.

So: what is the scenario here?

Answer 3

The issue has been fixed with .NET Core 2.1. I have requested to backport the solution to .NET Framework 4.8:

https://github.com/Microsoft/dotnet-framework-early-access/issues/46.

If you feel the issue should be fixed you can leave a comment that this is also important to you. The fix in .NET Core was to reuse the prime number generator present in Dictionary also for BinaryFormatter.

If you have so many objects serialized and you do not want wait 40 minutes to read them back make sure that you add to your App.Config this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
    <!-- Use this switch to make BinaryFormatter fast with large object graphs starting with .NET 4.7.2 -->
      <AppContextSwitchOverrides value="Switch.System.Runtime.Serialization.UseNewMaxArraySize=true" />
  </runtime>
</configuration>

to enable the BinaryFormatter deserialization fix which did finally arrive with .NET 4.7.2. More information about both issues can be found here:

https://aloiskraus.wordpress.com/2017/04/23/the-definitive-serialization-performance-guide/

Answer 4

Have you thought about the fact that Int32.MaxValue is 2,147,483,647 - over 2 billion.

You'd need 16GB of memory just to store the pointers (assuming a 64 bit machine), let alone the objects themselves. Half that on a 32bit machine, though squeezing 8GB of pointer data into the maximum of 3GB or so usable space would be a good trick.

I strongly suspect that your problem is not the number of objects, but that the serialization framework is going into some kind of infinite loop because you have referential loops in your data structure.

Consider this simple class:

public class Node
{
    public string Name {get; set;}
    public IList<Node> Children {get;}
    public Node Parent {get; set;}
    ...
}

This simple class can't be serialised, because the presence of the Parent property means that serialisation will go into an infinite loop.

Since you're already implementing ISerializable, you are 75% of the way to solving this - you just need to ensure you remove any cycles from the object graph you are storing, to store an object tree instead.

One technique that is often used is to store the name (or id) of a referenced object instead of the actual reference, resolving the name back to the object on load.

Answer 5

Depending on the structure of the data, maybe you can serialize / deserialize subgraphs of your large object graph? If the data could be somehow partitioned, you could get away with it, creating only small duplication of serialized data.

Answer 6

Binary serilization of very large objects

If you run into this limitation binaryformatter the internal array cannot expand to greater than int32.maxvalue elements using BinaryFormater, help yourself with this code snippet Step 1

Install nuget package: Install-Package Newtonsoft.Json.Bson -Version 1.0.2

using Newtonsoft.Json.Bson; //import require namesapace

//Code snippet for serialization/deserialization

public byte[] Serialize<T>(T obj)
{
    using (var memoryStream = new MemoryStream())
    {
        using (var writer = new BsonDataWriter(memoryStream))
        {
            var serializer = new JsonSerializer();
            serializer.Serialize(writer, obj);
        }
        return memoryStream.ToArray();
    }
}

public T Deserialize<T>(byte[] data)
{
    using (var memoryStream = new MemoryStream(data))
    {
        using (var reader = new BsonDataReader(memoryStream))
        {
            var serializer = new JsonSerializer();
            return serializer.Deserialize<T>(reader);
        }
    }
}