That particular exception isn't thrown because you are serializing an object that is too large. It is thrown because you are serializing an object graph with so many objects that the ObjectIDGenerator inside BinaryFormatter cannot allocate a large enough hash table to assign unique IDs to each pointer. The serializer uses ObjectIDGenerator to generate a run-time unique ID for each reference class serialized so as to correctly serialize multiple references to the same class instance as a single id-based indirect reference. Any graph serializer you choose will need to do something like this.
Rather than adopting a new graph serializer technology, which will be quite burdensome, is it possible for you to reduce the number of class instances you serialize simultaneously? For instance, can you break your object graph into disconnected segments, and serialize each one sequentially into the stream? (One tutorial on how to do this is here: Serializing lots of different objects into a single file.) Or do you have some classes that contain multiple small leaf classes that are never shared and could be replaced with a single proxy for all of them?
