Serialized objects disappearing (BinaryFormatter)

Question

Background

I have an object which I need to serialize in order to transfer to a high performance computing cluster for later use

Previously, I've used the out-of-the-box binary formatter for my object which represents a statistical shape model and all worked happily

My object became more complex and I decided to customize the serialization process by implementing ISerializable. I continue to support data stored in the previous format

The Problem

My problem is that one particular value appears to serialize successfully, but always has a value of null when I attempt deserialization. (no error, just a very unpleasant, un-useful null)

When I break at the point of serialization, I can see that the object is added to the SerializationInfo ok by inspecting the SerializationInfo and that it has values (it's nothing fancy, but will post the code for it below)

The serialization constructor is being called (I put a break-point there too), but when I inspect the SerializationInfo object of the constructor, it has no data (it does have an entry, just no data)

UPDATE - download console app here. Thanks for looking

or, look at code here:

The Code

Class causing problems: (the PointProfiles property is the offending object)

   [Serializable]
    public class TrainingSet : ITrainingSet, ISerializable
    {
        public Dictionary<Tuple<int, int>, IPointTrainingSet> PointProfiles { get; set; }

        public PrincipalComponentAnalysis PointPCA { get; set; }

        public double[] AlignedMean { get; set; }

        public List<Tuple<string, ITransform>> Transforms { get; set; }

        public string[] FileNames { get; set; }

        private static Lazy<BinaryFormatter> formatter = new Lazy<BinaryFormatter>();

        public static ITrainingSet Load(Guid modelId)
        {
            ModelSample s = DataProxy<ModelSample>.AsQueryable().Where(m => m.ModelId == modelId).SingleOrDefault();
            if (s == null)
                return null;

            byte[] raw = s.Samples.ToArray();
            using (MemoryStream ms = new MemoryStream(raw))
                return (ITrainingSet)formatter.Value.Deserialize(ms);

        }

        void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue("pca", PointPCA);

            info.AddValue("tp1", PointProfiles.Select(pp => pp.Key.Item1).ToArray());
            info.AddValue("tp2", PointProfiles.Select(pp => pp.Key.Item2).ToArray());

            var x = PointProfiles.Select(pp => (ProfileModel)pp.Value).ToArray();
            info.AddValue("ipts", x, typeof(ProfileModel[]));

            info.AddValue("am", AlignedMean);

            info.AddValue("tname", Transforms.Select(t => t.Item1).ToArray());
            info.AddValue("tval", Transforms.Select(t => t.Item2).ToArray());
            info.AddValue("fnames", FileNames);
            info.AddValue("version", 1);  // nb
        }

        public TrainingSet(SerializationInfo info, StreamingContext context)
        {
            int version = 0;
            foreach(SerializationEntry s in info)
            {
                if(s.Name == "version")
                    version = (int)s.Value;
            }

            switch(version)
            {
                case 0:
                    // old (default binary formatter)
                    PointPCA = info.GetValue("<PointPCA>k__BackingField", typeof(PrincipalComponentAnalysis)) as PrincipalComponentAnalysis;
                    PointProfiles = info.GetValue("<PointProfiles>k__BackingField", typeof(Dictionary<Tuple<int, int>, IPointTrainingSet>)) as Dictionary<Tuple<int, int>, IPointTrainingSet>;
                    AlignedMean = info.GetValue("<AlignedMean>k__BackingField", typeof(double[])) as double[];
                    Transforms = info.GetValue("<Transforms>k__BackingField", typeof(List<Tuple<string, ITransform>>)) as List<Tuple<string, ITransform>>;
                    FileNames = info.GetValue("<FileNames>k__BackingField", typeof(string[])) as string[];

            //stats.PointPCA = pointPCA;
            //stats.PointProfiles = pointProfiles;
            //stats.AlignedMean = alignedMean;
            //stats.Transforms = transforms;
            //stats.FileNames = fileNames;

                    break;

                case 1:
                    FileNames = info.GetValue("fnames", typeof(string[])) as string[];

                    var t = info.GetValue("tval", typeof(ITransform[])) as ITransform[];
                    var tn = info.GetValue("tname", typeof(string[])) as string[];

                    Transforms = new List<Tuple<string, ITransform>>();
                    for(int i = 0;i < tn.Length;i++)
                        Transforms.Add(new Tuple<string,ITransform>(tn[i], t[i]));

                    AlignedMean = info.GetValue("am", typeof(double[])) as double[];

                    PointPCA = info.GetValue("pca", typeof(PrincipalComponentAnalysis)) as PrincipalComponentAnalysis;


                    var ipts = info.GetValue("ipts", typeof(ProfileModel[]));

                    foreach (var x in info) 
                    {
                        int a = 0;
                        a++;   // break point here, info has an entry for key "ipts", but it's null  (or rather an array of the correct length, and each element of the array is null)
                    }

                    var xxx = ipts as IPointTrainingSet[];

                    var i2 = info.GetValue("tp2", typeof(int[])) as int[];

                    var i1 = info.GetValue("tp1", typeof(int[])) as int[];

                    PointProfiles = new Dictionary<Tuple<int, int>, IPointTrainingSet>();
                    for (int i = 0; i < i1.Length; i++)
                        PointProfiles.Add(new Tuple<int, int>(i1[i], i2[i]), xxx[i]);



                    break;

                default:
                    throw new NotImplementedException("TrainingSet version " + version + " is not supported");
            }

        }

        public TrainingSet()
        {

        }
    }

Profile class (also serializable, this is the base class for ProfileModel which is listed next)

    [Serializable]
    public class Profile : ISerializable, IProfile
    {
        public double Angle { get; private set; }
        public int PointIndex { get; private set; }
        public int Level { get; set; }


        public double[,] G { get; private set; }
        public virtual double[,] GBar { get { throw new InvalidOperationException(); } }

        public virtual int Width { get { return G.Length; } }

        public Profile(int level, int pointIndex, double angle, double[,] G)
        {
            this.G = G;
            PointIndex = pointIndex;
            Level = level;
            Angle = angle;
        }

        // deserialization
        public Profile(SerializationInfo info, StreamingContext context)
        {
            PointIndex = info.GetInt32("p");
            Angle = info.GetDouble("a");
            G = (double[,])info.GetValue("g", typeof(double[,]));

            Level = info.GetInt32("l");

            //_pca = new Lazy<PrincipalComponentAnalysis>(Pca);
        }

        // serialization
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue("p", PointIndex);
            info.AddValue("a", Angle);
            info.AddValue("g", G);
            info.AddValue("l", Level);
        }

    }

and (Finally) the ProfileModel class:

[Serializable]
public class ProfileModel : Profile, ISerializable, IPointTrainingSet
{

    public IProfile MeanProfile { get; private set; }

    private ProfileModel(int level, int PointIndex, IProfile[] profiles)
        : base(level, PointIndex, 0, null)
    {
        double[,] m = Matrix.Create<double>(profiles.Length, profiles[0].G.Columns(), 0);

        int idx = 0;
        foreach (var pg in profiles.Select(p => p.G.GetRow(0)))
            m.SetRow(idx++, pg);



        Profile meanProfile = new Profile(level, PointIndex, 0, m.Mean().ToMatrix());
        MeanProfile = meanProfile;
    }

    // deserialization
    public ProfileModel(SerializationInfo info, StreamingContext context) : base(info, context) {
        var ps = info.GetValue("mp", typeof(Profile));

        MeanProfile = (IProfile)ps;
    }

    // serialization
    public new void GetObjectData(SerializationInfo info, StreamingContext context)
    {
        info.AddValue("mp", MeanProfile, typeof(Profile));
        base.GetObjectData(info, context);
    }

    public override double[,] GBar
    {
        get
        {
            return MeanProfile.G;
        }
    }

    public override int Width { get {
        return GBar.Columns();
    } }
}

If you can spot anything I'm doing wrong that might cause this to happen, I'd be very very grateful :)

Answer 1

The array deserializes first and the inner deserilazations are performed afterwards. When looping through the array of ProfileModel its content haven' been derserialized yet.

You can probably fix this by implementing IDeserializationCallback (or by assigning the OnDeserilized attribute to a method that should be called upon deserilization completion). The OnDeserialzation method is called after the entire object graph is deserialized.

You would need to stash away your arrays in private fields:

private int []i1;
private int []i2;
private ProfileModel []  ipts;

Do the following under derserialization:

ipts = info.GetValue("ipts", typeof(ProfileModel[]));
i2 = info.GetValue("tp2", typeof(int[])) as int[];
i1 = info.GetValue("tp1", typeof(int[])) as int[];

And implement IDeserializationCallback:

public void OnDerserilization(object sender)
{
  PointProfiles = new Dictionary<Tuple<int, int>, IPointTrainingSet>();

  for (int i = 0; i < i1.Length; i++)
     PointProfiles.Add(new Tuple<int, int>(i1[i], i2[i]), ipts[i]);
}

Answer 2

I was able to get this to work by letting .NET handle List and Dictionary serialization instead of trying to do it manually:

void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
{
    info.AddValue("pca", PointPCA);
    info.AddValue("pps", PointProfiles);
    info.AddValue("am", AlignedMean);
    info.AddValue("transforms", Transforms);
    info.AddValue("fnames", FileNames);
}

public TrainingSet(SerializationInfo info, StreamingContext context)
{
    PointPCA = info.GetValue("pca", typeof(PrincipalComponentAnalysis)) as PrincipalComponentAnalysis;
    Transforms = (List<Tuple<string, ITransform>>)info.GetValue("transforms", typeof(List<Tuple<string, ITransform>>));
    AlignedMean = info.GetValue("am", typeof(double[])) as double[];
    PointProfiles = (Dictionary<Tuple<int, int>, IPointTrainingSet>)info.GetValue("pps", typeof(Dictionary<Tuple<int, int>, IPointTrainingSet>));
    FileNames = info.GetValue("fnames", typeof(string[])) as string[];
}

You have a very complex object graph. I tried breaking in the constructor of Profile and ProfileModel and had some strange results, but this simple version seems to work.