Eigen and boost::serialize

Question

I tried to write a generic serialize function which takes any dense matrix and serializes it: Some other questions which help but not to the end are here: Question1 Question

Answer 1

I've tested out your code and it also did not work when I tried to compile it. However, based on the documentation for Boost Serialize, I am under the impression that it is intended to be used with the stream operator <<. The following code works fine for me:

namespace boost {
   namespace serialization {
      template <class Archive, typename Derived> 
      void serialize( Archive & ar, Eigen::EigenBase<Derived> & g, const unsigned int version){
          ar & boost::serialization::make_array(g.derived().data(), g.size());
      }
   }
}

int main (int argc, char* argv[]){
    std::ofstream out("my_archive");
    boost::archive::text_oarchive oa (out);

    Eigen::Matrix <double, 4, 4> a;
    out << a;
    return 0;
}

The file my_archive is created in the working folder with non-zero values (just the uninitialized garbage in memory, with the above over-simplified code).

EDIT: I tried using the exact code above in my own application and found that I received the same error as you did. I honestly do not understand why that it, right now. The simplest fix that I found was to replace Eigen::EigenBase<Derived> with the actual Matrix type being used. The current code that I am using is:

namespace boost{
    namespace serialization {
        template <class Archive, typename Scalar>
        void serialize ( Archive & ar, Eigen::Matrix<Scalar, -1, -1, 0, -1, -1> & g, const unsigned int version ){ /* ... */ }
     }
}

The above code works for any scalar type (float, double, int) and for dynamic/runtime sized matrixes. For static sized, check into and update the template parameters accordingly.

EDIT #2 (April 09, 2014):

Despite the appearance of the above code working, when I tried to integrate it fully into my code and exercise it with appropriate unit testing, it stopped working. Unfortunately, the error messages that I was being given -- both from Visual Studio and clang -- were most unhelpful. Fortunately, gcc had buried within the horrible mess of error messages, a reference to CV-mismatch, which seems to have allowed me to fully address this.

The following code now appears to compile and run successfully. I've tried to make the formatting legible (without side-scrolling) -- hopefully the code below is clear:

namespace boost{
    namespace serialization{

        template<   class Archive, 
                    class S, 
                    int Rows_, 
                    int Cols_, 
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void save(
            Archive & ar, 
            const Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
            {
                int rows = g.rows();
                int cols = g.cols();

                ar & rows;
                ar & cols;
                ar & boost::serialization::make_array(g.data(), rows * cols);
            }

        template<   class Archive, 
                    class S, 
                    int Rows_,
                    int Cols_,
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void load(
            Archive & ar, 
            Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
        {
            int rows, cols;
            ar & rows;
            ar & cols;
            g.resize(rows, cols);
            ar & boost::serialization::make_array(g.data(), rows * cols);
        }

        template<   class Archive, 
                    class S, 
                    int Rows_, 
                    int Cols_, 
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void serialize(
            Archive & ar, 
            Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
        {
            split_free(ar, g, version);
        }


    } // namespace serialization
} // namespace boost

A few critical points on the above code:

• This code now has templated parameters for all of Eigen's Matrix parameters. This should allow it to work with all types of matrices and vectors, whether sized at compile-time or run-time. This is a major enhancement over the above code

• It is essential that the serialization code be split into separate save and load functions. Otherwise, the deserialization code will not resize the matrix to hold the original data. I believe that Boost::Serialize does provide some additional functions that can be overloaded to perform operations on serialization or deserialization, but this approach was easier to implement.

• The const qualifier on the save method is essential. This was the source of my troubles before an obscure g++ error keyed me in on this.

• I cannot say that I have fully stress-tested this code yet. If you (or anyone else) finds any more problems with it, please do let me know and I'll try and follow up with anything else that I find.

Trust that that helps.

Shmuel

Answer 2

Here is a more general and shorter version with the following features:

• works correctly whether the matrix is row or column-major
• uses name-value-pairs so it works with xml archives and the like
• does not need the split between save and load version
• adds a wrapper function for Eigen::Transform (Affine3d, Isometry3f etc.)
• unit-tested for various combinations, e.g. it works when you save a column-major 2x2 MatrixXd and load it as a row-major Eigen::Matrix when using XML archives. However this does apparently not work for binary or text archives! In these cases the types must match exactly

Code:

namespace boost { namespace serialization {

template<   class Archive,
            class S,
            int Rows_,
            int Cols_,
            int Ops_,
            int MaxRows_,
            int MaxCols_>
inline void serialize(Archive & ar,
                      Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & matrix,
                      const unsigned int version)
{
  int rows = matrix.rows();
  int cols = matrix.cols();
  ar & make_nvp("rows", rows);
  ar & make_nvp("cols", cols);    
  matrix.resize(rows, cols); // no-op if size does not change!

  // always save/load row-major
  for(int r = 0; r < rows; ++r)
    for(int c = 0; c < cols; ++c)
      ar & make_nvp("val", matrix(r,c));
}    

template<   class Archive,
            class S,
            int Dim_,
            int Mode_,
            int Options_>
inline void serialize(Archive & ar,
                      Eigen::Transform<S, Dim_, Mode_, Options_> & transform,
                      const unsigned int version)
{
  serialize(ar, transform.matrix(), version);
}    
}} // namespace boost::serialization

Answer 3

I make use of Eigen's plugin based extension:

/**
 * @file EigenDenseBaseAddons.h
 */
#ifndef EIGEN_DENSE_BASE_ADDONS_H_
#define EIGEN_DENSE_BASE_ADDONS_H_

friend class boost::serialization::access;
template<class Archive>
void save(Archive & ar, const unsigned int version) const {
  derived().eval();
  const Index rows = derived().rows(), cols = derived().cols();
  ar & rows;
  ar & cols;
  for (Index j = 0; j < cols; ++j )
    for (Index i = 0; i < rows; ++i )
      ar & derived().coeff(i, j);
}

template<class Archive>
void load(Archive & ar, const unsigned int version) {
  Index rows, cols;
  ar & rows;
  ar & cols;
  if (rows != derived().rows() || cols != derived().cols() )
    derived().resize(rows, cols);
  ar & boost::serialization::make_array(derived().data(), derived().size());
}

template<class Archive>
void serialize(Archive & ar, const unsigned int file_version) {
  boost::serialization::split_member(ar, *this, file_version);
}

#endif // EIGEN_DENSE_BASE_ADDONS_H_

Configure Eigen to use this pulgin:(simply define the macro before including any Eigen header)

#ifndef EIGEN_CONFIG_H_
#define EIGEN_CONFIG_H_

#include <boost/serialization/array.hpp>
#define EIGEN_DENSEBASE_PLUGIN "EigenDenseBaseAddons.h"

#include <Eigen/Core>

#endif // EIGEN_CONFIG_H_

Though I have not really tested this throughly, it works well and can also deal with Array or any other dense Eigen objects. It also works perfectly for expressions like vec.tail<4>() but may fail (without any compile error) for expression like mat.topRows<2>() or block operations. (See update: now works for sub matrices also)

In comparison to the other current answer, this works for more set of expression and might avoid some temporary. A non-intrusive version is also probably possible by passing PlainObjectBase<Derived> objects to the serialize functions..

---

/// Boost Serialization Helper

template <typename T>
bool serialize(const T& data, const std::string& filename) {
  std::ofstream ofs(filename.c_str(), std::ios::out);
  if (!ofs.is_open())
    return false;
  {
    boost::archive::binary_oarchive oa(ofs);
    oa << data;
  }
  ofs.close();
  return true;
}

template <typename T>
bool deSerialize(T& data, const std::string& filename) {
  std::ifstream ifs(filename.c_str(), std::ios::in);
  if (!ifs.is_open())
    return false;
  {
    boost::archive::binary_iarchive ia(ifs);
    ia >> data;
  }
  ifs.close();
  return true;
}

And some test code:

VectorXf vec(100);
vec.setRandom();
serializeText(vec.tail<5>(), "vec.txt");

MatrixXf vec_in;
deSerialize(vec_in, "vec.bin");
assert(vec_in.isApprox(vec.tail<5>()));

serialize(Vector2f(0.5f,0.5f), "a.bin");
Vector2f a2f;
deSerializeBinary(a2f, "a.bin");
assert(a2f.isApprox(Vector2f(0.5f,0.5f)));
VectorXf axf;
deSerialize(axf, "a.bin");
assert(aXf.isApprox(Vector2f(0.5f,0.5f)));

boost::shared_ptr<Vector4f> b = boost::make_shared<Vector4f>(Vector4f::Random());
serialize(b, "b.tmp");
boost::shared_ptr<Vector4f> b_in;
deSerialize(b_in, "b.tmp");
BOOST_CHECK_EQUAL(*b, *b_in);

Matrix4f m(Matrix4f::Random());
serialize(m.topRows<2>(), "m.bin");
deSerialize(m_in, "m.bin");

---

Update: I made some minor modifications,now serialization of sub matrices also works.