OpenCV - GrabCut with custom foreground/background models

Question

I want to use the GrabCut algorithm implemented on OpenCV.

As shown in the documentation this is the function signature:

void grabCut(
    InputArray img, 
    InputOutputArray mask, 
    Rect rect, 
    InputOutputArray bgdModel, // *
    InputOutputArray fgdModel, // *
    int iterCount, 
    int mode=GC_EVAL) 

The mode param, indicates how to initialize the algorithm, either with the rect (a rectangle bounding box) or with the mask (a matrix whose values correspond to user paintings of the foreground/background regions.

I already have the color models for both the FG and the BG, so ideally I shouldn’t need to provide a mask or a rectangle, but use those models as an initialization (I want to prevent OpenCV to compute new models and use mine instead). I see that bgdModel and fgdModel parameters somehow contain this model information. Unfortunatelly, the documentation does not provide any details on how the model information is stored there.

Is it possible to populate those models whith existing data and run the method with mode=GC_EVAL?, if so, how do I need to encode the models?

Answer 1

In opencv/sources/modules/imgproc/src/grabcut.cpp you can have a look how the models (GMMs) are encoded:

GMM::GMM( Mat& _model )
{
    const int modelSize = 3/*mean*/ + 9/*covariance*/ + 1/*component weight*/;
    if( _model.empty() )
    {
        _model.create( 1, modelSize*componentsCount, CV_64FC1 );
        _model.setTo(Scalar(0));
    }
    else if( (_model.type() != CV_64FC1) || (_model.rows != 1) || (_model.cols != modelSize*componentsCount) )
        CV_Error( CV_StsBadArg, "_model must have CV_64FC1 type, rows == 1 and cols == 13*componentsCount" );

    model = _model;

    coefs = model.ptr<double>(0);
    mean = coefs + componentsCount;
    cov = mean + 3*componentsCount;

    for( int ci = 0; ci < componentsCount; ci++ )
        if( coefs[ci] > 0 )
             calcInverseCovAndDeterm( ci );
}

So you need for every model a cv::Mat of 1 x 65 doubles (componentsCount equals 5). There are 3 means per component because its computing in RGB colorspace. Using GC_EVAL would indeed leave the models intact but I never tried it with pre-computed models.

Answer 2

Even I had a similar problem. This is how I solved it. I edited the GC_EVAL condition in the grabcut source code to this -

if( mode == GC_EVAL )
   { checkMask( img, mask );
    for( int i = 0; i < iterCount; i++ )
{
    GCGraph<double> graph;
    assignGMMsComponents( img, mask, bgdGMM, fgdGMM, compIdxs );

    constructGCGraph(img, mask, bgdGMM, fgdGMM, lambda, leftW, upleftW, upW, uprightW, graph );
    estimateSegmentation( graph, mask );
    return;    
}
}

Notice that the function learnGMMs is not called here. This is done because the Foreground and Background GMMs are precomputed.

You can save models in a .xml file using the following code snippet.

 FileStorage fs("mymodels.xml", FileStorage::WRITE);
        fs << "BgdModel" << bgdModel;
        fs << "FgdModel" << fgdModel;
fs.release();

You can retrieve the models using the following code.

FileStorage fs1("mymodels.xml", FileStorage::READ);

        fs1["BgdModel"] >> bgdModel1;

        fs1["FgdModel"] >> fgdModel1;

This worked for me.