Openni2 reading recording (.oni) in the wrong order

问题

I am having issues with reading a recording I made using the Recorder class in openni2 with the asus Xtion PRO LIVE. The problem is that once every ~50 frames a wrong frame is read from the recording, this was tested by storing the generated image (converted to an opencv matrix) as an .png image (using the opencv imwrite function) with an index. Shown below is my code, I also tried the code posted in the "OpenNI2: Reading in .oni Recording" question. This also doesnt work. The videostreams, device, videoFrameRefs and the playback controller are all initialized in an initialization function. I can post this as well if necessary.

openni::VideoStream depthStream;
openni::VideoStream colorStream;
openni::Device      device;

openni::VideoFrameRef       m_depthFrame;
openni::VideoFrameRef       m_colorFrame;

openni::PlaybackControl *controler;

typedef struct _ImagePair {
    cv::Mat GrayscaleImage;
    cv::Mat DepthImage;
    cv::Mat RGBImage;
    float AverageDistance;
    std::string FileName;
    int index;
}ImagePair;

ImagePair SensorData::getNextRawImagePair(){
    ImagePair result;
    openni::Status rc;
    std::cout<<"Getting next raw image pair...";
    rc = controler->seek(depthStream, rawPairIndex);
    if(!correctStatus(rc,"Seek depth"))return result;

    openni::VideoStream** m_streams = new openni::VideoStream*[2];
    m_streams[0] = &depthStream;
    m_streams[1] = &colorStream;
    bool newDepth=false,newColor=false;

    while(!newDepth || !newColor){
        int changedIndex;
        openni::Status rc = openni::OpenNI::waitForAnyStream(m_streams, 2, &changedIndex);
        if (rc != openni::STATUS_OK)
        {
            printf("Wait failed\n");
            //return 1;
        }

        switch (changedIndex)
        {
            case 0:
                rc = depthStream.readFrame(&m_depthFrame);
                if(!correctStatus(rc,"Read depth")){
                    return result;
                }
                newDepth = true;
                break;
            case 1:
                rc = colorStream.readFrame(&m_colorFrame); 
                if(!correctStatus(rc,"Read color")){
                    return result;
                }
                newColor = true;
                break;
            default:
                printf("Error in wait\n");
        }
    }

    //convert rgb to bgr cv::matrix
    cv::Mat cv_image;
    const openni::RGB888Pixel* colorImageBuffer = (const openni::RGB888Pixel*)m_colorFrame.getData();
    cv_image.create(m_colorFrame.getHeight(), m_colorFrame.getWidth(), CV_8UC3);
    memcpy( cv_image.data, colorImageBuffer,3*m_colorFrame.getHeight()*m_colorFrame.getWidth()*sizeof(uint8_t) );

    //convert to BGR opencv color order
    cv::cvtColor(cv_image,cv_image,cv::COLOR_RGB2BGR);
    result.RGBImage = cv_image.clone(); 

    //convert to grayscale
    cv::cvtColor(cv_image,cv_image,cv::COLOR_BGR2GRAY);
    result.GrayscaleImage =cv_image.clone();

    //convert depth to cv::Mat INT16
    const openni::DepthPixel* depthImageBuffer = (const openni::DepthPixel*)m_depthFrame.getData();
    cv_image.create(m_depthFrame.getHeight(), m_depthFrame.getWidth(), CV_16UC1);
    memcpy( cv_image.data, depthImageBuffer,m_depthFrame.getHeight()*m_depthFrame.getWidth()*sizeof(INT16) );

    result.DepthImage = cv_image.clone();
    result.index = rawPairIndex;

    rawPairIndex++;

    std::cout<<"done"<<std::endl;
    return result;
}

I also tried it without the using waitForAnyStream part but that only made it worse. The file I am trying to load is over 1GB, not sure if that is a problem. The behaviour seems random because the indexes of the wrong images are not always the same.

UPDATE: I changed the seek function to seek in the color stream instead of the depth stream. I also ensured that each stream is only waited for once in the waitForAnyStream function by setting the corresponding point in m_streams to null.

I found out that it is possible to find the actual frame index for each frame (.getFrameIndex) so I was able to compare the indices. After getting 1000 images there where 18 wrong color images, 17 of which had an index error of 53 and 1 had an error of 46. The depth images had an almost constant index error of 9.

The second thing I found was after adding a sleep of 1 ms (also tried 10ms but the results where the same) before the waitForAnyStream function the returned indices change significantly. The indices dont make large jumps anymore but the color images have a standard offset of 53 and the depth images have an offset of 9.

When I change the seek function to seek in the depth stream then with the delay the color stream has a constant error of 46 and the depth stream has an error of 1. Without the delay the color stream has an error of 45 and the depth stream has no error with occasional spikes of errors of 1.

I also looked at the indices when I dont use seek and waitForAnyStream but just do it as is proposed as an answer in "OpenNI2: Reading in .oni Recording". This shows that when the file is read by just calling m_readframe multiple times then the first color frame has index 59 instead of 0. After checking it turns out that frame 0 does excist and is an earlier frame than frame 59. So just opening the file and using readframe doesnt work at all. There are however no index errors, just like when I added the sleep to my own implementation.

UPDATE2: I have come to the conclusion that either the openni2 library doesnt work properly or I have installed it incorrectly. This is because I am also having problems setting the Xtion to a resolution of 640x480 for both streams. When I do this the depth image only gets updated once every ~100 seconds. I have written a quick fix for my original problem by just filtering out the frames who's indices are wrong and just continuing with the next image.

I would still like to know how to fix this, so if you know please tell me.

UPDATE 3: The code I am using for setting the framerate and fps for recording is:

//colorstream
openni::VideoMode color_videoMode  = m_colorStream.getVideoMode();
color_videoMode.setResolution(WIDTH,HEIGHT);
color_videoMode.setFps(30);
m_colorStream.setVideoMode(color_videoMode);

//depth stream
openni::VideoMode depth_videoMode  = m_depthStream.getVideoMode();
depth_videoMode.setResolution(WIDTH,HEIGHT);
depth_videoMode.setFps(30);
m_depthStream.setVideoMode(depth_videoMode);

I forgot to mention that I also tested the resolution by running the sampleviewer example program (I think, it was a few months ago) and changing the resolution in the xml config file. Here the color images would be shown fine but the depth images would update verry rarely.

来源：https://stackoverflow.com/questions/27381100/openni2-reading-recording-oni-in-the-wrong-order

标签

c++

OpenCV

openni