convert QVideoFrame to QImage

Question

I want to get every frames from a QMediaPlayer and convert it to QImage (or cv::Mat)

so I used videoFrameProbed

Answer 1

For anyone ending up here after finding that the qt private function "qt_imageFromVideoFrame" no longer exists in Qt5.15.0, you can now use the newly implemented method QImage QVideoFrame::image().

QImage img = frame.image();

I've tested this on Ubuntu and it works.

Answer 2

For QCamera output, that method doesn't always work. In particular, QVideoFrame::imageFormatFromPixelFormat() returns QImage::Format_Invalid when given QVideoFrame::Format_Jpeg, which is what's coming out of my QCamera. But this works:

QImage Camera::imageFromVideoFrame(const QVideoFrame& buffer) const
{
    QImage img;
    QVideoFrame frame(buffer);  // make a copy we can call map (non-const) on
    frame.map(QAbstractVideoBuffer::ReadOnly);
    QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(
                frame.pixelFormat());
    // BUT the frame.pixelFormat() is QVideoFrame::Format_Jpeg, and this is
    // mapped to QImage::Format_Invalid by
    // QVideoFrame::imageFormatFromPixelFormat
    if (imageFormat != QImage::Format_Invalid) {
        img = QImage(frame.bits(),
                     frame.width(),
                     frame.height(),
                     // frame.bytesPerLine(),
                     imageFormat);
    } else {
        // e.g. JPEG
        int nbytes = frame.mappedBytes();
        img = QImage::fromData(frame.bits(), nbytes);
    }
    frame.unmap();
    return img;
}

Answer 3

Qt has a private function qt_imageFromVideoFrame for converting QVideoFrame to QImage. If you want to use it, you'll need to:

1. Add QT += multimedia multimedia-private to your .pro file
2. Add #include "private/qvideoframe_p.h" to your .cpp file

3. Then you can use qt_imageFromVideoFrame with the following signature:

   QImage qt_imageFromVideoFrame( const QVideoFrame& f );

Be aware of the warning in the header:

//
//  W A R N I N G
//  -------------
//
// This file is not part of the Qt API.  It exists purely as an
// implementation detail.  This header file may change from version to
// version without notice, or even be removed.
//
// We mean it.
//

The implementation uses QVideoFrame::map() and does low-level manipulation with the bits. The converters handle a variety of conditions including YUV.

The qt_imageFromVideoFrame works on almost all platforms. i.e. Windows, macOS, Linux and iOS. The only platform it doesn't work reliably well is Android. There, you'll need to use OpenGL calls to extract the VideoFrame. Also, on Android, you need to call QImage:rgbSwapped() at the end because QImage stores images as ARGB whereas OpenGL retrieves them as ABGR.

QImage QVideoFrameToQImage( const QVideoFrame& videoFrame )
{
    if ( videoFrame.handleType() == QAbstractVideoBuffer::NoHandle )
    {
        QImage image = qt_imageFromVideoFrame( videoFrame );
        if ( image.isNull() ) return QImage();
        if ( image.format() != QImage::Format_ARGB32 ) return image.convertToFormat( QImage::Format_ARGB32 );
        return image;
    }
    if ( videoFrame.handleType() == QAbstractVideoBuffer::GLTextureHandle )
    {
        QImage image( videoFrame.width(), videoFrame.height(), QImage::Format_ARGB32 );
        GLuint textureId = static_cast<GLuint>( videoFrame.handle().toInt() );
        QOpenGLContext* ctx = QOpenGLContext::currentContext();
        QOpenGLFunctions* f = ctx->functions();
        GLuint fbo;
        f->glGenFramebuffers( 1, &fbo );
        GLint prevFbo;
        f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo );
        f->glBindFramebuffer( GL_FRAMEBUFFER, fbo );
        f->glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,  GL_TEXTURE_2D, textureId, 0 );
        f->glReadPixels( 0, 0,  videoFrame.width(),  videoFrame.height(), GL_RGBA, GL_UNSIGNED_BYTE, image.bits() );
        f->glBindFramebuffer( GL_FRAMEBUFFER, static_cast<GLuint>( prevFbo ) );
        return image.rgbSwapped();
    }
    return QImage();
}

I have a complete working app on GitHub that includes an implementation of the above function:

https://github.com/stephenquan/MyVideoFilterApp/blob/master/QVideoFrameToQImage.cpp

Answer 4

You can use QImage's constructor:

 QImage img( currentFrame.bits(),
             currentFrame.width(),
             currentFrame.height(),
             currentFrame.bytesPerLine(),
             imageFormat);

Where you can get imageFormat from pixelFormat of the QVideoFrame:

 QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(currentFrame.pixelFormat());