Unable to route webcam video to virtual video device on Linux (via OpenCV)

Question

I would like to put a video overlay onto an incoming webcam stream via OpenCV. As a first step I\'m trying to stream the incoming video from a webcam at /dev/video0

Answer 1

After a lot of researching, I finally was able to develop a working solution. There are a lot of steps which need to be performed and which I will discuss in detail below:

General

As described in my question above, the goal is to be able to take the incoming stream of a webcam and forward it to a virtual video device, which in turn can be opened with tools like VLC. This is considered to be a first step in order to be able to do further image manipulation.

1) v4l2loopback

v4l2loopback is a virtual video device (kernel module) for linux. Sources can be downloaded from here https://github.com/umlaeute/v4l2loopback. After downloading, the following steps must be performed in order to run it:

make
sudo make install
sudo depmod -a
sudo modprobe v4l2loopback

If you want to use this video device in Chrome (WebRTC) you need to execute the last line with an additional parameter:

sudo modprobe v4l2loopback exclusive_caps=1

Note that exlusive_caps is an array, so if the above doesn't work, try:

sudo modprobe v4l2loopback exclusive_caps=1,1,1,1,1,1,1,1

Information: It is important to note that the v4l2loopback device must be set to the same resolution like the resolution you want to use in the sample below. I have set the defines in the sample to FullHD as you can see. If you want e.g. 800x600, you either need to change the default in the v4l2loopback code before compilation or change the resolution, when inserting the module, via the additional cmd line parameters max_width and max_height. The kernel module operates by default with a resolution of 640x480. You can get more details and all supported parameters by using:

modinfo v4l2loopback

2) OpenCV

OpenCV is a library which supports capturing and live video manipulation. For building OpenCV please got to this page http://docs.opencv.org/3.0-beta/doc/tutorials/introduction/linux_install/linux_install.html which explains all steps in detail.

3) Sample code

You can build/run the sample code below in the following way:

g++ -ggdb `pkg-config --cflags --libs opencv` sample.cpp -o sample
./sample

Here is the code:

#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/videodev2.h>
#include "opencv2/opencv.hpp"

#define VIDEO_OUT "/dev/video0" // V4L2 Loopack
#define VIDEO_IN  "/dev/video1" // Webcam

#define WIDTH  1920
#define HEIGHT 1080


int main ( int argc, char **argv ) {
    cv::VideoCapture cap;
    struct v4l2_format vid_format;
    size_t framesize = WIDTH * HEIGHT * 3;
    int fd = 0;

    if( cap.open ( VIDEO_IN ) ) {
        cap.set ( cv::CAP_PROP_FRAME_WIDTH , WIDTH  );
        cap.set ( cv::CAP_PROP_FRAME_HEIGHT, HEIGHT );
    } else {
        std::cout << "Unable to open video input!" << std::endl;
    }

    if ( (fd = open ( VIDEO_OUT, O_RDWR )) == -1 )
        printf ("Unable to open video output!");

    memset ( &vid_format, 0, sizeof(vid_format) );
    vid_format.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;

    if ( ioctl ( fd, VIDIOC_G_FMT, &vid_format ) == -1 )
        printf ( "Unable to get video format data. Errro: %d\n", errno );

    vid_format.fmt.pix.width       = cap.get ( CV_CAP_PROP_FRAME_WIDTH  );
    vid_format.fmt.pix.height      = cap.get ( CV_CAP_PROP_FRAME_HEIGHT );
    vid_format.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
    vid_format.fmt.pix.sizeimage   = framesize;
    vid_format.fmt.pix.field       = V4L2_FIELD_NONE;

    if ( ioctl ( fd, VIDIOC_S_FMT, &vid_format ) == -1 )
        printf ( "Unable to set video format! Errno: %d\n", errno );

    cv::Mat frame ( cap.get(CV_CAP_PROP_FRAME_HEIGHT), 
    cap.get(CV_CAP_PROP_FRAME_WIDTH), CV_8UC3 );

    printf ( "Please open the virtual video device (/dev/video<x>) e.g. with VLC\n" );

    while (1) {
        cap >> frame;
        cv::cvtColor ( frame, frame, cv::COLOR_BGR2RGB ); // Webcams sometimes deliver video in BGR not RGB. so we need to convert
        write ( fd, frame.data, framesize );
    }
}