Reconstruct image from RTP packets

Question

I am trying to stream a user's webcam over the network to a C-based server. I have used Janus gateway.

I created a small plugin which is heavily based on the echotest demo example : I have my browser connecting to my janus server via WebRTC technology and I have it stream the user's webcam.

On the server side, I have janus_incomming_rtp function which gives me a char * buffer and int length. Upon inspection, the buffer of data that's incomming is about the length of the MTU : each frame of my video is sent upon several RTP packets.

I have inspected a bit the header by following this wikipedia page but I don't know how to reconstruct the image from that stream of UDP RTP packets. Ideally, I'd like to pass the stream to openCV to do realtime image processing.

I have heard of gstreamer, but I don't undertstand what it is nor how it could help me ; besides I don't know if openCV has any built in functions to "reconstruct" the images ? And I don't know in which format the video frames are being encoded : the PT (Payload Type) seems to be 116 which is defined as "dynamic" but I have no idea what it means.

Any help ?

Answer 1

Here are some guiding steps for handling the SRTP packets to decode them.

1. Make sure that rtp and RTCP are not multiplexed, you can remove that option from the SDP
2. Decrypt the SRTP packet to raw RTP, you will need access to the key exchange(not sure if you are already doing this but all media is encrypted and keys exchanged using DTLS and must be decrypted before handling)
3. Grab your media payload type and match it against the media from SDP(you can see from the RTPMAP in the SDP what media is what payload)
4. Remove the RTP Payload from the packet(Gstreamer has RtpDepay plugins for most common payloads, including VP8) and decode the stream. Quick example of command line pipelines using vp8
5. Now you have a raw video/audio packet that can be displayed.

SDP:

• If RTCP and RTP are being multiplexed you will see the line a=rtcp-mux and you will see that the port in a=rtcp:50111 IN IP4 <address> and the candidate media ports will be the same.
• If the media itself is being multiplexed you will see a=group:BUNDLE audio video

SRTP:

• Janus handles the DTLS exchange already and it seems that it may already decrypt the rtp before sending it but it does not look like it accounts for multiplexed rtp/rtcp and media.
• Here is a quick and dirty SRTP decrypter that works when you pass it the MasterKey that is exchanged in DTLS.

GStreamer:

• You may want to look into the GstAppSrc which allows you to char arrays into a gstreamer pipeline for decoding and you can push it to another udp port to grab it with OpenCV.
• Here is some example code from a websocket server I wrote that will grab raw media and push it to a pipeline. This example is not exactly what you want to do(it does not grab the RTP but instead raw media frames from the webpage) but it will show you how to use AppSrc.

Answer 2

We have the same concern regarding the streaming of WebRTC. What I did is send the video frame to a WebSocket Server, from there I decode the image buffer using imdecode().

I have a live demo here twistedcv and also host the source in github here twistedcv. But the streaming is not on real-time.

Answer 3

I ended up getting this working using Janus and GStreamer (1.9) by following the suggestions of others in this thread including @nschoe (the OP) and @Benjamin Trent. I figured that I would include my code to make life easier for the next person who comes along since so much trial-and-error was involved for me:

First build/install GStreamer with all its needed plugins (for my setup I needed to ensure that two plugin directories were in the GST_PLUGIN_SYSTEM_PATH environment variable). Now initialize GStreamer when your Janus plugin initializes (init() callback):

gst_init(NULL, NULL);

For each WebRTC session, you'll need to keep some GStreamer handles, so add the following to your Janus plugin session struct:

GstElement *pipeline, *appsrc, *multifilesink;

When a Janus plugin session is created (create_session() callback), setup the GStreamer pipeline for that session (in my case I needed to lower the frame rate, hence the videorate/capsrate; you may not need these):

GstElement *conv, *vp8depay, *vp8dec, *videorate, *capsrate, *pngenc;

session->pipeline = gst_pipeline_new("pipeline");

session->appsrc         = gst_element_factory_make("appsrc", "source");
vp8depay                = gst_element_factory_make("rtpvp8depay", NULL);
vp8dec                  = gst_element_factory_make("vp8dec", NULL);
videorate               = gst_element_factory_make("videorate", NULL);
capsrate                = gst_element_factory_make("capsfilter", NULL);
conv                    = gst_element_factory_make("videoconvert", "conv");
pngenc                  = gst_element_factory_make("pngenc", NULL);
session->multifilesink  = gst_element_factory_make("multifilesink", NULL);

GstCaps* capsRate = gst_caps_new_simple("video/x-raw", "framerate", GST_TYPE_FRACTION, 15, 1, NULL);
g_object_set(capsrate, "caps", capsRate, NULL);
gst_caps_unref(capsRate);

GstCaps* caps = gst_caps_new_simple ("application/x-rtp",
                 "media", G_TYPE_STRING, "video",
                 "encoding-name", G_TYPE_STRING, "VP8-DRAFT-IETF-01",
                 "payload", G_TYPE_INT, 96,
                 "clock-rate", G_TYPE_INT, 90000,
                 NULL);
g_object_set(G_OBJECT (session->appsrc), "caps", caps, NULL);
gst_caps_unref(caps);

gst_bin_add_many(GST_BIN(session->pipeline), session->appsrc, vp8depay, vp8dec, conv, videorate, capsrate, pngenc, session->multifilesink, NULL);
gst_element_link_many(session->appsrc, vp8depay, vp8dec, conv, videorate, capsrate, pngenc, session->multifilesink, NULL);

// Setup appsrc
g_object_set(G_OBJECT (session->appsrc), "stream-type", 0, NULL);
g_object_set(G_OBJECT (session->appsrc), "format", GST_FORMAT_TIME, NULL);
g_object_set(G_OBJECT (session->appsrc), "is-live", TRUE, NULL);
g_object_set(G_OBJECT (session->appsrc), "do-timestamp", TRUE, NULL);

g_object_set(session->multifilesink, "location", "/blah/some/dir/output-%d.png", NULL);
gst_element_set_state(session->pipeline, GST_STATE_PLAYING);

When an incoming RTP packet gets demultiplexed by Janus and is ready to read, (incoming_rtp() callback), feed it into the GStreamer pipeline:

if(video && session->video_active) {
    // Send to GStreamer
    guchar* temp = NULL;
    temp = (guchar*)malloc(len);
    memcpy(temp, buf, len);

    GstBuffer*  buffer = gst_buffer_new_wrapped_full(0, temp, len, 0, len, temp, g_free);
    gst_app_src_push_buffer(GST_APP_SRC(session->appsrc), buffer);
}

Finally, when the Janus plugin session is over (destroy_session() callback), be sure to free up the GStreamer resources:

if(session->pipeline) {
    gst_element_set_state(session->pipeline, GST_STATE_NULL);
    gst_object_unref(session->pipeline);
    session->pipeline = NULL;
}