文章目录
RTSP视频流处理方法
这里使用Gstreamer + OpenCV来处理RTSP视频流，因此对Gstreamer进行调查。
1. Gstreamer整体框架
Gstreamer是一个用于开发流式多媒体应用的开源框架，采用了基于插件（plugin）和管道（pipeline）的体系结构，框架中的所有的功能模块都被实现成可以插拔的组件（component）， 并且能够很方便地安装到任意一个管道上。由于所有插件都通过管道机制进行统一的数据交换，因此很容易利用已有的各种插件“组装”出一个功能完善的多媒体应用程序。
Nvidia为Gstreamer开发了许多plugin，这些plugin能够利用Nvidia硬件进行加速。Nvidia的deepstream就是基于gstreamer开发的。
下图是对基于Gstreamer框架的应用的简单分层：
1.1 Media Applications
最上面一层为应用，比如gstreamer自带的一些工具（gst-launch，gst-inspect等），以及基于gstreamer封装的库（gst-player，gst-rtsp-server，gst-editing-services等)根据不同场景实现的应用。
1.2 Core Framework
中间一层为Core Framework，主要提供：
- 上层应用所需接口
- Plugin的框架
- Pipline的框架
- 数据在各个Element间的传输及处理机制
- 多个媒体流（Streaming）间的同步（比如音视频同步）
- 其他各种所需的工具库
1.3 Plugins
最下层为各种插件，实现具体的数据处理及音视频输出，应用不需要关注插件的细节，会由Core Framework层负责插件的加载及管理。主要分类为：
- Protocols：负责各种协议的处理，file，http，rtsp等。
- Sources：负责数据源的处理，alsa，v4l2，tcp/udp等。
- Formats：负责媒体容器的处理，avi，mp4，ogg等。
- Codecs：负责媒体的编解码，mp3，vorbis等。
- Filters：负责媒体流的处理，converters，mixers，effects等。
- Sinks：负责媒体流输出到指定设备或目的地，alsa，xvideo，tcp/udp等。
2. Gstreamer组件
Gstreamer由许多基础的组件构成。
2.1 Element
Element是Gstreamer中最重要的对象类型之一。一个element实现了一个功能（读取文件，解码，输出等），程序需要创建多个element，并按顺序将其串连起来，构成一个完整的pipeline。
- Source Element 数据源元件 只有输出端，它仅能用来产生供管道消费的数据，而不能对数据做任何处理。一个典型的数据源元件的例子是音频捕获单元，它负责从声卡读取原始的音频数据，然后作为数据源提供给其它模块使用。
- Filter Element 过滤器元件 既有输入端又有输出端，它从输入端获得相应的数据，并在经过特殊处理之后传递给输出端。一个典型的过滤器元件的例子是音频编码单元，它首先从外界获得音频数据，然后根据特定的压缩算法对其进行编码，最后再将编码后的结果提供给其它模块使用。
- Sink Element 接收器元件 只有输入端，它仅具有消费数据的能力，是整条媒体管道的终端。一个典型的接收器元件的例子是音频回放单元，它负责将接收到的数据写到声卡上，通常这也是音频处理过程中的最后一个环节。
2.2 Pad
Pad是一个element的输入/输出接口，分为src pad（生产数据）和sink pad（消费数据）两种。
两个element必须通过pad才能连接起来，pad拥有当前element能处理数据类型的能力（capabilities），会在连接时通过比较src pad和sink pad中所支持的能力，来选择最恰当的数据类型用于传输。如果element不支持，程序会直接退出。
在element通过pad连接成功后，数据会从上一个element的src pad传到下一个element的sink pad然后进行处理。当element支持多种数据处理能力时，我们可以通过Cap来指定数据类型.
例如，下面的命令通过Cap指定了视频的宽高，videotestsrc会根据指定的宽高产生相应数据：
gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink
2.3 Bin和Pipeline
Bin是一个容器，用于管理多个element，改变bin的状态时，bin会自动去修改所包含的element的状态，也会转发所收到的消息。如果没有bin，我们需要依次操作我们所使用的element。通过bin降低了应用的复杂度。
Pipeline继承自bin，为程序提供一个bus用于传输消息，并且对所有子element进行同步。当将pipeline的状态设置为PLAYING时，pipeline会在一个/多个新的线程中通过element处理数据。
下面通过一个文件播放的例子来熟悉上述提及的概念：
测试文件：sintel_trailer-480p.ogv
gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink
通过上面的命令播放文件时，会创建如下pipeline：
可以看到这个pipeline由8个element构成，每个element都实现各自的功能：
- filesrc读取文件
- oggdemux解析文件，分别提取audio，video数据
- queue缓存数据
- vorbisdec解码audio
- autoaudiosink自动选择音频设备并输出
- theoradec解码video
- videoconvert转换video数据格式
- autovideosink自动选择显示设备并输出
不同的element拥有不同数量及类型的pad，只有src pad的element被称为source element，只有sink pad的被称为sink element。
element可以同时拥有多个相同的pad，例如oggdemux在解析文件后，会将audio，video通过不同的pad输出。
3. gstreamer tools
Gstreamer自带了gst-inspect-1.0和gst-launch-1.0等其他命令行工具，我们可以使用这些工具完成常见的处理任务。
3.1 gst-inspect-1.0
查看gstreamer的plugin、element的信息。直接将plugin/element作为参数，会列出其详细信息，包括plugin的功能、Pad的输入输出类型、plugin的属性等。
如果不跟任何参数，会列出当前系统gstreamer所能查找到的所有插件。
# gst-inspect-1.0 rtspsrc
Factory Details:
Rank none (0)
Long-name RTSP packet receiver
Klass Source/Network
Description Receive data over the network via RTSP (RFC 2326)
Author Wim Taymans <wim@fluendo.com>, Thijs Vermeir <thijs.vermeir@barco.com>, Lutz Mueller <lutz@topfrose.de>
Plugin Details:
Name rtsp
Description transfer data via RTSP
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtsp.so
Version 1.14.5
License LGPL
Source module gst-plugins-good
Source release date 2019-05-29
Binary package GStreamer Good Plugins (Ubuntu)
Origin URL https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBin
+----GstRTSPSrc
Implemented Interfaces:
GstChildProxy
GstURIHandler
Pad Templates:
SRC template: 'stream_%u'
Availability: Sometimes
Capabilities:
application/x-rtp
application/x-rdt
Element has no clocking capabilities.
URI handling capabilities:
Element can act as source.
Supported URI protocols:
rtsp
rtspu
rtspt
rtsph
rtsp-sdp
rtsps
rtspsu
rtspst
rtspsh
Pads:
none
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "rtspsrc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
async-handling : The bin will handle Asynchronous state changes
flags: readable, writable
Boolean. Default: false
message-forward : Forwards all children messages
flags: readable, writable
Boolean. Default: false
location : Location of the RTSP url to read
flags: readable, writable
String. Default: null
protocols : Allowed lower transport protocols
flags: readable, writable
Flags "GstRTSPLowerTrans" Default: 0x00000007, "tcp+udp-mcast+udp"
(0x00000000): unknown - GST_RTSP_LOWER_TRANS_UNKNOWN
(0x00000001): udp - GST_RTSP_LOWER_TRANS_UDP
(0x00000002): udp-mcast - GST_RTSP_LOWER_TRANS_UDP_MCAST
(0x00000004): tcp - GST_RTSP_LOWER_TRANS_TCP
(0x00000010): http - GST_RTSP_LOWER_TRANS_HTTP
(0x00000020): tls - GST_RTSP_LOWER_TRANS_TLS
debug : Dump request and response messages to stdout(DEPRECATED: Printed all RTSP message to gstreamer log as 'log' level)
flags: readable, writable, deprecated
Boolean. Default: false
retry : Max number of retries when allocating RTP ports.
flags: readable, writable
Unsigned Integer. Range: 0 - 65535 Default: 20
timeout : Retry TCP transport after UDP timeout microseconds (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 5000000
tcp-timeout : Fail after timeout microseconds on TCP connections (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 20000000
latency : Amount of ms to buffer
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 2000
drop-on-latency : Tells the jitterbuffer to never exceed the given latency in size
flags: readable, writable
Boolean. Default: false
connection-speed : Network connection speed in kbps (0 = unknown)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551 Default: 0
nat-method : Method to use for traversing firewalls and NAT
flags: readable, writable
Enum "GstRTSPNatMethod" Default: 1, "dummy"
(0): none - None
(1): dummy - Send Dummy packets
do-rtcp : Send RTCP packets, disable for old incompatible server.
flags: readable, writable
Boolean. Default: true
do-rtsp-keep-alive : Send RTSP keep alive packets, disable for old incompatible server.
flags: readable, writable
Boolean. Default: true
proxy : Proxy settings for HTTP tunneling. Format: [http://][user:passwd@]host[:port]
flags: readable, writable
String. Default: null
proxy-id : HTTP proxy URI user id for authentication
flags: readable, writable
String. Default: null
proxy-pw : HTTP proxy URI user password for authentication
flags: readable, writable
String. Default: null
rtp-blocksize : RTP package size to suggest to server (0 = disabled)
flags: readable, writable
Unsigned Integer. Range: 0 - 65536 Default: 0
user-id : RTSP location URI user id for authentication
flags: readable, writable
String. Default: null
user-pw : RTSP location URI user password for authentication
flags: readable, writable
String. Default: null
buffer-mode : Control the buffering algorithm in use
flags: readable, writable
Enum "GstRTSPSrcBufferMode" Default: 3, "auto"
(0): none - Only use RTP timestamps
(1): slave - Slave receiver to sender clock
(2): buffer - Do low/high watermark buffering
(3): auto - Choose mode depending on stream live
(4): synced - Synchronized sender and receiver clocks
port-range : Client port range that can be used to receive RTP and RTCP data, eg. 3000-3005 (NULL = no restrictions)
flags: readable, writable
String. Default: null
udp-buffer-size : Size of the kernel UDP receive buffer in bytes, 0=default
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 524288
short-header : Only send the basic RTSP headers for broken encoders
flags: readable, writable
Boolean. Default: false
probation : Consecutive packet sequence numbers to accept the source
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 2
udp-reconnect : Reconnect to the server if RTSP connection is closed when doing UDP
flags: readable, writable
Boolean. Default: true
multicast-iface : The network interface on which to join the multicast group
flags: readable, writable
String. Default: null
ntp-sync : Synchronize received streams to the NTP clock
flags: readable, writable
Boolean. Default: false
use-pipeline-clock : Use the pipeline running-time to set the NTP time in the RTCP SR messages(DEPRECATED: Use ntp-time-source property)
flags: readable, writable, deprecated
Boolean. Default: false
sdes : The SDES items of this session
flags: readable, writable
Boxed pointer of type "GstStructure"
tls-validation-flags: TLS certificate validation flags used to validate the server certificate
flags: readable, writable
Flags "GTlsCertificateFlags" Default: 0x0000007f, "validate-all"
(0x00000001): unknown-ca - G_TLS_CERTIFICATE_UNKNOWN_CA
(0x00000002): bad-identity - G_TLS_CERTIFICATE_BAD_IDENTITY
(0x00000004): not-activated - G_TLS_CERTIFICATE_NOT_ACTIVATED
(0x00000008): expired - G_TLS_CERTIFICATE_EXPIRED
(0x00000010): revoked - G_TLS_CERTIFICATE_REVOKED
(0x00000020): insecure - G_TLS_CERTIFICATE_INSECURE
(0x00000040): generic-error - G_TLS_CERTIFICATE_GENERIC_ERROR
(0x0000007f): validate-all - G_TLS_CERTIFICATE_VALIDATE_ALL
tls-database : TLS database with anchor certificate authorities used to validate the server certificate
flags: readable, writable
Object of type "GTlsDatabase"
tls-interaction : A GTlsInteraction object to promt the user for password or certificate
flags: readable, writable
Object of type "GTlsInteraction"
do-retransmission : Ask the server to retransmit lost packets
flags: readable, writable
Boolean. Default: true
ntp-time-source : NTP time source for RTCP packets
flags: readable, writable
Enum "GstRTSPSrcNtpTimeSource" Default: 0, "ntp"
(0): ntp - NTP time based on realtime clock
(1): unix - UNIX time based on realtime clock
(2): running-time - Running time based on pipeline clock
(3): clock-time - Pipeline clock time
user-agent : The User-Agent string to send to the server
flags: readable, writable
String. Default: "GStreamer/1.14.5"
max-rtcp-rtp-time-diff: Maximum amount of time in ms that the RTP time in RTCP SRs is allowed to be ahead (-1 disabled)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: 1000
rfc7273-sync : Synchronize received streams to the RFC7273 clock (requires clock and offset to be provided)
flags: readable, writable
Boolean. Default: false
max-ts-offset-adjustment: The maximum number of nanoseconds per frame that time stamp offsets may be adjusted (0 = no limit).
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-ts-offset : The maximum absolute value of the time offset in (nanoseconds). Note, if the ntp-sync parameter is set the default value is changed to 0 (no limit)
flags: readable, writable
Integer64. Range: 0 - 9223372036854775807 Default: 3000000000
default-rtsp-version: The RTSP version that should be tried first when negotiating version.
flags: readable, writable
Enum "GstRTSPVersion" Default: 16, "1-0"
(0): invalid - GST_RTSP_VERSION_INVALID
(16): 1-0 - GST_RTSP_VERSION_1_0
(17): 1-1 - GST_RTSP_VERSION_1_1
(32): 2-0 - GST_RTSP_VERSION_2_0
backchannel : The type of backchannel to setup. Default is 'none'.
flags: readable, writable
Enum "GstRTSPBackchannel" Default: 0, "none"
(0): none - No backchannel
(1): onvif - ONVIF audio backchannel
Element Signals:
"pad-added" : void user_function (GstElement* object,
GstPad* arg0,
gpointer user_data);
"pad-removed" : void user_function (GstElement* object,
GstPad* arg0,
gpointer user_data);
"no-more-pads" : void user_function (GstElement* object,
gpointer user_data);
"handle-request" : void user_function (GstElement* object,
gpointer arg0,
gpointer arg1,
gpointer user_data);
"on-sdp" : void user_function (GstElement* object,
GstSDPMessage* arg0,
gpointer user_data);
"select-stream" : gboolean user_function (GstElement* object,
guint arg0,
GstCaps* arg1,
gpointer user_data);
"new-manager" : void user_function (GstElement* object,
GstElement* arg0,
gpointer user_data);
"request-rtcp-key" : GstCaps * user_function (GstElement* object,
guint arg0,
gpointer user_data);
"accept-certificate" : gboolean user_function (GstElement* object,
GTlsConnection* arg0,
GTlsCertificate* arg1,
GTlsCertificateFlags arg2,
gpointer user_data);
"before-send" : gboolean user_function (GstElement* object,
GstRTSPMessage* arg0,
gpointer user_data);
Element Actions:
"push-backchannel-buffer" : GstFlowReturn user_function (GstElement* object,
guint arg0,
GstBuffer* arg1);
3.2 gst-launch-1.0
用于创建及执行一个Pipline，因此通常使用gst-launch先验证相关功能，然后再编写相应应用。
通过上面ogg视频播放的例子，我们已经看到，一个pipeline的多个element之间通过 “!" 分隔，同时可以设置element及Cap的属性。
下面是解析RTSP视频流的pipeline：
gst-launch-1.0 -v rtspsrc location=rtsp://10.201.0.158:8554/vlc ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, width=(int)2048, height=(int)1536, format=(string)BGRx ! videoconvert ! ximagesink sync=false
我们可以通过gst-inspect-1.0工具查看每个plugin的功能和属性，选择合适的插件来构成pipeline。
具体在python实现OpenCV+Gstreamer的方法是：OpenCV提供了cv2.VideoCapture()函数，只需把Gstreamer参数传给该函数即可。
具体代码如下：
def open_cam_rtsp(uri, width, height, latency):
gst_str = ("rtspsrc location={} latency={} ! rtph264depay ! h264parse ! omxh264dec ! "
"nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
"videoconvert ! appsink").format(uri, latency, width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
def open_cam_usb(dev, width, height):
# We want to set width and height here, otherwise we could just do:
# return cv2.VideoCapture(dev)
gst_str = ("v4l2src device=/dev/video{} ! "
"video/x-raw, width=(int){}, height=(int){}, format=(string)RGB ! "
"videoconvert ! appsink").format(dev, width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
def open_cam_onboard(width, height):
# On versions of L4T previous to L4T 28.1, flip-method=2
# Use Jetson onboard camera
gst_str = ("nvcamerasrc ! "
"video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! "
"nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
"videoconvert ! appsink").format(width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
4. 参考链接
https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html
https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html
来源：https://blog.csdn.net/han2529386161/article/details/102724856