问题
I'm working with ffmpeg to process an incoming MPEGTS stream from remote cameras, and deliver it to multiple clients using my app.
Technically, I'm using ffmpeg to convert the incoming stream to an MJPEG output, and piping the data chunks (from the ffmpeg process stdout) to a writeable stream on the client http response.
However, I'm facing a problem- not all data chunks represent a full 'whole' frame. thus, displaying them in a row in the browser, results in a flickering video, with half-complete frames, on a random basis. I know this because when printing each chunk length, results most of the time in a big value (X), but every now and then I get 2 consecutive chunks with length (2/5X) followed by (3/5X) for example.
So the question - is there a way to force the ffmpeg process to output only whole frames? if not, is there a way for me to check each data chunk 'manually' and look for headers/metadata/flags to indicate frame start/end?
my ffmpeg command for outputting MJPEG is:
ffmpeg -i - -c:v mjpeg -f mjpeg -
explained:
"-i -" : (input) is the stdin of the process (and not a static file)
"-c:v mjpeg" : using the mjpeg codec
"-f mjpeg" : output will be in the mjpeg format
"-" : output not specified (file or url) - will be the process stdout
Edit: here are some console.log prints to visualize the problem:
%%% FFMPEG Info %%%
frame= 832 fps= 39 q=24.8 q=29.0 size= 49399kB time=00:00:27.76 bitrate=14577.1kbits/s speed=1.29x
data.length: 60376
data.length: 60411
data.length: 60465
data.length: 32768
data.length: 27688
data.length: 32768
data.length: 27689
data.length: 60495
data.length: 60510
data.length: 60457
data.length: 59811
data.length: 59953
data.length: 59889
data.length: 59856
data.length: 59936
data.length: 60049
data.length: 60091
data.length: 60012
%%% FFMPEG Info %%%
frame= 848 fps= 38 q=24.8 q=29.0 size= 50340kB time=00:00:28.29 bitrate=14574.4kbits/s speed=1.28x
data.length: 60025
data.length: 60064
data.length: 60122
data.length: 60202
data.length: 60113
data.length: 60211
data.length: 60201
data.length: 60195
data.length: 60116
data.length: 60167
data.length: 60273
data.length: 60222
data.length: 60223
data.length: 60267
data.length: 60329
%%% FFMPEG Info %%%
frame= 863 fps= 38 q=24.8 q=29.0 size= 51221kB time=00:00:28.79 bitrate=14571.9kbits/s speed=1.27x
As you can see, a whole frame is about ~60k (my indication is a clean video stream i'm viewing on the browser), but every now and then the output consists of 2 consecutive chunks that add up to ~60k. when delivered to the browser, these are 'half frames'.
回答1:
Following the comments here and on StackExchange, it seems that the MJPEG stream outputted from the ffmpeg process should consist of whole frames. listening to the ffmpeg ChildProcess stdout yields data chunks of varying size- which means they dont always represent a whole frame (full JPEG) image.
So instead of just pushing them to the consumer (currently a web browser showing the video stream) I wrote a bit of code to handle 'half-chunks' in memory and append them together until the frame is complete.
This seems to solve the problem, as I'm getting no flickering in the video.
const _SOI = Buffer.from([0xff, 0xd8]);
const _EOI = Buffer.from([0xff, 0xd9]);
private size: number = 0;
private chunks: any[] = [];
private jpegInst: any = null;
private pushWholeMjpegFrame(chunk: any): void {
const chunkLength = chunk.length;
let pos = 0;
while (true) {
if (this.size) {
const eoi = chunk.indexOf(_EOI);
if (eoi === -1) {
this.chunks.push(chunk);
this.size += chunkLength;
break;
} else {
pos = eoi + 2;
const sliced = chunk.slice(0, pos);
this.chunks.push(sliced);
this.size += sliced.length;
this.jpegInst = Buffer.concat(this.chunks, this.size);
this.chunks = [];
this.size = 0;
this.sendJpeg();
if (pos === chunkLength) {
break;
}
}
} else {
const soi = chunk.indexOf(_SOI, pos);
if (soi === -1) {
break;
} else {
pos = soi + 500;
}
const eoi = chunk.indexOf(_EOI, pos);
if (eoi === -1) {
const sliced = chunk.slice(soi);
this.chunks = [sliced];
this.size = sliced.length;
break;
} else {
pos = eoi + 2;
this.jpegInst = chunk.slice(soi, pos);
this.sendJpeg();
if (pos === chunkLength) {
break;
}
}
}
}
}
I would love to get some more educated input on my solution if it can be improved and optimized, as well as some more knowledge as to the origin of the problem and perhaps a way to get the desired behavior out-of-the-box with ffmpeg, so feel free to keep this question alive with more answers and comments.
回答2:
I had the same problem and ended up here. As others have stated, this ffmpeg behavior is by design and the problem can be easily solved outside of ffmpeg, as OP has shown. Consider ffmpeg output to be a stream. And as with streams in general, the contents are sent in pieces. This makes the flow of data much more consistent because the size of the chunks are not directly related to the size of each frame. It allows the throughput to be somewhat uniform (relative to its neighbor chunks) even when the compression scheme results in some frames that are drastically different in size due to motion, solid colors, etc.
OP's answer helped point me in the right direction and I wrote my own slightly simpler implementation for constructing full JPG images in vanilla ES6. In case it helps anyone else, the following is working well for me. It takes ffmpeg mjpeg chunks piped to standard output, and looks for SOI and EOI markers (see https://en.wikipedia.org/wiki/JPEG_File_Interchange_Format#File_format_structure) to build complete base64 JPG images ready for use in <img> or <canvas> elements.
let chunks = [];
// See https://en.wikipedia.org/wiki/JPEG_File_Interchange_Format#File_format_structure
// for SOI and EOI explanation.
const SOI = Buffer.from([0xff, 0xd8]);
const EOI = Buffer.from([0xff, 0xd9]);
function handleFfmpegOutputData(chunk) {
const eoiPos = chunk.indexOf(EOI);
const soiPos = chunk.indexOf(SOI);
if (eoiPos === -1) {
// No EOI - just append to chunks.
chunks.push(chunk);
} else {
// EOI is within chunk. Append everything before EOI to chunks
// and send the full frame.
const part1 = chunk.slice(0, eoiPos + 2);
if (part1.length) {
chunks.push(part1);
}
if (chunks.length) {
writeFullFrame(chunks);
}
// Reset chunks.
chunks = [];
}
if (soiPos > -1) {
// SOI is present. Ensure chunks has been reset and append
// everything after SOI to chunks.
chunks = [];
const part2 = chunk.slice(soiPos)
chunks.push(part2);
}
}
function writeFullFrame(frameChunks) {
// Concatenate chunks together.
const bufferData = Buffer.concat([...frameChunks]);
// Convert buffer to base64 for display.
const base64Data = Buffer.from(bufferData).toString('base64');
const imageSrc = `data:image/jpeg;base64,${base64Data}`;
// Do whatever you want with base64 src string...
}
来源：https://stackoverflow.com/questions/59468403/ffmpeg-how-to-force-mjpeg-output-of-whole-frames