Fragmented MP4 - problem playing in browser

Question

I try to create fragmented MP4 from raw H264 video data so I could play it in internet browser\'s player. My goal is to create live streaming system, where media server would se

Answer 1

I finally found the solution. My MP4 now plays in Chrome (while still playing in other tested browsers).

In Chrome chrome://media-internals/ shows MSE logs (of a sort). When I looked there, I found a few of following warnings for my test player:

ISO-BMFF container metadata for video frame indicates that the frame is not a keyframe, but the video frame contents indicate the opposite.

That made me think and encouraged to set AV_PKT_FLAG_KEY for packets with keyframes. I added following code to section with filling AVPacket structure:

    //Check if keyframe field needs to be set
    int allowedNalsCount = 3; //In one packet there would be at most three NALs: SPS, PPS and video frame
    packet.flags = 0;
    for(int i = 0; i < frameSize && allowedNalsCount > 0; ++i)
    {
        uint32_t *curr =  (uint32_t*)(frameBuffer + i);
        if(*curr == synchMarker)
        {
            uint8_t nalType = frameBuffer[i + sizeof(uint32_t)] & 0x1F;
            if(nalType == KEYFRAME)
            {
                std::cout << "Keyframe detected at frame nr " << framesTotal << std::endl;
                packet.flags = AV_PKT_FLAG_KEY;
                break;
            }
            else
                i += sizeof(uint32_t) + 1; //We parsed this already, no point in doing it again

            --allowedNalsCount;
        }
    }

A KEYFRAME constant turns out to be 0x5 in my case (Slice IDR).

Answer 2

You need to fill in extradata with AVC Decoder Configuration Record, not just SPS/PPS

Here's how the record should look like: AVCDCR

Answer 3

We can find this explanation in [Chrome Source] (https://chromium.googlesource.com/chromium/src/+/refs/heads/master/media/formats/mp4/mp4_stream_parser.cc#799) "chrome media source code":

// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.


  // Use |analysis.is_keyframe|, if it was actually determined, for logging
  // if the analysis mismatches the container's keyframe metadata for
  // |frame_buf|.
  if (analysis.is_keyframe.has_value() &&
      is_keyframe != analysis.is_keyframe.value()) {
    LIMITED_MEDIA_LOG(DEBUG, media_log_, num_video_keyframe_mismatches_,
                      kMaxVideoKeyframeMismatchLogs)
        << "ISO-BMFF container metadata for video frame indicates that the "
           "frame is "
        << (is_keyframe ? "" : "not ")
        << "a keyframe, but the video frame contents indicate the "
           "opposite.";
    // As of September 2018, it appears that all of Edge, Firefox, Safari
    // work with content that marks non-avc-keyframes as a keyframe in the
    // container. Encoders/muxers/old streams still exist that produce
    // all-keyframe mp4 video tracks, though many of the coded frames are
    // not keyframes (likely workaround due to the impact on low-latency
    // live streams until https://crbug.com/229412 was fixed).  We'll trust
    // the AVC frame's keyframe-ness over the mp4 container's metadata if
    // they mismatch. If other out-of-order codecs in mp4 (e.g. HEVC, DV)
    // implement keyframe analysis in their frame_bitstream_converter, we'll
    // similarly trust that analysis instead of the mp4.
    is_keyframe = analysis.is_keyframe.value();
  }

As the code comment show, chrome trust the AVC frame's keyframe-ness over the mp4 container's metadata. So nalu type in H264/HEVC should be more important than mp4 container box sdtp & trun description.

Answer 4

MP4 atoms for both files look very similiar (they have identical avcc section for sure)

Double check that, The code supplied suggests otherwise to me.

What's interesting (but not sure if it's of any importance), both files have different NALs format than input file (RPI camera produces video stream in Annex-B format, while output MP4 files contain NALs in AVCC format... or at least it looks like it's the case when I compare mdat atoms with input H264 data).

It is very important, mp4 will not work with annex b.