Why AVSampleBufferDisplayLayer stops showing CMSampleBuffers taken from AVCaptureVideoDataOutput's delegate?

Question

I want to display some CMSampleBuffer\'s with the AVSampleBufferDisplayLayer, but it freezes after showing the first sample.

I get the samplebuffers from the AVCaptureV

Answer 1

I came across this problem in the same context, trying to take the output from AVCaptureVideoDataOutput and display it in a AVSampleDisplay layer.

If your frames come out in display order, then the fix is very easy, just set the display immediately flag on the CMSampleBufferRef.

Get the sample buffer returned by the delegate and then...

CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);

If your frames come out in encoder order (not display order), then the time stamps on the CMSampleBuffer need to be zero biased and restamped such that the first frames timestamp is equal to time 0.

 double pts = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer));

 // ptsStart is equal to the first frames presentationTimeStamp so playback starts from time 0.
 CMTime presentationTimeStamp = CMTimeMake((pts-ptsStart)*1000000,1000000);

 CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

Update:

I ran into a situation where some video still wasn't playing smoothly when I used the zero bias method and I investigated further. The correct answer seems to be using the PTS from the first frame you intend to play.

My answer is here, but I will post it here, too.

Set rate at which AVSampleBufferDisplayLayer renders sample buffers

The Timebase needs to be set to the presentation time stamp (pts) of the first frame you intend to decode. I was indexing the pts of the first frame to 0 by subtracting the initial pts from all subsequent pts and setting the Timebase to 0. For whatever reason, that didn't work with certain video.

You want something like this (called before a call to decode):

CMTimebaseRef controlTimebase;
CMTimebaseCreateWithMasterClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase );

displayLayer.controlTimebase = controlTimebase;

// Set the timebase to the initial pts here
CMTimebaseSetTime(displayLayer.controlTimebase, CMTimeMake(ptsInitial, 1));
CMTimebaseSetRate(displayLayer.controlTimebase, 1.0);

Set the PTS for the CMSampleBuffer...

CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

And maybe make sure display immediately isn't set....

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanFalse);

This is covered very briefly in WWDC 2014 Session 513.