Very slow framerate with AVFoundation and Metal in MacOS

Question

I\'m trying to adapt Apple\'s AVCamFilter sample to MacOS. The filtering appears to work, but rendering the processed image through Metal gives me a framerate of several seconds

Answer 1

Capture device delegates don't own the sample buffers they receive in their callbacks, so it's incumbent on the receiver to make sure they're retained for as long as their contents are needed. This project doesn't currently ensure that.

Rather, by calling CMSampleBufferGetImageBuffer and wrapping the resulting pixel buffer in a texture, the view controller is allowing the sample buffer to be released, meaning that future operations on its corresponding pixel buffer are undefined.

One way to ensure the sample buffer lives long enough to be processed is to add a private member to the camera view controller class that retains the most-recently received sample buffer:

private var sampleBuffer: CMSampleBuffer!

and then set this member in the captureOutput(...) method before calling processVideo. You don't even have to reference it further; the fact that it's retained should prevent the stuttery and unpredictable behavior you're seeing.

This solution may not be perfect, since it retains the sample buffer for longer than strictly necessary in the event of a capture session interruption or other pause. You can devise your own scheme for managing object lifetimes; the important thing is to ensure that the root sample buffer object sticks around until you're done with any textures that refer to its contents.