How to improve People Occlusion in ARKit 3.0

问题

We are working on a demo app using people occlusion in ARKit. Because we want to add videos in the final scene, we use SCNPlanes to render the video using a SCNBillboardConstraint to ensure they are facing the right way. These videos are also partially transparent, using a custom shader on the SCNMaterial we apply (thus playing 2 videos at once).

Now we have some issues where the people occlusion is very iffy (see image). The video we are using to test is a woman with a dark pants and skirt (if you were wondering what the black is in the image).

The issues we have are that the occlusion do not always line up with the person (as visible in the picture), and that someone's hair is not always correctly detected.

Now our question is what causes these issues? And how can we improve the problems until they look like this? We are currently exploring if the issues are because we are using a plane, but simply using a SCNBox does not fix the problem.

回答1:

I reckon you can't improve People Occlusion feature inside ARKit 3.0 module. Perhaps, you can do it in ARKit 4.0 or higher. But now it's possible only by using Metal and, believe me, it isn't easy.

Why does this issue happen when you use People Occlusion?

It's all due to the nature of fifth channel – ZDepth channel. We all know that a rendered image of 3D scene can contain 5 main channels for digital compositing – Red, Green, Blue, Alpha, and ZDepth.

There are, of course, other useful render passes (a.k.a AOVs) for compositing: MotionVectors, IDs, AmbientOcclusion, Specularity, Deep, Normals, PointPosition, UVs, Reflection, Refraction, SubSurfaceScattering, Disparity, etc.

But here, we mainly interested in first five ones, especially in ZDepth.

However, ZDepth channel has three serious drawbacks in ARKit.

Problem 1. No Anti-aliasing for ZDepth.

Rendering ZDepth channel in any High-End software (like NUKE, Fusion, Maya or Houdini), most of the time, results in edge artefacts (called aliased edges). There's no exception for game engines too – SceneKit, RealityKit, Unity, Unreal, Stingray or Metal.

You could say that before rendering we must turn on a feature called Anti-aliasing. And, yes, it works fine for almost all the channels, but not for ZDepth. This channel is non-anti-aliased at all. Frankly saying, there's one contemporary solution for robust compositing fixing ZDepth issue – you need to use a Deep channel instead of a ZDepth channel. But no one game engine supports it because Deep channel is dauntingly huge. So deep channel comp is neither for game engines, nor for ARKit.

Please, read about Deep compositing vs Depth compositing here.

Problem 2. Image Resolution of ZDepth.

Regular ZDepth channel must be rendered in 32-bit, even if RGB and Alpha channels are both 8-bit only. Color bit depth of 32-bit files is a heavy burden for CPU and GPU. And remember about ARKit image resolution – here are a compositing of Foreground Character over 3D model and over Background Character. Don't you think it's too much for your device, especially at high resolution? However, rendering ZDepth channel in 16-bit or 8-bit compresses the depth of your real scene.

To lower a burden on CPU and GPU and to save battery life, Apple engineers decided to use a scaled-down ZDepth image at capture stage and then scale a rendered ZDepth image up to a Full-Res and multiply it by Alpha channel (a.k.a. segmentation). Thus, this led us to such nasty artefacts that we can see at your picture.

Please, look at Presentation Slides pdf of Bringing People into AR here.

Problem 3. Frame rate of ZDepth.

Third problem stems from the fact that ARKit works at 60 fps. Lowering only ZDepth image resolution doesn't totally fix a problem. So, the next logical step for Apple engineers is to lower a frame rate to 15 fps. This leads us to artefacts too (like fps delay for ZDepth channel).

You can't change the quality of your Final Composited Image when you use a Type Property:

static var personSegmentationWithDepth: ARConfiguration.FrameSemantics { get }

because it's a gettable property and there's no settings for ZDepth quality in ARKit.

P.S.

I believe that someone might improve ZDepth compositing features in ARKit 3.0 via developing with Metal module but it's a real challenge now!

You must look at sample code of People Occlusion in Custom Renderers app here.

来源：https://stackoverflow.com/questions/58269394/how-to-improve-people-occlusion-in-arkit-3-0