Mapping image onto 3D face mesh

Question

I am using the iPhone X and ARFaceKit to capture the user\'s face. The goal is to texture the face mesh with the user\'s image.

I\'m only looking at a si

Answer 1

The start point is different:

Apply the following changes to your code:

//let vty = projectedPt!.y / (theCamera?.imageResolution.height)!
let vty = ((theCamera?.imageResolution.height)! - projectedPt!.y) / (theCamera?.imageResolution.height)!

You can get Normal Face.

Answer 2

Texturing the face mesh with the user's image is now available in the Face-Based sample code published by Apple (section Map Camera Video onto 3D Face Geometry).

One can map camera video onto 3D Face Geometry using this following shader modifier.

// Transform the vertex to the camera coordinate system.
float4 vertexCamera = scn_node.modelViewTransform * _geometry.position;

// Camera projection and perspective divide to get normalized viewport coordinates (clip space).
float4 vertexClipSpace = scn_frame.projectionTransform * vertexCamera;
vertexClipSpace /= vertexClipSpace.w;

// XY in clip space is [-1,1]x[-1,1], so adjust to UV texture coordinates: [0,1]x[0,1].
// Image coordinates are Y-flipped (upper-left origin).
float4 vertexImageSpace = float4(vertexClipSpace.xy * 0.5 + 0.5, 0.0, 1.0);
vertexImageSpace.y = 1.0 - vertexImageSpace.y;

// Apply ARKit's display transform (device orientation * front-facing camera flip).
float4 transformedVertex = displayTransform * vertexImageSpace;

// Output as texture coordinates for use in later rendering stages.
_geometry.texcoords[0] = transformedVertex.xy;

Answer 3

For proper UV mapping you need to use ARSCNFaceGeometry class instead of ARFaceGeometry class you're using in your code.

ARSCNFaceGeometry is a SceneKit's representation of face topology for use with face information provided by an ARSession. It's used for a quick visualization of face geometry using SceneKit's rendering engine.

ARSCNFaceGeometry class is a subclass of SCNGeometry that wraps the mesh data provided by the ARFaceGeometry class. You can use ARSCNFaceGeometry to quickly and easily visualize face topology and facial expressions provided by ARKit in a SceneKit view.

But ARSCNFaceGeometry is available only in SceneKit views or renderers that use Metal. This class is not supported for OpenGL-based SceneKit rendering.