Face detection & draw circle using Android Camera2 API

Question

Currently I am trying to convert Camera2.Face to actual view\'s rect in order to draw circle over the face detected by the Camera2 API.

I am able to get number of fa

Answer 1

NEW: I've manage all my phone rotations. The offsetDxDy I guess depends on my layout, but if I've to tell you the truth I don't know why I put a value of 100. It works well on my Huawei P9 and I've found it in an empirical way. I still not have tried to find out if depends on my phone or on my XML layout or both.

Anyway the Matrices now are found, so you could adapt them so that they can fit your needs.

Note: my setRotation is not so general, because I didn't parametrized it upon

int orientationOffset = mCameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);

You can try do do it so that to have a full general code working with SENSOR_ORIENTATION different from the one of this example that is 270.

So this code works with a phone with an hardware camera sensor with orientation of 270.

The Huawei P9 has it.

Just to give you an idea of making the rotation bind to se HW sensor orientation that also works well on my P9 (but I don't have any other hardware to test)

if (mSwappedDimensions) {
    // Display Rotation 0
    mFaceDetectionMatrix.setRotate(orientationOffset);
    mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
    mFaceDetectionMatrix.postTranslate(mPreviewSize.getHeight() + offsetDxDy, mPreviewSize.getWidth() + offsetDxDy);
} else {
    // Display Rotation 90 e 270
    if (displayRotation == Surface.ROTATION_90) {
        mFaceDetectionMatrix.setRotate(orientationOffset + 90);
        mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
        mFaceDetectionMatrix.postTranslate(mPreviewSize.getWidth() + offsetDxDy, -offsetDxDy);
    } else if (displayRotation == Surface.ROTATION_270) {
        mFaceDetectionMatrix.setRotate(orientationOffset + 270);
        mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
        mFaceDetectionMatrix.postTranslate(-offsetDxDy, mPreviewSize.getHeight() + offsetDxDy);
    }
}

Here my final code (also available on GitHub)

int orientationOffset = mCameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);
Rect activeArraySizeRect = mCameraCharacteristics.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);

// Face Detection Matrix
mFaceDetectionMatrix = new Matrix();

Log.i("Test", "activeArraySizeRect1: (" + activeArraySizeRect + ") -> " + activeArraySizeRect.width() + ", " + activeArraySizeRect.height());
Log.i("Test", "activeArraySizeRect2: " + mPreviewSize.getWidth() + ", " + mPreviewSize.getHeight());
float s1 = mPreviewSize.getWidth() / (float)activeArraySizeRect.width();
float s2 = mPreviewSize.getHeight() / (float)activeArraySizeRect.height();
//float s1 = mOverlayView.getWidth();
//float s2 = mOverlayView.getHeight();
boolean mirror = (facing == CameraCharacteristics.LENS_FACING_FRONT); // we always use front face camera
boolean weAreinPortrait = true;
int offsetDxDy = 100;
if (mSwappedDimensions) {
    // Display Rotation 0
    mFaceDetectionMatrix.setRotate(270);
    mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
    mFaceDetectionMatrix.postTranslate(mPreviewSize.getHeight() + offsetDxDy, mPreviewSize.getWidth() + offsetDxDy);
} else {
    // Display Rotation 90 e 270
    if (displayRotation == Surface.ROTATION_90) {
        mFaceDetectionMatrix.setRotate(0);
        mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
        mFaceDetectionMatrix.postTranslate(mPreviewSize.getWidth() + offsetDxDy, -offsetDxDy);
    } else if (displayRotation == Surface.ROTATION_270) {
        mFaceDetectionMatrix.setRotate(180);
        mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
        mFaceDetectionMatrix.postTranslate(-offsetDxDy, mPreviewSize.getHeight() + offsetDxDy);
    }
}

This is the public github repo where you can find the code: https://github.com/shadowsheep1/android-camera2-api-face-recon. Hope it could help you.

Anyway just to give you also some theory, what you are doing is a 2D plane transformation. I mean you have a plane (the HW Sensor) and you have to remap the object on that plane on your preview plane.

So you have to take care of:

• Rotation: That depends on your HW Sensor rotation and the Phone Rotation.
• Mirroring: Horizontal mirroring that depends if you are using the front face camera or not and the Vertical mirroring that depends on the phone rotation). Mirroring is done with a '-' sign in the scaling matrix.
• Translation: That depends where your object has been placed by the rotation (that depends also from which rotation center your are dealing with) and translation. So you have to replace in your preview View your objects.

Math Theory

I've also write some technical post in my blog some time ago but they are in Italian.

• http://www.versionestabile.it/blog/trasformazioni-nel-piano/
• http://www.versionestabile.it/blog/coordinate-omogenee/