Android camera2 API - Display processed frame in real time

Question

I'm trying to create an app that processes camera images in real time and displays them on screen. I'm using the camera2 API. I have created a native library to process the images using OpenCV.

So far I have managed to set up an ImageReader that receives images in YUV_420_888 format like this.

        mImageReader = ImageReader.newInstance(
                mPreviewSize.getWidth(),
                mPreviewSize.getHeight(),
                ImageFormat.YUV_420_888,
                4);
        mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mImageReaderHandler);

From there I'm able to get the image planes (Y, U and V), get their ByteBuffer objects and pass them to my native function. This happens in the mOnImageAvailableListener:

        Image image = reader.acquireLatestImage();

        Image.Plane[] planes = image.getPlanes();
        Image.Plane YPlane = planes[0];
        Image.Plane UPlane = planes[1];
        Image.Plane VPlane = planes[2];

        ByteBuffer YPlaneBuffer = YPlane.getBuffer();
        ByteBuffer UPlaneBuffer = UPlane.getBuffer();
        ByteBuffer VPlaneBuffer = VPlane.getBuffer();

        myNativeMethod(YPlaneBuffer, UPlaneBuffer, VPlaneBuffer, w, h);

        image.close();

On the native side I'm able to get the data pointers from the buffers, create a cv::Mat from the data and perform the image processing.

Now the next step would be to show my processed output, but I'm unsure how to show my processed image. Any help would be greatly appreciated.

Answer 1

Generally speaking, you need to send the processed image data to an Android view.

The most performant option is to get an android.view.Surface object to draw into - you can get one from a SurfaceView (via SurfaceHolder) or a TextureView (via SurfaceTexture). Then you can pass that Surface through JNI to your native code, and there use the NDK methods:

• ANativeWindow_fromSurface to get an ANativeWindow
• The various ANativeWindow methods to set the output buffer size and format, and then draw your processed data into it.

Use setBuffersGeometry() to configure the output size, then lock() to get an ANativeWindow_Buffer. Write your image data to ANativeWindow_Buffer.bits, and then send the buffer off with unlockAndPost().

Generally, you should probably stick to RGBA_8888 as the most compatible format; technically only it and two other RGB variants are officially supported. So if your processed image is in YUV, you'd need to convert it to RGBA first.

You'll also need to ensure that the aspect ratio of your output view matches that of the dimensions you set; by default, Android's Views will just scale those internal buffers to the size of the output View, possibly stretching it in the process.

You can also set the format to one of Android's internal YUV formats, but this is not guaranteed to work!

Answer 2

I've tried the ANativeWindow approach, but it's a pain to set up and I haven't managed to do it correctly. In the end I just gave up and imported OpenCV4Android library which simplifies things by converting camera data to a RGBA Mat behind the scenes.