Has anyone managed to obtain a YUV_420_888 frame using RenderScript and the new Camera API?

Question

I'm using RenderScript and Allocation to obtain YUV_420_888 frames from the Android Camera2 API, but once I copy the byte[] from the Allocation I receive only the Y plane from the 3 planes which compose the frame, while the U and V planes values are set to 0 in the byte[]. I'm trying to mimic the onPreviewframe from the previos camera API in order to perform in app processing of the camera frames. My Allocation is created like:

Type.Builder yuvTypeBuilderIn = new Type.Builder(rs, Element.YUV(rs));
    yuvTypeBuilderIn.setX(dimensions.getWidth());
    yuvTypeBuilderIn.setY(dimensions.getHeight());
    yuvTypeBuilderIn.setYuvFormat(ImageFormat.YUV_420_888);
    allocation = Allocation.createTyped(rs, yuvTypeBuilderIn.create(),
            Allocation.USAGE_IO_INPUT | Allocation.USAGE_SCRIPT);

while my script looks like:

#pragma version(1)
#pragma rs java_package_name(my_package)
#pragma rs_fp_relaxed

rs_allocation my_frame;

Answer 1

The Android sample app HdrViewfinderDemo uses RenderScript to process YUV data from camera2.

https://github.com/googlesamples/android-HdrViewfinder

Specifically, the ViewfinderProcessor sets up the Allocations, and hdr_merge.rs reads from them.

Answer 2

Yes I did it, since I couldn't find anything useful. But I didn't go the proposed way of defining an allocation to the surface. Instead I just converted the output of the three image planes to RGB. The reason for this approach is that I use the YUV420_888 data twofold. First on a high frequency basis just the intensity values (Y). Second, I need to make some color Bitmaps too. Thus, the following solution. The script takes about 80ms for a 1280x720 YUV_420_888 image, maybe not ultra fast, but ok for my purpose.

UPDATE: I deleted the code here, since I wrote a more general solution here YUV_420_888 -> Bitmap conversion that takes into account pixelStride and rowStride too.

Answer 3

I think that you can use an ImageReader to get the frames of you camera into YUV_420_888

reader = ImageReader.newInstance(previewSize.getWidth(), previewSize.getHeight(), ImageFormat.YUV_420_888, 2);

Then you set an OnImageAvailableListener to the reader :

 reader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
            @Override
            public void onImageAvailable(ImageReader reader) {
             int jump = 4; //Le nombre d'image à sauter avant d'en traiter une, pour liberer de la mémoire
             Image readImage = reader.acquireNextImage();

             readImage.getPlane[0] // The Y plane
             readImage.getPlane[1] //The U plane
             readImage.getPlane[2] //The V plane

             readImage.close();
            }
        }, null);

Hope that will help you

Answer 4

I'm using almost the same method as widea in their answer.

The exception you keep getting after ~50 frames might be due to the fact that you're processing all the frames by using acquireNextImage. The documentation suggest to:

Warning: Consider using acquireLatestImage() instead, as it will automatically release older images, and allow slower-running processing routines to catch up to the newest frame. [..]

So in case your exception is a IllegalStateException, switching to acquireLatestImage might help.

And make sure you call close() on all images retrieved from ImageReader.