How to use ScriptIntrinsicYuvToRGB (converting byte[] yuv to byte[] rgba)

Question

I have byte[] yuvByteArray (540x360 image captured from Camera.PreviewCallback.onPreviewFrame method and dumped into assets/yuv.bin file). I want to convert byte[] yuv to byte[] rgba array, using the following code (based on LivePreview android example).

But I receive outBytes rgba array filled with zeros after forEach and copying out allocation to outBytes. What's wrong with my code?

---

package hellorender;        
import android.app.Activity; 
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.os.Bundle;
import android.support.v8.renderscript.Allocation;
import android.support.v8.renderscript.Element;
import android.support.v8.renderscript.RenderScript;
import android.support.v8.renderscript.ScriptIntrinsicYuvToRGB;
import android.support.v8.renderscript.Type;
import android.widget.ImageView;
import hellorender.R;

import java.io.IOException;
import java.io.InputStream;

public class HelloRenderActivity extends Activity {

    public static final int W = 540;
    public static final int H = 360;
    private RenderScript rs;
    private ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        AssetManager assets = getAssets();
        byte[] yuvByteArray = new byte[291600];
        byte[] outBytes = new byte[W * H * 4];

        InputStream is = null;
        try {
            is = assets.open("yuv.bin");
            System.out.println("read: " + is.read(yuvByteArray));
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        ImageView iv = (ImageView) findViewById(R.id.image);
        rs = RenderScript.create(this);
        yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.RGBA_8888(rs));


        Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs))
                .setX(W).setY(H)
                .setYuvFormat(android.graphics.ImageFormat.NV21);
        Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);


        Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs))
                .setX(W).setY(H);
        Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);

        in.copyFrom(yuvByteArray);

        yuvToRgbIntrinsic.setInput(in);
        yuvToRgbIntrinsic.forEach(out);

        out.copyTo(outBytes);

        Bitmap bmpout = Bitmap.createBitmap(W, H, Bitmap.Config.ARGB_8888);
        out.copyTo(bmpout);

        iv.setImageBitmap(bmpout);
    }

}

Answer 1

yuv.bin file is definitely in NV21 format, as it captures here http://developer.android.com/reference/android/hardware/Camera.PreviewCallback.html#onPreviewFrame

setYuvFormat method is from API level 18, I removed it

So this code works fine:

    rs = RenderScript.create(this);
    yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

    Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(yuvByteArray.length);
    Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);

    Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(W).setY(H);
    Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);

    in.copyFrom(yuvByteArray);

    yuvToRgbIntrinsic.setInput(in);
    yuvToRgbIntrinsic.forEach(out);

Answer 2

Using API18+ wilddev´s answer can be slightly improved as you don´t need Type.Builder objects. Do all this stuff in the onCreate() method:

aIn = Allocation.createSized(rs, Element.U8(rs), H*W*3/2); // what the f**k ?  This is 12 bit per pixel, giving the length of the camera data byte array !

bmpout = Bitmap.createBitmap(W, H, Bitmap.Config.ARGB_8888); // you will need this bitmap for output anyway

aOut = Allocation.createFromBitmap(rs, bmpout);  // tada !

// and set the script´s input
yuvToRgbIntrinsic.setInput(aIn);

// as pixel data are stored as int (one byte for red, green, blue, alpha), you first need an int[] array
int rgba[] = new int[w*h];   // the rgba[] array

Now you can continue with these lines or put them into the onPreviewFrame() method:

aIn.copyFrom(data);  // or aIn.copyFromUnchecked(data);  // which is faster and safe for camera data

yuvToRgbIntrinsic.forEach(aOut);  // execute the script

aOut.copyTo(bmpout);  // copy result from aOut to bmpout

// if you need the rgba-values, do
bmpout.getPixels(rgba, 0, W, 0, 0, W, H);

// now you may loop through the rgba[] array and extraxt the r,g,b,a values 
// and put them into a byte[] array(s), BUT this will surely have impact on the performance when doing in Java

Answer 3

Our internal test app uses the following sequence to create the YUV allocation.

    tb = new Type.Builder(mRS, Element.createPixel(mRS, 
              Element.DataType.UNSIGNED_8, Element.DataKind.PIXEL_YUV));
    tb.setX(mWidth);
    tb.setY(mHeight);
    tb.setYuvFormat(android.graphics.ImageFormat.NV21);
    mAllocationIn = Allocation.createTyped(mRS, tb.create(), Allocation.USAGE_SCRIPT);

Then inside the callback that new YUV data is available do

    mAllocationIn.copyFrom(theYuvByteArray);

Answer 4

Change your output type passed to the constructor of ScriptIntrinsicYubToRGB to be Element.U8_4 instead of Element.RGBA_8888. You'll need the same for the Type.Builder used to create the output Allocation.

Answer 5

Kotlin

        val rs = RenderScript.create(CONTEXT_HERE)
        val yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs))

        val yuvType = Type.Builder(rs, Element.U8(rs)).setX(byteArray.size)
        val inData = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT)

        val rgbaType = Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height)
        val outData = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT)

        inData.copyFrom(byteArray)

        yuvToRgbIntrinsic.setInput(inData)
        yuvToRgbIntrinsic.forEach(outData)

        val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
        outData.copyTo(bitmap)