Android camera2 api. Setting multiple ImageReader surfaces gives blank output

Question

I have a camera2 implementation. The current setup is, it uses a texture view surface to display the actual camera view and an ImageReader surface for capturing images.

Now I want to capture preview frames as well. So I tried adding a new ImageReader surface for capturing frames. But when I add that surface to createCaptureSession request, the screen goes blank. What could possibly be wrong? Below is the code that I use to add surfaces to createCaptureSession

val surface = preview.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

val previewIRSurface = previewImageReader?.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

val captureSurface = captureImageReader?.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

try {
    val template = if (zsl) CameraDevice.TEMPLATE_ZERO_SHUTTER_LAG else CameraDevice.TEMPLATE_PREVIEW

    previewRequestBuilder = camera?.createCaptureRequest(template)
            ?.apply { addTarget(surface) }
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

    val surfaces: ArrayList<Surface> = arrayListOf(surface, previewIRSurface, captureSurface)

    camera?.createCaptureSession(surfaces, sessionCallback, backgroundHandler)

} catch (e: CameraAccessException) {
    throw RuntimeException("Failed to start camera session")
}

The initialization of ImageReaders is like this.

private fun prepareImageReaders() {

    val largestPreview = previewSizes.sizes(aspectRatio).last()

    previewImageReader?.close()

    previewImageReader = ImageReader.newInstance(
            largestPreview.width,
            largestPreview.height,
            internalOutputFormat,
            4 // maxImages
    ).apply { setOnImageAvailableListener(onPreviewImageAvailableListener, backgroundHandler) }

    val largestPicture = pictureSizes.sizes(aspectRatio).last()

    captureImageReader?.close()

    captureImageReader = ImageReader.newInstance(
            largestPicture.width,
            largestPicture.height,
            internalOutputFormat,
            2 // maxImages
    ).apply { setOnImageAvailableListener(onCaptureImageAvailableListener, backgroundHandler) }
}

More clarifications about the parameters used above:

• internalOutput format is either ImageFormat.JPEG or ImageFormat.YUV_420_888.
• Image sizes are based on best possible sizes
• It works good with either of the image readers individually but as soon as I add both together, blank screen!
• Testing on Samsung Galaxy S8 with Android Oreo (8.0)

The original code is here https://github.com/pvasa/cameraview-ex/blob/development/cameraViewEx/src/main/api21/com/priyankvasa/android/cameraviewex/Camera2.kt

Answer 1

maxImages == 4 may be too much and exhaust your RAM. Also, it's not clear what internalOutputFormat you use, and whether it is compatible with the largestPreview size.

The bottom line is, study the long list of tables for supported surface list parameter of createCaptureSession(). Depending on your camera capabilities, the three surfaces that you use, could be too much.

From the comments below, a working solution: "The error itself doesn't say much [...] but upon searching, it is found that multiple surfaces are not supported for JPEG format. Upon changing it to YUV_420_888 it works flawlessly."