How to increase frame rate with Android CameraX ImageAnalysis?

Question

I\'m investigating the new CameraX API, in relation to how viable it might be to switch over from our current Camera2 system.

In our Camera2 system, we use an OpenG

Answer 1

Unfortunately, Iain Stanford answer did not help me. My app just crash after adding any of these lines to Camera2Config.Extender:

.setCaptureRequestOption(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF)

I got this error:

IllegalArgumentException: Unsupported session configuration combination

But, luckily, I found a way to get 60 FPS another way on Pixel 2 XL. Took some code from here. Unfortunately, it works only if I draw Preview as TextureView. If I use don't use AutoFitPreviewBuilder function and don't draw Preview as TextureView, all Extender settings will be ignored.

So, my code:

imageAnalysis = ImageAnalysis(createImageAnalysisConfig())

val previewConfig  = createImagePreviewConfig()
Camera2Config.Extender(previewConfig)
    .setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(60, 60))             
    .setCaptureRequestOption(CaptureRequest.CONTROL_AE_EXPOSURE_COMPENSATION, 1)

val preview = AutoFitPreviewBuilder.build(previewConfig.build(), viewFinder)

CameraX.bindToLifecycle(lifecycleOwner, imageAnalysis, preview)
preview.enableTorch(true)

Where AutoFitPreviewBuilder is function from android repo examples and viewFinder is TextureView, imageAnalysis is ImageAnalysisConfig.Builder().build and createImagePreviewConfig() returns PreviewConfig.Builder(). Btw, do not forget to set max resolution for camera in ImageAnalysisConfig & PreviewConfig:

.setMaxResolution(Size(800, 800))

Hope, it will help you.

Answer 2

So I've spent some more time investigating and I think I've come up with a solution for now.

It turns out, the ImageAnalysisConfig is not extendable, so you can't alter the camera configuration when just using one of those, so the default camera settings will be used which on my phone I think resulted in AE being on and hitting 16ish FPS.

If you spin up a PreviewConfig to run along side it at the same time, you can then extend this with a Camera2Config.Extender and alter the camera2 properties directly. This can increase the camera preview frame rate, and the Analyser will also start getting frames at the same rate.

So for example, I add this to my PreviewConfig...

    // Create Camera2 extender
    var camera2Extender = Camera2Config.Extender(previewConfig)
        .setCaptureRequestOption(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_TORCH)
        .setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
        .setCaptureRequestOption(CaptureRequest.SENSOR_FRAME_DURATION, 16666666)
        .setCaptureRequestOption(CaptureRequest.SENSOR_EXPOSURE_TIME, 20400000)

So this started hitting 30fps fine in the ImageAnalyser.

If I want to hit 60, I can set...

.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(60,60))

Obviously assuming the device support (60,60) target FPS range.

So it seems the full Camera2 logic is still available in CameraX, its just a bit clunky that its a little hidden away in a Camera2Config extender, and this only works with Preview use cases.

Answer 3

Alright, this was driving me crazy for hours.

Expanding upon Ian's answer for the newest version of CameraX, you can now extend ImageAnalysis directly. See CameraX equivalent of Camera2's CaptureRequest

So to get 60FPS, we can use this modified code (example in Java):

ImageAnalysis.Builder builder = new ImageAnalysis.Builder();
Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, new Range<Integer>(60, 60));
ImageAnalysis imageAnalysis = builder.build();