Zoom Camera2 Preview using TextureView

Question

i have a Problem with my Preview Zoom for the Camera2 API. I am using a TextureView.

I want to zoom only the preview Stream that was showed in the TextureView.

I want to zoom the Area where i use the Zoom Gesture.

I use the SimpleOnScaleGestureListener!

I added following Code. The zoomingFactor and the x and y Position are right.

 private void updateTextureViewSize(float xPosi,float yPosi, float scale){
        float scaleX = 1.0f;
        float scaleY = 1.0f;




        float mVideoWidth = mCamcontrol.getmPreviewSize().getWidth();
        float mVideoHeight = mCamcontrol.getmPreviewSize().getHeight();

        int rotation = getWindowManager().getDefaultDisplay().getRotation();
        RectF viewRect = new RectF(0, 0, 1440, 2560);
        RectF bufferRect = new RectF(0, 0, mVideoHeight, mVideoWidth);

        bufferRect.offset(xPosi - bufferRect.centerX(), yPosi -    bufferRect.centerY());

         //16:9 faktor
        scaleX = ((mScale * scale) / 9f) * 16f;
        scaleY = ((mScale * scale) / 16f) * 9f;

        Matrix matrix = new Matrix();

        matrix.setRectToRect(bufferRect, viewRect, Matrix.ScaleToFit.FILL);
        scalefactorView.setText(String.valueOf(xPosi) + "  " + String.valueOf(yPosi));

        matrix.setScale(scaleY, scaleX, xPosi, yPosi);
        matrix.postRotate(90 * (rotation - 2), xPosi, yPosi);

        mTextureView.setTransform(matrix);


}

Zooming is Right, but not the Position where i Zoom. For Example! When i zoom on the position right/middle i see only the left/top rectangle of the Stream.

I added the following pictures to unterstand the problem.

Answer 1

Android Camera2 api : Pinch Zoom In/Out

Use this sample code for Camera2Basic from google developers. https://github.com/googlesamples/android-Camera2Basic

Now declare two class variables –

public float finger_spacing = 0;
public int zoom_level = 1;

and update the given onTouch() method.

public boolean onTouch(View v, MotionEvent event) {
    try {
        Activity activity = getActivity();
        CameraManager manager = (CameraManager) activity.getSystemService(Context.CAMERA_SERVICE);
        CameraCharacteristics characteristics = manager.getCameraCharacteristics(mCameraId);
        float maxzoom = (characteristics.get(CameraCharacteristics.SCALER_AVAILABLE_MAX_DIGITAL_ZOOM))*10;

        Rect m = characteristics.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);
        int action = event.getAction();
        float current_finger_spacing;

        if (event.getPointerCount() > 1) {
            // Multi touch logic
            current_finger_spacing = getFingerSpacing(event);
            if(finger_spacing != 0){
                if(current_finger_spacing > finger_spacing && maxzoom > zoom_level){
                    zoom_level++;
                } else if (current_finger_spacing < finger_spacing && zoom_level > 1){
                    zoom_level--;
                }
                int minW = (int) (m.width() / maxzoom);
                int minH = (int) (m.height() / maxzoom);
                int difW = m.width() - minW;
                int difH = m.height() - minH;
                int cropW = difW /100 *(int)zoom_level;
                int cropH = difH /100 *(int)zoom_level;
                cropW -= cropW & 3;
                cropH -= cropH & 3;
                Rect zoom = new Rect(cropW, cropH, m.width() - cropW, m.height() - cropH);
                mPreviewRequestBuilder.set(CaptureRequest.SCALER_CROP_REGION, zoom);
            }
            finger_spacing = current_finger_spacing;
        } else{
            if (action == MotionEvent.ACTION_UP) {
                //single touch logic
            }
        }

        try {
            mCaptureSession
                .setRepeatingRequest(mPreviewRequestBuilder.build(), mCaptureCallback, null);
        } catch (CameraAccessException e) {
            e.printStackTrace();
        } catch (NullPointerException ex) {
            ex.printStackTrace();
        }
    } catch (CameraAccessException e) {
        throw new RuntimeException("can not access camera.", e);
    }
    return true;
}


//Determine the space between the first two fingers
@SuppressWarnings("deprecation")
private float getFingerSpacing(MotionEvent event) {
    float x = event.getX(0) - event.getX(1);
    float y = event.getY(0) - event.getY(1);
    return (float) Math.sqrt(x * x + y * y);
}

Answer 2

Thanks to @arin 's answer, I made an improved version.

His code is basically working, but there are 2 problems:
1) Readability - actually I don't know what is going on calculating the Rect zoom
2) In my Android 7.1.1 device, the preview will freeze if the zoom is big to a certain extent. Since I solved this problem with the code below, I am pretty sure it is because the original code allowed over-zooming beyond camera's maximum zoom ratio.
(In fact, I don't know why he needs to apply *10 on the ratio returned by CameraCharacteristics)

Below are my codes: (I do this all inside my custom TextureView, which also stores my Camera2 objects and logics):

Related Member variables:

protected CameraCharacteristics cameraCharacteristics;
protected CameraCaptureSession captureSession;
protected CaptureRequest.Builder previewRequestBuilder;

//Zooming
protected float fingerSpacing = 0;
protected float zoomLevel = 1f;
protected float maximumZoomLevel;
protected Rect zoom;

Right after you get CameraCharacteristics from CameraManager, probably in some initial setup:

maximumZoomLevel = cameraCharacteristics.get(CameraCharacteristics.SCALER_AVAILABLE_MAX_DIGITAL_ZOOM);

override onTouchEvent:

@Override
public boolean onTouchEvent(MotionEvent event) {
    try {
        Rect rect = cameraCharacteristics.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);
        if (rect == null) return false;
        float currentFingerSpacing;

        if (event.getPointerCount() == 2) { //Multi touch.
            currentFingerSpacing = getFingerSpacing(event);
            float delta = 0.05f; //Control this value to control the zooming sensibility
            if (fingerSpacing != 0) {
                if (currentFingerSpacing > fingerSpacing) { //Don't over zoom-in
                    if ((maximumZoomLevel - zoomLevel) <= delta) {
                        delta = maximumZoomLevel - zoomLevel;
                    }
                    zoomLevel = zoomLevel + delta;
                } else if (currentFingerSpacing < fingerSpacing){ //Don't over zoom-out
                    if ((zoomLevel - delta) < 1f) {
                        delta = zoomLevel - 1f;
                    }
                    zoomLevel = zoomLevel - delta;
                }
                float ratio = (float) 1 / zoomLevel; //This ratio is the ratio of cropped Rect to Camera's original(Maximum) Rect
                //croppedWidth and croppedHeight are the pixels cropped away, not pixels after cropped
                int croppedWidth = rect.width() - Math.round((float)rect.width() * ratio);
                int croppedHeight = rect.height() - Math.round((float)rect.height() * ratio);
                //Finally, zoom represents the zoomed visible area
                zoom = new Rect(croppedWidth/2, croppedHeight/2,
                        rect.width() - croppedWidth/2, rect.height() - croppedHeight/2);
                previewRequestBuilder.set(CaptureRequest.SCALER_CROP_REGION, zoom);
            }
            fingerSpacing = currentFingerSpacing;
        } else { //Single touch point, needs to return true in order to detect one more touch point
            return true;
        }
        captureSession.setRepeatingRequest(previewRequestBuilder.build(), captureCallback, null);
        return true;
    } catch (final Exception e) {
        //Error handling up to you
        return true;
    }
}

And the getFingerSpacing method:

private float getFingerSpacing(MotionEvent event) {
    float x = event.getX(0) - event.getX(1);
    float y = event.getY(0) - event.getY(1);
    return (float) Math.sqrt(x * x + y * y);
}

Finally don't forget to set the crop region when you actually take the photo. My code is base on this Camera2Basic, I do this inside the captureStillPicture() method:

        //Zoom
        if (zoom != null) {
            captureBuilder.set(CaptureRequest.SCALER_CROP_REGION, zoom);
        }

Answer 3

@arin Answer is working thank @arin just one thing zoom sensitivity too high.

To control this i make some changes in might be useful to you.

Change zoom_level data type to double

public int zoom_level = 1; to public double zoom_level = 1;

Then increase or decrease zoom_level with low value i use 0.4

if (current_finger_spacing > finger_spacing && maxzoom > zoom_level) {
        zoom_level = zoom_level + .4;
        //zoom_level++;
    } else if (current_finger_spacing < finger_spacing && zoom_level > 1) {
        zoom_level = zoom_level - .4;
        //zoom_level--;
      }

Answer 4

Here is a Pan and Zoom object from Camera2 that I made to work using the OnScaleGestureListener and SimpleOnGestureListener-onScroll outputs. This will only work as expected if you have a camera with support level > LEGACY, as LEGACY only supports crop to center.

Two caveats: One is that this is currently NOT set up to output to JPEG output, as rectangles for JPEG outputs must have dimensions which are multiple of 16 (See why here). The second is that I've locked my screen to landscape mode, and my camera is locked to landscape as well, but it should be possible to deal with screen rotations after a few tweaks.

You'll need to pass in the screen dimensions

DisplayMetrics displayMetrics = new DisplayMetrics();
((Activity) mView.getContext()).getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);

the Maximum Camera Digital Zoom

try {
    CameraManager manager = (CameraManager) mView.getContext().getSystemService(Context.CAMERA_SERVICE);
    CameraCharacteristics characteristics = manager.getCameraCharacteristics(mCameraID);
    float maxZoom = (characteristics.get(CameraCharacteristics.SCALER_AVAILABLE_MAX_DIGITAL_ZOOM));
} catch (CameraAccessException e) {
   e.printStackTrace();
}

the Camera Sensor's Active Array Size

try {
    CameraManager manager = (CameraManager) mView.getContext().getSystemService(Context.CAMERA_SERVICE);
    CameraCharacteristics characteristics = manager.getCameraCharacteristics(mCameraID);
    Rect rectInit = characteristics.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);
} catch (CameraAccessException e) {
    e.printStackTrace();
}

Here is my object initialization

mScaler = new CamScaler(maxZoom, rectInit.width(), rectInit.height(), displayMetrics.heightPixels, displayMetrics.widthPixels);

the CamScaler class

public class CamScaler {
    private final float ZOOM_MIN = 1.0f;
    private final int X_MIN = 0;
    private final int Y_MIN = 0;
    private int displayWidth;
    private int displayHeight;

    private Rect current_rect;
    private int xCenter;
    private int yCenter;
    private int xWidth;
    private int yHeight;
    private int xMax;
    private int yMax;
    private float zoomMax;
    private float zoomCurrent;

    public CamScaler(float zoomMax, int xMax, int yMax, int displayHeight, int displayWidth) {
        this.xMax = xMax;
        this.yMax = yMax;
        this.zoomMax = zoomMax;

        current_rect = new Rect(X_MIN,Y_MIN, xMax, yMax); //(0,0,xMax,yMax) as the starting rectangle
        zoomCurrent = ZOOM_MIN;
        xWidth = current_rect.width();
        yHeight = current_rect.height();
        xCenter = current_rect.centerX();
        yCenter = current_rect.centerY();

        this.displayHeight = displayHeight;
        this.displayWidth = displayWidth;
    }

    public void pan(float distanceX, float distanceY){
        //calculate the shift in the we want to take on the camera sensor with respect to the distance moved on the screen
        int xShift = Math.round((distanceX/displayWidth)*xWidth); //scales down to a percentage of the current view width->converts to a pixel shift
        int yShift = Math.round((distanceY/displayHeight)*yHeight); //scales down to a percentage of the current view height->converts to a pixel shift

        //check if the shift will push us pass our maximums, this should account for both negative and positive values of xShift and yShift correctly
        if ( !((xCenter + Math.round(xWidth/2.0) + xShift < xMax) && (xCenter - Math.round(xWidth/2.0) + xShift > 0))) { //if not within xBounds, set xShift to 0
            xShift = 0;
        }
        if ( !((yCenter + Math.round(yHeight/2) + yShift < yMax) && (yCenter - Math.round(yHeight/2.0) + yShift > 0))) { //if not within yBounds, set yShift to 0
            yShift = 0;
        }

        Log.d("Scaler", "pan: xShift" + xShift + " yShift " + yShift);
        current_rect.offset(xShift,yShift);
        Log.d("Scaler", "pan: current_rect" + current_rect.toString());
        xCenter = current_rect.centerX(); //update center
        yCenter = current_rect.centerY(); //update center
    }

    public void zoom(float scale_change){
        if ( (zoomCurrent*scale_change < zoomMax) && (zoomCurrent*scale_change > ZOOM_MIN) ){ //if we are within zoom bounds
            zoomCurrent *= scale_change; //update the zoom factor
            int newWidthHalf = (int)Math.floor(xMax/zoomCurrent/2.0);
            int newHeightHalf = (int)Math.floor(yMax/zoomCurrent/2.0);
            int xTempCenter = xCenter;
            int yTempCenter = yCenter;

            //if at edge we need to shift and scale
            if (xCenter + newWidthHalf > xMax) { //if at right edge
                xTempCenter = xMax - newWidthHalf; //shift center to the left
            } else if (xCenter - newWidthHalf < 0) { //if at left edge
                xTempCenter = newWidthHalf; //shift center to the right
            }
            if (yCenter + newHeightHalf > yMax) { //if at bottom
                yTempCenter = yMax - newHeightHalf; //shift center up
            } else if (yCenter - newHeightHalf < 0) { //if at top
                yTempCenter = newHeightHalf; //shift center down
            }
            Log.d("Scaler", "zoom: " + zoomCurrent);
            Log.d(TAG, "current center(x,y) " + xTempCenter + " " + yTempCenter + "current halfwidths(x,y) " + newWidthHalf + " " + newHeightHalf);
            current_rect.set(xTempCenter - newWidthHalf, yTempCenter - newHeightHalf,xTempCenter + newWidthHalf, yTempCenter + newHeightHalf);
            Log.d("Scaler", "zoom: current_rect" + current_rect.toString());
            xWidth = current_rect.width();
            yHeight = current_rect.height();
            xCenter = current_rect.centerX(); //update center
            yCenter = current_rect.centerY(); //update center
        } //if not in digital zoom bounds, do nothing
    }

    public Rect getCurrentView() {
        return current_rect;
    }
}

And how to use it

public void pan(float distanceX, float distanceY){
    if (mScaler != null) {
        synchronized (mScaler) {
            mScaler.pan(distanceX, distanceY);
            try {
                mPreviewRequestBuilder.set(CaptureRequest.SCALER_CROP_REGION, mScaler.getCurrentView());
                mCaptureSession.setRepeatingRequest(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
    }
}

public void zoom(float scale_factor) {
    if (mScaler!= null) {
        synchronized (mScaler) {
            mScaler.zoom(scale_factor);
            try {
                mPreviewRequestBuilder.set(CaptureRequest.SCALER_CROP_REGION, mScaler.getCurrentView());
                mCaptureSession.setRepeatingRequest(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
    }
}

The inputs to these functions are directly passed through from the gesture listeners

I hope this helps someone!

Answer 5

In addition to arin answer,Need to add captureBuilder.set(CaptureRequest.SCALER_CROP_REGION, zoom); To captureStillPicture() method to let zoom take effect on capture