Calculating homography matrix using arbitrary known geometrical relations

Question

I am using OpenCV for an optical measurement system. I need to carry out a perspective transformation between two images, captured by a digital camera. In the field of view of t

Answer 1

This question is quite old, but it is interesting and it might be useful to someone.

First, here is how I understood the problem presented in the question:

You have two images I₁ and I₂ acquired by the same digital camera at two different positions. These images both show a set of markers which all lie in a common plane p_m. There is also a measured object, whose visible surface lies in a plane p_o parallel to the marker's plane but with a small offset. You computed the homography H^m₁₂ mapping the markers positions in I₁ to the corresponding markers positions in I₂ and you measured the offset d_m-o between the planes p_o and p_m. From that, you would like to calculate the homography H^o₁₂ mapping points on the measured object in I₁ to the corresponding points in I₂.

A few remarks on this problem:

First, notice that an homography is a relation between image points, whereas the distance between the markers' plane and the object's plane is a distance in world coordinates. Using the latter to infer something about the former requires to have a metric estimation of the camera poses, i.e. you need to determine the euclidian and up-to-scale relative position & orientation of the camera for each of the two images. The euclidian requirement implies that the digital camera must be calibrated, which should not be a problem for an "optical measurement system". The up-to-scale requirement implies that the true 3D distance between two given 3D points must be known. For instance, you need to know the true distance l₀ between two arbitrary markers.

Since we only need the relative pose of the camera for each image, we may choose to use a 3D coordinate system centered and aligned with the coordinate system of the camera for I₁. Hence, we will denote the projection matrix for I₁ by P₁ = K₁ * [ I | 0 ]. Then, we denote the projection matrix for I₂ (in the same 3D coordinate system) by P₂ = K₂ * [ R₂ | t₂ ]. We will also denote by D₁ and D₂ the coefficients modeling lens distortion respectively for I₁ and I₂.

As a single digital camera acquired both I₁ and I₂, you may assume that K₁ = K₂ = K and D₁ = D₂ = D. However, if I₁ and I₂ were acquired with a long delay between the acquisitions (or with a different zoom, etc), it will be more accurate to consider that two different camera matrices and two sets of distortion coefficients are involved.

Here is how you could approach such a problem:

The steps in order to estimate P₁ and P₂ are as follows:

1. Estimate K₁, K₂ and D₁, D₂ via calibration of the digital camera

2. Use D₁ and D₂ to correct images I₁ and I₂ for lens distortion, then determine the marker positions in the corrected images

3. Compute the fundamental matrix F₁₂ (mapping points in I₁ to epilines in I₂) from the corresponding markers positions and infer the essential matrix E₁₂ = K₂^T * F₁₂ * K₁

4. Infer R₂ and t₂ from E₁₂ and one point correspondence (see this answer to a related question). At this point, you have an affine estimation of the camera poses, but not an up-to-scale one since t₂ has unit norm.

5. Use the measured distance l₀ between two arbitrary markers to infer the correct norm for t₂.

6. For the best accuracy, you may refine P₁ and P₂ using a bundle adjustment, with K₁ and ||t₂|| fixed, based on the corresponding marker positions in I₁ and I₂.

At this point, you have an accurate metric estimation of the camera poses P₁ = K₁ * [ I | 0 ] and P₂ = K₂ * [ R₂ | t₂ ]. Now, the steps to estimate H^o₁₂ are as follows:

7. Use D₁ and D₂ to correct images I₁ and I₂ for lens distortion, then determine the marker positions in the corrected images (same as 2. above, no need to re-do that) and estimate H^m₁₂ from these corresponding positions

8. Compute the 3x1 vector v describing the markers' plane p_m by solving this linear equation: Z * H^m₁₂ = K₂ * ( R₂ - t₂ * v^T ) * K₁^-1 (see HZ00 chapter 13, result 13.5 and equation 13.2 for a reference on that), where Z is a scaling factor. Infer the distance to origin d_m = ||v|| and the normal n = v / ||v||, which describe the markers' plane p_m in 3D.

9. Since the object plane p_o is parallel to p_m, they have the same normal n. Hence, you can infer the distance to origin d_o for p_o from the distance to origin d_m for p_m and from the measured plane offset d_m-o, as follows: d_o = d_m ± d_m-o (the sign depends of the relative position of the planes: positive if p_m is closer to the camera for I₁ than p_o, negative otherwise).

10. From n and d_o describing the object plane in 3D, infer the homography H^o₁₂ = K₂ * ( R₂ - t₂ * n^T / d_o ) * K₁^-1 (see HZ00 chapter 13, equation 13.2)

11. The homography H^o₁₂ maps points on the measured object in I₁ to the corresponding points in I₂, where both I₁ and I₂ are assumed to be corrected for lens distortion. If you need to map points from and to the original distorted image, don't forget to use the distortion coefficients D₁ and D₂ to transform the input and output points of H^o₁₂.

The reference I used:

[HZ00] "Multiple view geometry for computer vision", by R.Hartley and A.Zisserman, 2000.