Get heading and pitch from pixels on Street View

Question

I think this is the best place for this question.

I am trying to get the heading and pitch of any clicked point on an embedded Google Street View.

The only piec

Answer 1

Here is an attempt to give a mathematical derivation of the answer to your question.

Note: Unfortunately, this derivation only works in 1D and the conversion from a pair of angular deviations to heading and pitch is wrong.

Notations:

f: focal length of the camera
h: height in pixels of the viewport
w: width in pixels of the viewport
dy: vertical deviation in pixels from the center of the viewport
dx: horizontal deviation in pixels from the center of the viewport
fov_y: vertical field of view
fov_x: horizontal field of view
dtheta_y: relative vertical angle from the center of the viewport
dtheta_x: relative horizontal angle from the center of the viewport

Given dy, the vertical offset of the pixel from the center of the viewport (this pixel corresponds to the green ray on the figure), we are trying to find dtheta_y (the red angle), the relative vertical angle from the center of the viewport (the pitch of the center of the viewport is known to be theta_y0).

From the figure, we have:

tan( fov_y / 2 ) = ( h / 2 ) / f

tan( dtheta_y ) = dy / f

so:

tan( dtheta_y ) = dy / ( ( h / 2 ) / tan( fov_y / 2 ) )
                = 2 * dy * tan( fov_y /  2 ) / h

and finally:

dtheta_y = atan( 2 * dy * tan( fov_y / 2 ) / h )

This is the relative pitch angle for the pixel at dy from the center of the viewport, simply add to it the pitch angle at the center of the viewport to get the absolute pitch angle (i.e. theta_y = theta_y0 + dtheta_y).

similarly:

dtheta_x = atan( 2 * dx * tan( fov_x / 2 ) / w )

This is the relative heading angle for the pixel at dx from the center of the viewport.

Complements:

• Both relations can be inverted to get the mapping from relative heading / pitch angle to relative pixel coordinates, for instance:

  dy = h tan( dtheta_y ) / ( 2 * tan( fov_y / 2 ) )
  

• The vertical and horizontal fields of view fov_y and fov_x are linked by the relation:

  w / h = tan( fov_x / 2 ) / tan( fov_y / 2 )
  

  so:

  fov_x = 2 * atan( w * tan( fov_y / 2 ) / h )
  

• The vertical and horizontal deviations from the viewport center dy and dx can be mapped to absolute pixel coordinates:

  x = w / 2 + dx
  y = h / 2 - dy
  

• Proof of concept fiddle