How to calculate look at point to move the camera with the mouse in OpenGL/GLUT?

Question

This will be confusing for me to explain so please bear with me.

I\'ve already implemented most type of movements and rotations in my camera class, everything is wor

Answer 1

Your problem appears to be in the statement:

UpVector = CrossProduct(&Reference, &RightVector) * (-1);

Once you've rotated Reference toward RightVector in the previous statement, their cross product will no longer result in an UpVector that gives you a horizontal horizon. Try it with your arms. Furthermore, Reference and RightVector are not separated by 90 degrees, so UpVector won't even be a unit vector either. (Finally, you should really just switch the order of the cross product for clarity, rather than multiplying by (-1).)

Honestly, if I were doing it, I would take a different approach. I don't see any logical reason why the two rotations have to be in one function. I also avoid explicit sines and cosines at all costs when working with vectors. I think what you really need is a function to Rotate About an Arbitrary Axis. If nothing else, it's very useful. Fortunately all the details are taken care of by Mr. Murray! If you implement this function, then it becomes very simple. Define a constant SkyVector that always points upward. Then in pseudocode,

AxisRotation( Vector vec, Vector axis, float angle ) {
    Vector result;

    // The axis is assumed to be normalized:  
    //    (just make sure you're not modifying the original)
    axis = NormalizeVector( &axis );

    // expanded for clarity:
    float u = axis.x;
    float v = axis.y;
    float w = axis.z;
    float x = vec.x;
    float y = vec.y;
    float z = vec.z;
    float c = cos(angle);
    float s = sin(angle);

    // Apply the formula verbatim from the linked page:
    result.x = u*(u*x + v*y + w*z)*(1.-c) + x*c + (-w*y + v*z)*s;
    result.y = v*(u*x + v*y + w*z)*(1.-c) + y*c + ( w*x - u*z)*s;
    result.z = w*(u*x + v*y + w*z)*(1.-c) + z*c + (-v*x + u*y)*s;

    return result;
}

Yaw(angleX) {
    Reference = AxisRotation( &Reference, &SkyVector, angleX );
    RightVector = NormalizeVector( CrossProduct( &Reference, &SkyVector ) );
    UpVector = CrossProduct( &RightVector, &Reference );
}

Pitch(angleY) {
    Reference = AxisRotation( &Reference, &RightVector, angleY );
    //RightVector doesn't change!
    UpVector = CrossProduct( &RightVector, &Reference );
}

If you go through that operation by operation, it should hopefully make some sense. Finally, I'll add that quaternions are really the 'correct' way to do this stuff and avoid gimbal lock, but I usually do pretty much exactly what you've done. You might have to check every now and then to make sure your vectors stay nice and perpendicular. Quaternions are more stable.

Edit: If the axis rotation function is overkill, you can still implement this with simple vectors and rotation matrices. The only thing is you'll have to start projecting things into the horizontal plane so that you can do the two rotations independently And it'll still take some sines and cosines. Your time is probably better spent implementing the axis rotation function!