OpenGL camera rotation using gluLookAt

Question

I am trying to use gluLookAt to implement an FPS style camera in OpenGL fixed function pipeline. The mouse should rotate the camera in any given direction.

I store the p

Answer 1

Start with a set of vectors:

fwd = (0, 0, -1);
rht = (1, 0, 0);
up = (0, 1, 0);

Given that Your x and y, taken from the mouse positions You mentioned, are small enough You can take them directly as yaw and pitch rotations respectively. With yaw value rotate the rht and fwd vectors over the up vector, than rotate fwd vactor over the rht with pitch value. This way You'll have a new forward direction for Your camera (fwd vactor) from which You can derive a new look-at point (L = P + fwd in Your case).

You have to remember to restrict pitch rotation not to have fwd and up vectors parallel at some point. You can prevent that by recreating the up vector every time You do pitch rotation - simply do a cross product between rht and fwd vactors. A side-note here though - this way up will not always be (0,1,0).