Cannot move view matrix and direction view matrix is pointing in simultaneously

Question

I started learning how to use the mouse to look around in OpenGL using a perspective projection. When I move around the scene (without moving my \'head\', which is where my