For some reason raycaster.direction is inversed. I implemented many times raycasting and never got this problem. The ray goes from camera to infinity but not towards the clicked