As you may be able to tell from this screenshot, I am trying to make a physics engine for a platformer I am working on, but I have run into a definite problem: I need to be able
The cross product is the right answer. Dont forget to normalise the result, and dont forget that if the triangle has zero area, the result is invalid because there is no well defined normal. Basically, if your three vertices are p0, p1 and p2:
vector temp = cross(p1 - p0, p2 - p0);
if (length(temp) < epsilon) then
Degenerate_triangle_error;
else
return normalize(temp);
Also, as the other answer says, whether you get the 'up facing' or 'down facing' normal will depend on the ordering of your vertices.