I agree with Andreas Brinck, barycentric coordinates are very convenient for this task. Note that there is no need to solve an equation system every time: just evaluate the analytical solution. Using Andreas' notation, the solution is:
s = 1/(2*Area)*(p0y*p2x - p0x*p2y + (p2y - p0y)*px + (p0x - p2x)*py);
t = 1/(2*Area)*(p0x*p1y - p0y*p1x + (p0y - p1y)*px + (p1x - p0x)*py);
where Area
is the (signed) area of the triangle:
Area = 0.5 *(-p1y*p2x + p0y*(-p1x + p2x) + p0x*(p1y - p2y) + p1x*p2y);
Just evaluate s
, t
and 1-s-t
. The point p
is inside the triangle if and only if they are all positive.
EDIT: Note that the above expression for the area assumes that the triangle node numbering is counter-clockwise. If the numbering is clockwise, this expression will return a negative area (but with correct magnitude). The test itself (s>0 && t>0 && 1-s-t>0
) doesn't depend on the direction of the numbering, however, since the expressions above that are multiplied by 1/(2*Area)
also change sign if the triangle node orientation changes.
EDIT 2: For an even better computational efficiency, see coproc's comment below (which makes the point that if the orientation of the triangle nodes (clockwise or counter-clockwise) is known beforehand, the division by 2*Area
in the expressions for s
and t
can be avoided). See also Perro Azul's jsfiddle-code in the comments under Andreas Brinck's answer.