If you are looking for speed, here is a procedure that might help you.
Sort the triangle vertices on their ordinates. This takes at worst three comparisons. Let Y0, Y1, Y2 be the three sorted values. By drawing three horizontals through them you partition the plane into two half planes and two slabs. Let Y be the ordinate of the query point.
if Y < Y1
if Y <= Y0 -> the point lies in the upper half plane, outside the triangle; you are done
else Y > Y0 -> the point lies in the upper slab
else
if Y >= Y2 -> the point lies in the lower half plane, outside the triangle; you are done
else Y < Y2 -> the point lies in the lower slab
Costs two more comparisons. As you see, quick rejection is achieved for points outside of the "bounding slab".
Optionally, you can supply a test on the abscissas for quick rejection on the left and on the right (X <= X0' or X >= X2'). This will implement a quick bounding box test at the same time, but you'll need to sort on the abscissas too.
Eventually you will need to compute the sign of the given point with respect to the two sides of the triangle that delimit the relevant slab (upper or lower). The test has the form:
((X - Xi) * (Y - Yj) > (X - Xi) * (Y - Yj)) == ((X - Xi) * (Y - Yk) > (X - Xi) * (Y - Yk))
The complete discussion of i, j, k combinations (there are six of them, based on the outcome of the sort) is out of the scope of this answer and "left as an exercise to the reader"; for efficiency, they should be hard-coded.
If you think that this solution is complex, observe that it mainly involves simple comparisons (some of which can be precomputed), plus 6 subtractions and 4 multiplies in case the bounding box test fails. The latter cost is hard to beat as in the worst case you cannot avoid comparing the test point against two sides (no method in other answers has a lower cost, some make it worse, like 15 subtractions and 6 multiplies, sometimes divisions).
UPDATE:
Faster with a shear transform
As explained just above, you can quickly locate the point inside one of the four horizontal bands delimited by the three vertex ordinates, using two comparisons.
You can optionally perform one or two extra X tests to check insideness to the bounding box (dotted lines).
Then consider the "shear" transform given by X'= X - m Y, Y' = Y, where m is the slope DX/DY for the highest edge. This transform will make this side of the triangle vertical. And since you know on what side of the middle horizontal you are, it suffices to test the sign with respect to a single side of the triangle.
Assuming you precomputed the slope m, as well as the X' for the sheared triangle vertices and the coefficients of the equations of the sides as X = m Y + p, you will need in the worst case
- two ordinate comparisons for vertical classification;
- optionally one or two abscissa comparisons for bounding box rejection;
- computation of
X' = X - m Y;
- one or two comparisons with the abscissas of the sheared triangle;
- one sign test
X >< m' Y + p' against the relevant side of the sheared triangle.
