OK, I know this sounds like it should be asked on math.stackoverflow.com, but this is embarrassingly simple maths that I\'ve forgotten from high-school, rather than advanced pos
for oblique triangles: c^2 = a^2 + b^2 - 2ab * cos(C)
where a, b, c are the lengths of the sides (regardless of length) and A, B, C are the angles opposite the side with the same letter.
Use the above to figure out the angle from one of the endpoints you know, then use the angle, the position of the vertex, and the angle between the adjacent sides to determine where the unknown vertex is.
And the complexity of the problem doesn't determine which site it should go on, only the subject matter. So you should move this to math.
EDIT: I had a serious brainfart previously, but this should work. Use the law of cosines
/* use the law of cosines to get the angle of CAB */
c² = a² + b² - 2ab cos(Cangle)
cos(Cangle) = (a²+b²-c²) / 2ab
Cangle = acos((a²+b²-c²) / 2ab)
AB = B.xy - A.xy;
normalize(AB);
len = length(AC)
C.x = len*AB.x* cos(Cangle) * len*AB.y*sin(Cangle);
C.y = len*AB.x*-sin(Cangle) * len*AB.y*cos(Cangle);