I tried the algorithms in Line of intersection between two planes
but all not return correct results
I have two rectangle in 3D each defined by three points ,
The link you provided most probably has the correct solution :) Did you correctly transform your three points info the Ax+By+Cz+D = 0 form? Check if all those points satisfy this formula. If you have the correct {A, B, C, D} then it's easy to calculate the rest as described in the link..
Here is a link which explains how to get this formula using 3 points.
Ok, here a simple summary:
Given three points in space (x1,y1,z1), (x2,y2,z2), (x3,y3,z3), calculate this:
A = y1 (z2 - z3) + y2 (z3 - z1) + y3 (z1 - z2)
B = z1 (x2 - x3) + z2 (x3 - x1) + z3 (x1 - x2)
C = x1 (y2 - y3) + x2 (y3 - y1) + x3 (y1 - y2)
D = -(x1 (y2 z3 - y3 z2) + x2 (y3 z1 - y1 z3) + x3 (y1 z2 - y2 z1))
for both planes. Which means you have A1, B1, C1, D1 and A2, B2, C2, D2.
Using A, B, C, D calculate this:
x1 = 0
z1 = (B2/B1)*D1 - D2)/(C2 - C1*B2/B1)
y1 = (-C1 * z1 - D1) / B1
Then this:
x2 = some value..
z2 = (B2/B1)*(A1 * x2 + D1) - A2 * x2 - D2)/(C2 - C1*B2/B1)
y2 = (-C1 * z2 -A1 * x2 - D1) / B1
Basically just combine both ways described in those two links..