I used the following code based on this
ballA.vx = (u1x * (m1 - m2) + 2 * m2 * u2x) / (m1 + m2);
ballA.vy = (u1y * (m1 - m2) + 2 * m2 * u2y) / (m1 + m2);
ballB.
I strongly recommend you familiarize yourself with the center of momentum frame. It makes collisions much easier to understand. (And without that understanding you're just manipulating cryptic equations and you'll never know why things go wrong.)
Anyway, to determine the angle, you can use the impact parameter, basically how far "off center" one ball hits the other. The two balls are approaching each other in opposite directions (in the center-of-momentum frame), and the distance between their centers perpendicular to those velocities is the impact parameter h
. Then the angle of deflection is 2 acos(h/(r1+r2)).
Once you get that working perfectly, you can worry about inelastic collisions and the coefficient of restitution.