This question has one major question, and one minor question. I believe I am right in either question from my research, but not both.
For my physics loop, the first thin
You should use if(fabs(dist) < 0.0001f) { /* collided */ }
This is to acocunt for floating point accuracies. You most certainly would not get an exact 0.0f at most angles or contact.
the value of dist
if negative, is in fact the actual amount you need to shift the body back onto the surface of the plane in case it goes through the plane surface. sphere.position = sphere.position - plane.Normal * fabs(dist);
Once you have moved it back to the surface, you can optionally make it bounce in the opposite direction about the plane normal; or just stay on the plane.
parallel_vec = Vec3.dot(plane.normal, -sphere.velocity);
perpendicular_vec = sphere.velocity - parallel_vec;
bounce_velocity = parallel - perpendicular_vec;
you cannot blindly do totalforce = external_force + velocity
unless everything has unit mass.
EDIT:
Vector3 planeToSphere = sphere.point - plane.point;
float dist = Vector3.dot(plane.normal, planeToSphere) - plane.radius;
if(dist < 0)
{
// collided.
}
I suggest you study more Maths first if this is the part you do not know.
NB: Sorry, the formatting is messed up... I cannot mark it as code block.
EDIT 2: Based on my understanding on your code, either you are naming your variables badly or as I mentioned earlier, you need to revise your maths and physics theory. This line does not do anything useful.
float denom = Vec3Dot(&plane->GetNormal(), &forces);
A at any instance of time, a force on the sphere can be in any direction at all unrelated to the direction of travel. so denom essentially calculates the amount of force in the direction of the plane surface, but tells you nothing about whether the ball will hit the plane. e.g. gravity is downwards, but a ball can have upward velocity and hit a plane above. With that, you need to Vec3Dot(plane.normal, velocity)
instead.
Alternatively, Mark Phariss and Gerhard Powell had already give you the physics equation for linear kinematics, you can use those to directly calculate future positions, velocity and time of impact.
e.g. s = 0.5 * (u + v) * t;
gives the displacement after future time t. compare that displacement with distance from plane and you get whether the sphere will hit the plane. So again, I suggest you read up on http://en.wikipedia.org/wiki/Linear_motion and the easy stuff first then http://en.wikipedia.org/wiki/Kinematics .
Yet another method, if you expect or assume no other forces to act on the sphere, then you do a ray / plane collision test to find the time t at which it will hit the plane, in that case, read http://en.wikipedia.org/wiki/Line-plane_intersection .