I\'d initially posted this on the OpenCV forums, but unfortunately, I didn\'t get too many views/replies so I\'m posting here with the hopes that someone might have a direction
I found similar problems when using OpenCV (v.2.4.6) 3D reconstruction function. Some people, like Martin Peris, have implemented again by themselves. http://blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html
Sadly, sometimes I have found problems in both implementations with my data. So, I supposed my problem in these cases is due to a poor extrinsic camera parameter description. Perhaps, it is your case too. :-?
PS. In order to get rid of the background you'll need segment it. Or, at least, check depth values bigger than a threshold once the reconstruction works. handleMissingValues flag only eliminates points at "infinity" PS2. Please, tell us if you solve this problem. I think it is valuable for all comunity. Thnx