This question raises several issues. The bounty will go to an answer which addresses them holistically.
Here\'s a problem I\'ve been playing with.
KD Trees allow you to efficiently search within a fixed radius of a point. If all of a point's neighbour's are within d1 units and maximum displacement of any point from the previous measurement is d2, then you only need to search within d1+d2 units of the point.
If you were dealing with a Minkowski distance, then you could have used David Mount's ANN library. I am not sure, if there is a library that would take arbitrary distance functions, though.