This question raises several issues. The bounty will go to an answer which addresses them holistically.
Here\'s a problem I\'ve been playing with.
Have you tried using a quad tree?
That is, recursively partition the "world" into a graph with four subnodes each. The tree can then quickly check which objects are inside a particular square of the world and discard the rest. A very effective culling technique often used for improving performance of collision detection in games.
If this is 3D, extend it to an octree.