I have a set of simple (no holes, no self-intersections) polygons, and I need to check that they don\'t intersect each other (one can be entirely contained in another; that is o
To test for intersections you could use my freeware clipper library: http://sourceforge.net/projects/polyclipping/ .
To test for containment, first exclude intersections as above. Then use Clipper again adding all the polygons - Clipper.AddPolygon(). Then perform a Union (boolean OR) operation on the polygons - Clipper.Execute(ctUnion, solution). If the Clipper.ForceAlternateOrientation property is true, Clipper will return in the solution outer polygons in clockwise orientation and contained polygons (holes) in counter-clockwise orientation. It should then be a simple matter of testing polygon orientations and applying a PointInPolygon from one vertex in a counter-clockwise polygon against the other clockwise polygons.
Determining whether none of the polygons intersect can be done in O(n*log(n))
by applying the Shamos-Hoey algorithm. Depending on what the Shamos-Hoey algorithm returns, a polygon Pi contains polygon Pj if any vertex from Pj is inside Pi which is done in O(n)
for two polygons.
For code, see gpc.
First, your algorithm for testing containment does not test for intersection correctly. Imagine two rectangles like this:
+--+
+--+--+--+
| | | |
+--+--+--+
+--+
Vertices would be at (1, 2) (1,3) (4,2) (4,3) and (2,1) (3,1) (2,4) (3,4) -- no vertex lies inside any polygon, but the polygons do in fact intersect.
To test for this kind of intersection, it is necessary to determine whether any of the polygons' edges intersect. For your purposes, if edges intersect but one polygon is not contained within the other, then you know they overlap in a non-permitted fashion.
As for determining the containment tree, one way to do that is to sort the polygons from smallest to largest by area. Provided the polygons don't overlap-without-containment, then any polygon's parent in the tree will be the first containing polygon coming after it in the list.
Edit: Oh, also, I advise writing a fast bounding-box or bounding-circle overlap routine, and using that to avoid having to do all the line intersection and vertex containment tests. If that still isn't fast enough, you probably want to build a quad or BSP tree; that will complicate things quite a bit but will also eliminate a lot of the intersection checks entirely.