I have a problem where I have a series of points for a simple, non-convex polygon (I hope I have the terminology correct). But the points are not necessarily in order (ie,
Perhaps you mean that the polygon is convex (i.e. it's non-concave). Otherwise I don't think what you're doing is actually possible (there could be multiple ways to "join the dots" and form a polygon).
One technique I can think of in that case: First, the first two items in the list must form an edge, by the rules you give. Then try adding vertices as they appear in list order; in each case, if the result could only be concave, remove the previous element from the result list and ignore it for now. Once you have gone through the source list, if you still vertices that you skipped over, continue for those skipped vertices.
Edit: Ok, just looked at your example from wikipedia, and you do mean non-convex. I don't think it's possible unfortunately.
A simple example showing that the problem is not uniquely soluble is the points a=(0,0), b=(2,0), c=(1,1), d=(1,2); each of the orders (a,b,c,d,a), (a,b,d,c,a) (a,c,b,d,a) are simple polygons.
I'm thinking of an O(n^2)
algorithm:
Since you have the points in order that they are drawn (hopefully), you know the endpoints of each edge.
Choose a point to start on, then continue until you intersect another edge.
Then you mark that spot, continue on the new edge until you reach yet another edge or an endpoint. Whenever you reach a point where you change edges, list it. Once you reach the start point, you are done.
Well, you're not going to like it, but it's not possible. If you think about removing all the lines in your demonstration polygon, you could connect them in a different way and still have a valid non-convex polygon, so how's a sort supposed to know which way to go? That's what sucks about non-convex polygons.