Given an undirected graph, what would be an algorithm to find all polygons within such graph? Here is an example graph with polygons in colour.
Re AndreyT's suggestion to use a DCEL: the salient feature of the doubly-connected edge list representation is that, for each undirected edge, there are two list nodes, one for each direction. We refer to these nodes as darts and think of them as having a head and a tail. Given a dart (e.g., H->G, with tail H and head G), we can find the reverse dart (e.g., G->H) and the next dart with the same head in counterclockwise order (e.g., J->G). The DCEL can be constructed straightforwardly given a primitive that can be used to sort by angle (the easiest way is to sort by atan2(); the best is to find a determinant test that yields consistent results in the face of floating-point malfeasance).
The polygons (usually called faces) can be found by finding the permutation cycles of the permutation that maps each dart to the reverse of the next dart with the same head in counterclockwise order. For example, if we start with the dart C->D, then we follow the cycle
C->D (E->D is next) D->E (G->E is next) E->G (C->G is next) G->C (D->C is next) C->D
and recover the face C-D-E-G-C. Starting with A->B, we get
A->B B->C C->I I->H H->G G->J J->K K->L L->M M->L L->K K->A A->B,
which is the face A-B-C-I-H-G-J-K-L-M-L-K-A.
This method sort of requires a connected graph. (It will work on a disconnected graph, but it might not give the results that you want.) It also yields the infinite face, which you have indicated is undesirable. To find a dart on the infinite face (which can be used to identify it), find the vertex with the least y-coordinate, breaking ties by least x-coordinate. Then find the last dart with that head in counterclockwise order from the ray shooting straight rightward.
step | description
1a | while vertices with deg(v) = 0 exist
1b | mark vertices with deg(v) = 0 as leaf
|
2 | run algorithm on all vertices which are not marked as leaf
|
3a | for each vertex marked as leaf
3b | if vertex is inside a polygon
3c | check its edges // you have to decide what to do in which case
3d | adjust polygon
I will illustrate this with your example:
step | result
1a | find F and M
1b | mark F and M as leaf
1a | find L
1b | mark L as leaf
1a | find nothing: go to step 2
|
2 | finds polygons: AKJGHICB (1), CIHG (2), and CGED (3)
|
3a | we have F, M, and L
3b | check F:
| poly (1): cast ray: even result -> outside
| poly (2): cast ray: even result -> outside
| poly (3): cast ray: even result -> outside
| since F is always outside: no further action needed, unmark F
3b* | check M:
| poly (1): cast ray: odd result -> inside
| since M is inside a polygon: check how to add it
3c | check edge M-L:
| check if L is part of poly (1)
| if yes: add path to poly (1) (step 3d)
| if no: check if L is inside poly (1)
| -> no check L: odd result -> inside
| if inside: follow path, i.e. step 3c with edge L-K
| if outside: undefined behaviour
| -> inside
3c | check edge L-K:
| check if K is part of poly (1)
| -> yes: add path to poly
3d | Take poly (1) AKJGHICB
| replace K with KLK
| unmark K // note that K was never marked)
| remove K from path
| replace L with LML
| unmark L
| remove L from path
| unmark M // note that you should check if there are more
| // vertices to come for the replacement
| remove M from path
| poly (1) is now AKLMLKJGHICB
3a | we have no marked vertices left
| finish
* note that in step 3b we could first have found L/checked L. Then it would be like this:
3b | check L:
| poly (1): cast ray: odd result -> inside
| since L is inside a polygon: check how to add it
3c | check L-K (or M-L, that would work as above and eventually try L-K)
| check if K is part of poly (1)
| if yes: add path to poly (1)
| -> yes
3d | Take poly (1) AKJGHICB
| replace K with KLK
| unmark K
| remove K from path
| unmark L
| remove L from path
| poly (1) is now AKLKJGHICB
3a | we have M left // from now on a bit less detailed because it's the same again
3b | check M:
| poly (1): cast ray: odd result -> inside
| ...
3c | check M-L
| L is part of poly (1)
3d | replace L in the poly with LML and unmark L and M
| finish
This should be a rough idea of how an algorithm with the one you are already familiar with should work. However, it's probably open for many improvements.