Let\'s take this map, where \'#\' illustrates a taken square and \'.\' illustrates a free square:
1 . # # # . . 2 . # . . # . 3 # . . . . # 4 . # # # . . 5 . . . . .
You could attack this by processing each '.' node.
Definition: A '.' node is enclosed if there does not exist a path from the node to the boundary of the map.
If you agree with the above definition, the algorithm would be to maintain a graph of '.' nodes, where adjacent nodes are connected.
Every time a node is changed to '#', remove it from this graph, and check each remaining '.' node to see if a path exists from it to one of the nodes on the map border.
Depending on the size of your map, you made need to attempt various optimizations to limit the number of path searches performed each turn.