Concave and Convex Polygon - Turn arbitrary polygon into a convex polygon

Question

How can I identify and remove those four RED points drawn in image

Those four points

Answer 1

You can detect concave points by looking at the interior angle — if it's greater than a half circle then the point is concave. Indeed they're usually called reflex points because the interior angle is reflex.

A quick way to check is the dot product. For example, look at the three points P14, P15, P16. P16 is behind the line that the segment between P14 to P15 lies on (ie, the dot product of the vector from P15 to P16 with the normal to that line is negative), so P15 is a convex point.

P18 is in front of the line that the segment from P16 to P16 lies on (ie, the dot product of the vector from P17 to P18 with the normal of that line is positive), so P17 is a reflex point.

In 2d, the normal of the line is as simple as flipping the x and y coordinates and negating one.

However, what I think you might want is the convex hull — consider whether there might be convex points that nonetheless create a concave hull. The most obvious example is if I left P17 where it is but moved P18 and P16 even further into the shape, behind it. If so then you want to check out convex hull algorithms.