Concave and Convex Polygon - Turn arbitrary polygon into a convex polygon

Question

How can I identify and remove those four RED points drawn in image

Those four points

Answer 1

aioobe has it right—you sound as though you are looking to compute the convex hull of your polygon, in which case you want one of the convex hull algorithms like Graham scan or Chan's algorithm.

However, if you just want to know whether an angle is convex or concave there's a quick way to compute this that avoids trigonometry.

If A, B, and C are consecutive vertices going clockwise around the polygon, then the vertex at B is convex if

(B − A)^⟂ · (C − B) < 0

Here V^⟂ is a vector that's V rotated by 90° anti-clockwise, which can be computed like so: (x, y)^⟂ = (−y, x).