Concave and Convex Polygon - Turn arbitrary polygon into a convex polygon

Question

How can I identify and remove those four RED points drawn in image

Those four points

Answer 1

I am guessing you have the coordinates for the points (P1, P2, etc.). You can get the angle generated between each three points and remove it if it is less than 180. For figuring out the angle, check How to calculate an angle from three points?