thats my first post, so please be kind.
I have a matrix with 3~10 coordinates and I want to connect these points to become a polygone with maximum size.
I tried
You said you only have 3...10 points to connect. In this case, I suggest you just take all possible combinations, compute the areas with polyarea and take the biggest one.
Only if your number of points increases or if you have to compute it frequently so that compuation time matters, it's worth investing some time in a better algorithm. However I think it's difficult to come up with an algorithm and prove its completeness.