(sed) Remove comma between any two numbers inside double quote (but keep the number)

Question

I need to sanitize a CSV that lists some numbers. For 4 and more digits, it gives the number in double quotes with comma. I need to remove this type of comma.

So far