If I change the access modifier of the setter of a public property from private to public, is that causes any breaking change in the other assemblies that refer it?
It might be, if those other assemblies use reflection to test for the presence of the setter.
But early-bound code will not break.
Another thing to consider is whether this is a change in semantics. If the property was formerly set only during construction, and now can be modified at any time, that definitely could break consumers who cached the value, used it as a Dictionary key, etc.