Disallow adding|removing|updating item in relational field based on user permission

Question

This is a follow up to my previous question Security voter on relational entity field when not using custom subresource path

So the situation is, image you having ManyToM