Is there a way for a UNIX domain socket listener to only accept connection from certain user (chmod/chown does not work for abstract socket afaik), or
chmod
chown
Yes -- this operation, along with FD passing, is supported via an ancilliary message with the SCM_CREDENTIALS type. The calls involved are documented in man 7 unix.
SCM_CREDENTIALS