SELinux has persmission setting for more than just regular files, but also device and special files.
http://seedit.sourceforge.net/doc/access_vectors/access_vectors.html#SECTION00044000000000000000 is what you're looking for. Give read/write/etc permissions to the "sem" object.