I have the following declaration of my service:
It should work without the WiX util extension. Here is my complete test installer. Create your own test project and copy and past my installer into your .wxs file. Then replace the File, ServiceInstall and ServiceControl paths and names with your own service. If you still get the same error message, could it be that you don't actually have the privileges to do the install on your machine?