I would like to restrict my Eclipse-RCP application to a single instance. By this, I mean that once a user opens the application for the first time, it listens on a port and
I have an app that does this same thing. The trick is that the new instance can't bring the old instance to the front. But, the old instance can bring itself to the front after it contacts the new instance.
So your old instance needs to call
PlatformUI.getWorkbench().getActiveWorkbenchWindow().getShell().forceActive();
after it notifies the new instance. For my app, the new instance doesn't show an error message, it just closes transparently and the old instance pops itself back up.