I used to have the following configuration with xDebug 2:
xdebug.default_enable=1
xDebug did not slow down execution when no debug client was list