On top of @CrazyCoder's solution, in version 2016.3 I came across this issue where I just saw 2 icons like this,
You should hit the 2nd one!
IF you hit the 2nd (like me) next thing I know, it opened in a browser tab. It is a configuration to open in an embeded web server.
So I wanted to delete this configuration and start a new one. So, I found the entry to through this drop down in the top right corner,
While picking Edit Configuration, I could change the test configuration.
Hope that helps someone like me, although clearly not the initial poster.
Cheers