My problem: I have 'Run Tests After Build' Off. Although the test cases are not running, the test discovery itself is running for minutes after every build. Below setting fixed my problem.
Under Test->Test Settings-> uncheck 'Keep Test Execution Engine Running'. I am using VS 2015.