I followed the instructions in this link: http://msdn.microsoft.com/en-us/library/bt727f1t.aspx to install the remote debugger (2012) on my server where the application is runni
I recently had someone else report this and debugged the issue on their machine. The "Invalid access to memory location" errors are due to an issue in Windows, it can be addressed with this hotfix.
I have had this problem in VS 2012, 2013, 2015 and 2017. Based on other answers it is likely that the problem is related to running a 32 bit version of Visual Studio on a 64 bit PC. Sometimes, as others have recommended, restarting Visual Studio fixes the problem but the best solution I've found so far is to start Visual Studio without a solution, open Debug -> Attach to Process, change the Connection Target to the remove server and wait for the process list to load. Then Cancel, do not attach yet. Load your desired solution and then come back to Attach to Process and the remote process list will still be loaded. Connect to your desired process and everything should work properly from then on.
It turns out the one thing I missed was to tell Visual Studio where to find the .pdb symbols relating to the remote process. To do this go to Tools -> Options -> Debugging then in the Symbol (.pdb) locations add the remote location to the pdb files.
To clarify, I was attaching fine but could not break into code. Now I can. Be aware though that there are other hurdles before you get to my stage where I was attaching to the process successfully but could not catch a breakpoint.
I've found the only way to correct this is by restarting Visual Studio.
Worked for me. I found it at this blog post about invalid access and remote debugging.