I am using VS 2010 on a win 7 64 bit system with 8 GB of memory. My application is 32 bit. While in the VS 2010 .Net IDE, the app shows up in the Windows task manager as \"M
the IDE fails with an Out of memory exception
No, your program fails with that exception, not the IDE. You'd have to run editbin.exe in a post build event to set the flag:
set path=%path%;$(devenvdir);$(devenvdir)\..\..\vc\bin
editbin /largeaddressaware $(targetpath)
That will not work on the vshost.exe version, you'll have to turn off the hosting process. Project + Properties, Debug tab.
Some options I can think of:
Disable the VSHost process - Do you actually require the extra debugging features from the VS hosting process? If not just to untick the "Enable Visual Studio Hosting Process" option.
Externalise the problematic DLL - Wrap the COM DLL in a simple 32-bit process and compile the rest as 64-bit using an appropriate variety of IPC to connect the two.
Force the flag in the VSHost process - Use a post-build command to forcibly set the flag in the .vshost.exe (no idea if that would work though!)