In some my project I notice that during executing unit tests under VSTS2008 its VSTestHost\'s memory consuming grows. As I have very many tests in my solution it leads to Ou
I don't think the unit test engine creates a new AppDomain for each test. Since creating an AppDomain is a relatively expensive operation, doing so for each test would slow down execution of unit tests considerably!
Visual Studio 2008 uses a seperate executable called vstesthost.exe to run unit tests. VS communicates with vstesthost.exe (how it does this I don't know) to tell it what tests to run. vstesthost.exe returns the execution results to VS which displays those results.
If you are getting OutOfMemoryExceptions when running your unit tests I would say that's a strong indicator that your code under test is actually not cleaning things up. Are you sure that you aren't retaining handles to unmanaged objects/memory? I would recommend running your unit tests under a Performance Analysis (you can do that by finding the unit test under the "Test View", right-clicking on it, and selecting "Create Performance Session"). This might shed some light at least on your object allocations.