问题

I am trying to run MSBuild on my (mostly C++) projects (imagine a really humongous code base). Visual Studio 2015 is the toolset in question (Windows 7 SP1 and VS 2015 Update 2). Even with /m:1 (and thereby forcing it to use only one processor) I am finding some completely random project constantly hanging at the compile phase. For example, when this issue happens, if I look at the offending project and the files it comprises of, I can see that the .obj files have been created successfully for every translation unit. However the system just never moves on to the link phase. I see two instances of cl.exe sitting idle on Task Manager and doing nothing. Maybe after 30 minutes or so when I kill one of the instances I get something like:

cl : Command line error D8040: error creating or communicating with child 
process [Path_To_The_Project_Where_The_Compiler_Was_Stuck.vcxproj]

After this, surprisingly enough, the compiler just continues on its merry way picking up from where it left off.

Has anyone run into something similar? This has been driving me bat shit for the past couple of weeks!

Updated with answers to IInspectable's queries:

This is going to look a bit hairy. Please, someone with better editing skills, can you help me format this in a better way so people don't gorge their eyes out?

0:002> ~*kv

   0  Id: 842c.86b0 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0024e008 00000000`774b8db8 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`774c3649 : ntdll!ZwWaitForSingleObject+0xa
00000000`0024e010 00000000`774b8cb4 : 00000000`00000001 00000000`0024e1d0 ffffffff`ffffffff 00000000`775a2490 : ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0024e0c0 00000000`774c25d4 : 000007fe`e012e3a0 00000000`0024e2c0 00000000`77350000 00000000`0000003c : ntdll!RtlEnterCriticalSection+0xd1
00000000`0024e0f0 00000000`774c2631 : 000007fe`fd25ace0 00000000`00004f03 ffffffff`ffffff00 00000000`00000000 : ntdll!LdrGetProcedureAddressEx+0x42d
00000000`0024e260 000007fe`fd2133d0 : 000007fe`fd25ace0 000007fe`fd256ee0 00000000`00004f03 00000000`0024e438 : ntdll!LdrGetProcedureAddress+0x11
00000000`0024e2a0 000007fe`e008f77d : 000007fe`e0163c58 000007fe`fd210000 000007fe`e012d890 00000000`0024e438 : KERNELBASE!GetProcAddress+0x5c
00000000`0024e2e0 000007fe`e008f5f1 : 00000000`00000000 ffffffff`ffffffff 00000000`0024e4d0 000007fe`fd1014bd : ucrtbase!try_get_function+0xed
00000000`0024e330 000007fe`e008f2c0 : 00000000`0024e4d0 00000000`00489198 00000000`004891b8 00000000`00000000 : ucrtbase!GetLocaleNameFromDefault+0x4d
00000000`0024e430 000007fe`e00871d8 : 00000000`00000004 00000000`004892c2 00000000`004891b8 00000000`00493ba0 : ucrtbase!__acrt_get_qualified_locale+0x74
00000000`0024e480 000007fe`e008aae3 : 00000000`00000000 00000000`0024e738 00000000`00000000 00000000`0024e820 : ucrtbase!_expandlocale+0x2c8
00000000`0024e6f0 000007fe`e008a5cd : 00000000`00200000 00000000`00240016 00000000`00000007 00000000`00493ba0 : ucrtbase!_wsetlocale_set_cat+0xab
00000000`0024ea60 000007fe`e008752a : 00000020`00000000 00000000`0000000a 003d0041`004c0019 00000000`0024ed50 : ucrtbase!_wsetlocale_nolock+0x10d
00000000`0024ecb0 000007fe`e00888f7 : 00000000`00000000 00000000`00000000 00000000`00000158 00000000`00000000 : ucrtbase!<lambda_30712929f77e709619002f448b6a9510>::operator()+0x3a
00000000`0024ed00 000007fe`e0085e2f : 00000000`0024ee50 00000000`00000000 00000000`77371eb0 00000000`0024ed90 : ucrtbase!__crt_seh_guarded_call<void>::operator()<<lambda_d67e8342c384adda8f857579ab50b2ae>,<lambda_30712929f77e709619002f448b6a9510> & __ptr64,<lambda_4525336fd7e478d965fb7ca7a337cad8> >+0x3b
00000000`0024ed30 000007fe`e0088935 : 00000000`00000004 00000000`00000004 00000000`0024ee38 00000000`0024ee59 : ucrtbase!<lambda_2af78c5f5901b1372d98f9ab3177dfa6>::operator()+0x8b
00000000`0024ed90 000007fe`e0086c18 : 00000000`00000000 00000000`00000000 00000000`00000005 00000000`0024ee38 : ucrtbase!__crt_seh_guarded_call<void>::operator()<<lambda_5df02c53a8f32f81fd64e5bbb78039f1>,<lambda_2af78c5f5901b1372d98f9ab3177dfa6> & __ptr64,<lambda_f51fe5fd7c79a33db34fc9310f277369> & __ptr64>+0x15
00000000`0024edc0 000007fe`e0085e62 : 00000000`00000003 00000000`00000002 00000000`00000005 00000000`00489100 : ucrtbase!call_wsetlocale+0x288
00000000`0024eec0 00000001`3f258409 : 00000000`00000100 00000000`00000000 00000000`00000100 063707fe`00030000 : ucrtbase!setlocale+0x22
00000000`0024ef30 00000001`3f27cd2c : 00000000`00000000 00000001`3f32e270 000007fe`e0163818 00000000`00000000 : link!wmain+0xed
00000000`0024f790 00000000`773659bd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : link!MOD::FHasCorMeta+0x238
00000000`0024f7d0 00000000`7749a2e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0024f800 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

   1  Id: 842c.73bc Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0287f118 00000000`774b8db8 : 00000000`003c0a18 00000000`000000e0 00000000`003c0000 00000000`0000000a : ntdll!ZwWaitForSingleObject+0xa
00000000`0287f120 00000000`774b8cb4 : 00000000`00000000 00000000`00000002 00000000`00000043 000007fe`e01639c0 : ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0287f1d0 000007fe`e008872f : 00000000`0287f2c0 00000000`0287f260 00000000`0287f290 00000000`00000000 : ntdll!RtlEnterCriticalSection+0xd1
00000000`0287f200 000007fe`e0089255 : 000007fe`e012db80 000007fe`e01625c0 00000000`00000000 00000000`0287f260 : ucrtbase!__crt_seh_guarded_call<void>::operator()<<lambda_5e887d1dcbef67a5eb4283622ba103bf>,<lambda_4466841279450cc726390878d4a41900> & __ptr64,<lambda_341c25c0346d94847f1f3c463c57e077> >+0x3f
00000000`0287f240 00000000`7749a52c : 000007fe`e0089060 000007fe`00000005 000007ff`fffd8000 00000000`00000005 : ucrtbase!__acrt_DllMain+0x1f5
00000000`0287f2e0 00000000`7749a1ef : 000007ff`fffd8000 00000000`00000000 000007ff`fffdc000 00000000`00000000 : ntdll!LdrpInitializeThread+0x17c
00000000`0287f3e0 00000000`7749a0ee : 00000000`0287f4a0 00000000`00000000 000007ff`fffd8000 00000000`00000000 : ntdll!LdrpInitialize+0x9f
00000000`0287f450 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

#  2  Id: 842c.71d8 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0275f9e8 00000000`77562c88 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgBreakPoint
00000000`0275f9f0 00000000`773659bd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgUiRemoteBreakin+0x38
00000000`0275fa20 00000000`7749a2e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0275fa50 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

Here is the output of locks extension:

0:002> !locks

CritSec ntdll!LdrpLoaderLock+0 at 00000000775a2490
WaiterWoken        No
LockCount          1
RecursionCount     1
OwningThread       73bc
EntryCount         0
ContentionCount    2
*** Locked

CritSec ucrtbase!wenviron_table+c0 at 000007fee01639c0
WaiterWoken        No
LockCount          1
RecursionCount     1
OwningThread       86b0
EntryCount         0
ContentionCount    1
*** Locked

Scanned 417 critical sections

Its obvious threads 86b0 and 73bc are waiting on each other but I can't figure out how they got into this state!