Can WinDBG be made to find mscordacwks.dll in the symbol store?

Question

The Question

There are plenty of manual ways to make WinDBG find mscordacwks.dll without a symbol store (putting the file in the path somewhere, putting it in the same folder as windbg.exe, putting it in my Framework\v folder, specifying the path in WinDBG using .cordll -lp c:\dacFolder, etc.), but they all only fix it for me. I need to fix it more generally for everyone who uses my symbol store.

The possible solutions I can imagine are:

• WinDBG be made to check the symbol store using mscordacwks.dll's subfolder name instead of mscorwks.dll's folder name.
• SymStore.exe be made to add mscordacwks.dll under mscorwks.dll's subfolder name so WinDBG finds it when it looks there.

Q: Are either of these things possible, or is there another way that I'm not thinking of to solve the problem?

The Background

When analyzing a .NET process, I encountered the (apparently common) problem that psscor2 (and sosex) could not find the appropriate mscordacwks.dll on my machine. The error in WinDBG is:

Failed to load data access DLL, 0x80004005 
Verify that 1) you have a recent build of the debugger (6.2.14 or newer) 
            2) the file mscordacwks.dll that matches your version of mscorwks.dll is 
                in the version directory 
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path. 
            4) you are debugging on the same architecture as the dump file. 
                For example, an IA64 dump file must be debugged on an IA64 
                machine.

You can also run the debugger command .cordll to control the debugger's 
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload. 
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable 
path is pointing to mscorwks.dll as well.

There are plenty of SO questions on this and plenty of good answers, practically all of which ultimately reference Doug Stewart's outstanding blog post, What is mscordacwks.dll?.

Thanks to that, I got my situation all straightened out by obtaining the correct mscordacwks.dll and placing it here:

"C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"

where I knew WinDBG would look because I had previously tried it with !sym noisy.

So I'm all set now, but I had to put it in that path physically rather than adding it to my symbol server through the normal symstore.exe mechanism. Since my symbol store is used by more than just me, I need to do it the right way for everyone else using the store.

And that's the problem. When I add using symstore.exe instead of going into the above path, it goes into:

"C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545CB1bd000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"

The only difference being that the subfolder name is 4E1545CB1bd000 here instead of the 4E1545829a3000 that WinDBG is looking for.

The reason for this is that when adding a binary to the symbol store, symstore.exe uses the PE of the binary to get the image timestamp and the image size. In the case of this particular .dll, dumpbin.exe /headers mscordacwks.dll reveals these to be:

• image timestamp: 0x4E1545CB (Thu Jul 07 01:36:11 2011)
• image size: 0x1BD000

Hence, the subfolder name 4E1545CB1BD000.

What WinDBG is looking for, on the other hand, is a subfolder based on the image timestamp and image size of mscorwks.dll, not mscordacwks.dll, because the former is loaded into the process, not the latter. WinDBG can't know the timestamp and size of the DAC module because that module is not in the process dump.

As further verification of this explanation, dumpbin.exe /headers mscorwks.dll reveals:

• image timestamp: 0x4E154582 (Thu Jul 07 01:34:58 2011)
• image size: 0x9A3000

which you can see add up to subfolder name 4E1545829A3000.

Knowing this, now it makes a lot more sense why all these many versions of the mscordacwks.dll that people keep running into seem to be missing from Microsoft's symbol servers. I'm sure they're there, it's just that WinDBG and psscor2 can't find them because they're picking the wrong subfolder name. Why it even bothers searching the symbols path is beyond me since it's guaranteed never to find it!

So that's my challenge. Can I somehow force symstore.exe to add mscordacwks.dll using the PE info of mscorwks.dll? If not, am I missing something about WinDBG and psscor2, might there be a way for them to know the correct timestamp and size of mscordacwks.dll even though it's not loaded (and a way for them to actually use those instead of mscorwks.dll)?

Answer 1

Since no other solution has appeared and my workaround seems to handle everything nicely, I'm just going to keep on with that, and I would recommend anyone else who never wants to see the annoying Failed to load data access DLL, 0x80004005 error again do the same.

So to make this work for you (and everyone who uses your symbol store, so I really wish Microsoft would do this to save us all a lot of trouble) simply place the compressed DAC file (mscordacwks.dl_) by hand into the correct path within your local SYM store.

Here are the steps I follow to accomplish this:

1. In WinDBG do a !sym noisy
2. In WinDBG do a .cordll -ve -u -l
3. In WinDBG do another !CLRStack or other psscor2 command if necessary to force it to load symbols again
4. The symbol search logging will reveal the dll it’s looking for and where it’s looking in your symbol store by showing lines like this: C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll which indicates two things:
   • you need the 64-bit mscordacwks.dll of version 2.0.50727.4216; see https://stackoverflow.com/a/12024171/1910619 for ways to get it
   • it needs to go in a subfolder called 4E1545829a3000 under a folder called mscordacwks_AMD64_AMD64_2.0.50727.4216.dll in your symbol store
5. Once you obtain the file, rename it according to the name WinDBG is looking for, e.g. "mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"
6. Manually compress this file using makecab.exe like this: makecab /D CompressionType=LZX /D CompressionMemory=21 mscordacwks_AMD64_AMD64_2.0.50727.4216.dll mscordacwks_AMD64_AMD64_2.0.50727.4216.dl_
7. Copy that compressed file to the expected place in the symbol store. (That you found in step 4 above, so C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dl_ in our running example here.)

Answer 2

You have to put the CLR dll and the associated mscordacwks dll in the same folder and register the CLR dll with symstore. In that case symstore will add the clr and the mscordacwks dll on the symbol store.

More importantly, it will use the timestamp and file size of the clr dll to create the mscordacwks subfolder, so windbg can find the mscordacwks dll when debugging a dump.

The mscordacwks name must follow the mscordacwks_ARCH_ARCH_fileversion pattern, otherwise symstore won't add it to the symbol store.

I didn't find any documentation on that feature so it may be removed in the future.

Here is the command and the symstore output:

symstore.exe add /o /f 4.6.1076.00\clr.dll /t clr.dll /s \\mystore\microsoft
SYMSTORE MESSAGE: 0 alternate indexers registered
SYMSTORE MESSAGE: LastId.txt reported id 0
SYMSTORE MESSAGE: History.txt reported id 58228
SYMSTORE MESSAGE: Final id is 0000058228
SYMSTORE MESSAGE: Copying C:\Users\build.robot\symstore\4.6.1076.00\clr.dll to \\mystore\microsoft\clr.dll\56D79ED4990000\clr.dll [Force: T, Compress: F]
SYMSTORE MESSAGE: Copying 4.6.1076.00\mscordacwks_AMD64_AMD64_4.6.1076.00.dll to \\mystore\microsoft\mscordacwks_AMD64_AMD64_4.6.1076.00.dll\56D79ED4990000\mscordacwks_AMD64_AMD64_4.6.1076.00.dll [Force: T, Compress: F]

SYMSTORE: Number of files stored = 2
SYMSTORE: Number of errors = 0
SYMSTORE: Number of files ignored = 0