Using MS crypto library on server 2012 - CryptCreateHash error code 87: ERROR_INVALID_PARAMETER

Question

I am attempting to host a web application on a new Windows server 2012 environment, however I am receiving an unprecidented error. This code has existed in our codebase for

Answer 1

If you have a type ULONG_PTR, it needs to be defined as IntPtr in .NET. You'll also need a DllImportAttribute Your CryptCreateHash should be:

Declare Auto Function CryptCreateHash Lib "advapi32.dll" _
    (ByVal hProv As IntPtr, _
     ByVal algId As Integer, _
     ByVal hKey As IntPtr, _
     ByVal dwFlags As Integer, _
     ByRef phHast As IntPtr) As <MarshalAs(UnmanagedType.Bool)> Boolean

Also, be sure you've told it to set the last error. In C# we use a DllImportAttribute and make sure that SetLastError=true. Otherwise, calling Marshal.GetLastWin32Error doesn't return what's expected.

Update

Your CryptDecrypt prototype should be:

Declare Function CryptDecrypt Lib "advapi32.dll" 
    (ByVal hkey As IntPtr, _
     ByVal hHash As IntPtr, _
     <MarshalAs(UnmanagedType.Bool)> ByVal final As Boolean, _
     ByVal flags As Integer, _
     ByVal data As Byte(), 
     ByRef dataLen As Integer) As <MarshalAs(UnmanagedType.Bool)> Boolean

You'll need to convert your string to a byte array. Also note that the dataLen parameter is the length of the byte buffer, not the length of the string.

You should check out pinvoke.net, which has managed prototypes and examples for most Windows API calls.