Correct way to marshall uchar[] from native dll to byte[] in c#

Question

I\'m trying to marshall some data that my native dll allocated via CoTaskMemAlloc into my c# application and wondering if the way I\'m doing it is just plain wrong or I\'m m

Answer 1

There are two things.

First, the P/Invoke layer does not handle reference parameters in C++, it can only work with pointers. The last two parameters (pOutputBuffer and uOutputSize) in particular are not guaranteed to marshal correctly.

I suggest you change your C++ method declaration to (or create a wrapper of the form):

extern "C" __declspec(dllexport) bool __stdcall CompressData(  
    unsigned char* pInputData, unsigned int inSize, 
    unsigned char** pOutputBuffer, unsigned int* uOutputSize)

That said, the second problem comes from the fact that the P/Invoke layer also doesn't know how to marshal back "raw" arrays (as opposed to say, a SAFEARRAY in COM that knows about it's size) that are allocated in unmanaged code.

This means that on the .NET side, you have to marshal the pointer that is created back, and then marshal the elements in the array manually (as well as dispose of it, if that's your responsibility, which it looks like it is).

Your .NET declaration would look like this:

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport(dllName)]
public static extern bool CompressData(byte[] inputData, uint inputSize, 
    ref IntPtr outputData, ref uint outputSize);

Once you have the outputData as an IntPtr (this will point to the unmanaged memory), you can convert into a byte array by calling the Copy method on the Marshal class like so:

var bytes = new byte[(int) outputSize];

// Copy.
Marshal.Copy(outputData, bytes, 0, (int) outputSize);

Note that if the responsibility is yours to free the memory, you can call the FreeCoTaskMem method, like so:

Marshal.FreeCoTaskMem(outputData);

Of course, you can wrap this up into something nicer, like so:

static byte[] CompressData(byte[] input, int size)
{
    // The output buffer.
    IntPtr output = IntPtr.Zero;

    // Wrap in a try/finally, to make sure unmanaged array
    // is cleaned up.
    try
    {
        // Length.
        uint length = 0;

        // Make the call.
        CompressData(input, size, ref output, ref length);

        // Allocate the bytes.
        var bytes = new byte[(int) length)];

        // Copy.
        Marshal.Copy(output, bytes, 0, bytes.Length);

        // Return the byte array.
        return bytes;
    }
    finally
    {
        // If the pointer is not zero, free.
        if (output != IntPtr.Zero) Marshal.FreeCoTaskMem(output);
    }
}