How to use WndProc as a class function [duplicate]

Answer 1

Unfortunately you cannot use a class function as a wndproc because as the compiler tries to tell you the calling convention differs, even though the two functions have the same signature, a class function expects the this pointer to be passed to it. On 64 bit builds it will expect it to be in the RCX/ECX registry while on 32 bit builds it will expect the this pointer to be the last argument pushed on the stack. The window code won't do that when calling your WndProc essentially turning this into a function call on a garbage pointer.

What you can do is make a static method that does something like the following:

LRESULT Client::CreateMen(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{
    // The OS makes sure GWLP_USERDATA is always 0 before being initialized by the application
    Client* client = (Client*)GetWindowLongPtr(hwnd, GWLP_USERDATA);

    if(msg == WM_INIT)
    {
        client = new Client();
        SetWindowLongPtr(hwnd, GWLP_USERDATA, client);
    }

    if(msg == WM_DESTROY)
    {
        client = (Client*)GetWindowLongPtr(hwnd, GWLP_USERDATA, client);
        SetWindowLongPtr(hwnd, GWLP_USERDATA, NULL);
        delete client;
        client = NULL;
    }

    if(client)
    {
        // Do stuff with the client instance
    }

    return DefWindowProc(hwnd, msg, wparam, lparam);
}

I haven't tested this, so it might have some bugs, but let me know if you have any problems with it and I'll refine it if need be.

Answer 2

A non-static class method has a hidden this parameter. That is what prevents the method from being used as a WndProc (or any other API callback). You must declare the class method as static to remove that this parameter. But as you already noticed, you cannot access non-static members from a static method. You need a pointer to the object in order to access them.

In the specific case of a WndProc callback, you can store the object pointer in the HWND itself (using either SetWindowLong/Ptr(GWL_USERDATA) or SetProp()), then your static method can retrieve that object pointer from the hWnd parameter (using GetWindowLong/Ptr(GWL_USERDATA) or GetProp()) and access non-static members using that object pointer as needed. For example:

private:
    HWND m_Wnd;
    static LRESULT CALLBACK WndProc(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam);

LRESULT CALLBACK Client::WndProc(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
    Client *pThis;

    if (msg == WM_NCCREATE)
    {
        pThis = static_cast<Client*>(reinterpret_cast<CREATESTRUCT*>(lParam)->lpCreateParams);

        SetLastError(0);
        if (!SetWindowLongPtr(hwnd, GWL_USERDATA, reinterpret_cast<LONG_PTR>(pThis)))
        {
            if (GetLastError() != 0)
                return FALSE;
        }
    }
    else
    {
        pThis = reinterpret_cast<Client*>(GetWindowLongPtr(hwnd, GWL_USERDATA));
    }

    if (pThis)
    {
        // use pThis->member as needed...
    }

    return DefWindowProc(hwnd, msg, wParam, lParam);
}

m_Wnd = CreateWindowEx(..., this);