Call of statically link function crash everytimes on windows 8/10 but not 7

Question

The issue :

I have build https://github.com/reorg/pg_repack project whose generate a binary. This binary is linked with postgres 9.6 redistributable (I use those delie

Answer 1

i look your binary package and view that you exec pg_repack.exe which statically import from postgres.exe. and root of problem - is import from exe file. more exactly from PE file which have not flag IMAGE_FILE_DLL in IMAGE_FILE_HEADER.Characteristics , because formal extension (exe, dll, etc not play role). for short i will in next under exe mean PE file without IMAGE_FILE_DLL in Characteristics, but not formal file extension

at first load exe as dll is incorrect - when exe loaded this way - it entry point not called. and can not be called, because it not designed be called as callback, receiving DLL_PROCESS_* notification, and call ExitProcess at the end. when entry point of module is not called - in general it not initialized. assume you call some exported function from exe, but what if this exported function use some data, which normally initialized in exe entry point ? so conclusion - we can call exported functions from exe only from dll loaded in this exe process. and in every process must be only one exe (as executable code).

at second specific postgres.exe have no relocation's ( IMAGE_FILE_RELOCS_STRIPPED flag in IMAGE_FILE_HEADER.Characteristics) - as result this PE can be loaded only at hard-coded address. this not problem for exe, which always mapped first to process, when almost all address space is free. but this in general is problem when PE loaded as DLL - not first in process - the hard-code imagebase can be already busy. so conclusion - you can not safe use any exe without relocation's as DLL

however the root of crash on windows 10 - because windows 10 not resolve PE ("exe") import if this pe have no flag IMAGE_FILE_DLL. in other words it process this PE like LoadLibraryEx with flag DONT_RESOLVE_DLL_REFERENCES - does not load additional executable modules that are referenced by the specified module and nor resolve imports. as result this PE not initialized and will crash at first import function call (in your case this is strcmp).

and this is happens only on win10. on win 8.1 (build 9600) and win 7 import for exe files (loaded as dll) resolved. (you say that on your win 8.1 it crashed - may be you use more new build or some update ? or better check)

the simplest test for this behavior call from test exe MatchToken, function, which exported from Netsh.exe. code can be next:

#include <Netsh.h>
#pragma comment(linker, "/defaultlib:Netsh.lib")

MatchToken(L"*", L"*");// crash here on win 10

crash on win10 because MatchToken internally try call _wcsnicmp from msvcrt.dll, but in win10 this import not resolved. but on win8.1, win7, win xp - this code work well.

little more complex example:

if (HMODULE hmod = LoadLibraryW(L"wshelper.dll"))
{
    DWORD (WINAPI * InitHelperDll)(_In_ DWORD dwNetshVersion, PVOID pReserved);

    if (*(void**)&InitHelperDll = GetProcAddress(hmod, "InitHelperDll"))
    {
        InitHelperDll(1, 0);// crash here on win10 only
    }
    FreeLibrary(hmod);
}

here we load standard windows NetShell helper dll - "wshelper.dll" and call InitHelperDll callback function. internally InitHelperDll call the RegisterHelper function from Netsh.exe. but again, because Netsh.exe not initialized (import not resolved) when it loades as DLL in win 10 - it crashed inside RegisterHelper