In the past few years, malware (and some pen-test tools like Metasploit\'s meterpreter payload) have begun to use reflective DLL injection (PDF) to load a DLL into the memory of
What about hooking the VirtualProtect API. Because DLLs that load itself will certainly set execute on its memory code range. This is because (as you mentioned) they use User access rights so they have to use the process userspace API.
NTSYSAPI NTSTATUS NTAPI ZwProtectVirtualMemory(
IN HANDLE ProcessHandle,
IN PVOID * BaseAddress,
IN SIZE_T * NumberOfBytesToProtect,
IN ULONG NewAccessProtection,
OUT PULONG OldAccessProtection
);
If you hook that at the very beginning of your program, you can filter out suspicious protection calls (the one that enable code execution). I would then scan for PE header or such in front of the requested pages to know that its a loadable module... note: i think this is not called for regular DLLs as LoadLibrary handles this inside the Kernel space. right? TODO: verify
Normally the PE header is located 0x1000 (4096) bytes or one page in front of the first executable code. So a VERY basic approach can be to scan for the "MZ" tag:
char* pe = ((char*)BaseAddress) - 0x1000;
if ((NewAccessProtection == PAGE_EXECUTE || ... ) & pe[0] == 'M' && pe[0] == 'Z')
{
// do checks here
}
If you need further info on API hooking just ask or read tons of articles on the net. Another hooking candidate is: FlushInstructionCache(...). But I think only Blizzard is using this for warden anti cheat modules as theres no reason on x86 architecture to call this.
... just a thought,
will