How does a Windows antivirus hook into the file access process?

Question

The subject says it all. A normal antivirus has to intercept all file accesses, scan the files and then optionally deny access to the file (possibly even displaying a prompt

Answer 1

In general, these products intercept functions to get a HANDLE to a process like OpenProcess or NtOpenProcess. They also, hook CreateRemoteThread functions and memory allocation in a remote process: VirtualAlloc and VirtualProcect. Some AVs also hook SetWindowsHookEx function to detect global hooks to avoid key loggers.

Hooking these APIs they can control which modules (or dlls) can access remote processes and allow only those that the user know what they are doing.

You can use HookShark to see what user-mode functions are intercepted by each AV product.

To make your own user hooks you can use detours library but you have to develop an agent to run your in-process hooks and then communicate with an agent server. You can also use Deviare API Hook which is a framework that makes all the complex staff so you can code your hooks in your own process using any programming language.

Answer 2

File access is monitored using filesystem filter driver, which works in kernel mode. Filter drivers can be not just notified about filesystem operations, but alter the data passed via filters or deny filesystem requests.

You can create a minifilter yourself, yet maintenance and support of your kernel-mode code can be non-trivial, especially without kernel-mode development experience. One of problems is conflicts between various filters.

Our company offers CallbackFilter product, which provides a ready-to-use driver and lets you write business logic, related to filtering, in user mode.

Answer 3

You can read about the detours library from microsoft and try it for free - it allows you to write user mode hooks in c#. No need for you to learn about drivers :]

However - for kernel mode hooks - you will need to know c and play around with the DDK - atleast afaik :[

And most modern anti-virus software intercept quite a few calls - registry apis, thread and process apis etc - not just the file system api. Again - afaik.

edit: There are also a few open source rootkits - google them and see how they perform their hooking, it will be educational I guess.

Answer 4

Through File System Filter Drivers. However, implementing such drivers is quite complicated and "fragile".

Answer 5

As you already noted, hooking is a key to what of-the-shelf AV software with "realtime" protection does.

You could have a look on the (widely discussed) winpooch, which already does API Hooking, but there are some major flaws in this software. Sourceforge of Winpooch

There is also an article on Codeproject on API hooking, providing some library to do hooking "in three layers". Dll Injection is somewhat hard, as you can image. CodeProject: EasyHook, reinvention of API Hooking

As you are probably interested in Antivirus strategies, i also suggest having a look at ClamAV, or WinClam, which is opensource (under GPL) ClamAV for windows

But i do not have a clue how to do API hooking with C#, i have to admit. In C / C++ this is (quite) easy...

ADD ON You may be interested in the sources of FileMon, a widely known FileSystem Monitor that was once by SysInternals and now by Microsoft: It uses Driver-Filter API by Microsoft, which is at least known as fragile.

Link may be found here in Sysinternals forum

Answer 6

In the recent versions of windows (at least XP onwards) there is the concept 'filters' which can be viewed using MS Filter Manager, (fltmc.exe from a command prompt)

This provides a low level I/O hook that AV programs can access and automatically register to be passed all I/O requests to the file system. It is a kit you can get the drivers for an develop your own filters for.

http://www.microsoft.com/whdc/driver/filterdrv/default.mspx is a starting place to get in depth info.