FileSystemWatcher and windows 7

Question

I am writing a tool that monitors a network directory and is running off of a Windows Server 2008 machine, the OnChanged event for the FileSystemWatcher is being fired corre

Answer 1

From MSDN:

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

If increasing the buffer size is not sufficient and you cannot control how many files are triggering events at a time you would have to add additional polling.

See also this related question:

FileSystemWatcher does not work properly when many files are added to the directory at the same time…

Update:

It might be tempting to simply increase the buffer size but this should be done with care. In fact, there is a 64k limitation when it comes to network access. The FileSystemWatcher class is using the Windows API function ReadDirectoryChangesW underneath which has this limit:

ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.

If you want to get a deeper understanding on the cost of modifying the buffer size you should have a look at the post of Walter Wang of Microsoft here:

FileSystemWatcher across the network (full post quoted below)

I'm sorry that the documentation of FileSystemWatcher.InternalBufferSize didn't state very clear about the buffer size when monitoring network path. It's recommended not exceeds 64K when monitoring network path.

FileSystemWatcher is basically a .Net wrapper for the Win32 ReadDirectoryChangesW API. To use ReadDirectoryChangesW, you create and specify a buffer that the OS will populate with the changes. However, what is not mentioned in the ReadDirectoryChangesW documentation (but is hinted in the FileSystemWatcher docs) is that the file system creates an internal kernel buffer to store the change information temporarily until it has the chance to update the user buffer. The size of the kernel buffer that is created is the same size that is specified in ReadDirectoryChangesW and is created in non-paged pooled memory. Every time a FileSystemWatcher / ReadDirectoryChangesW is created / called, a new kernel buffer is also created.

The kernel memory pools (paged and non-paged) are set aside in the system address space for device drivers and other kernel components to use. They grow and shrink dynamically as necessary. The current size of the pools can be easily seen by going to the Performance tab of the Task Manager. The pools will grow dynamically until they hit a maximum value which is calculated at boot time and depends on available system resources (mostly RAM). You do not want to hit this maximum value or else various system services and drivers will start failing. However, this calculated maximum value is not easily available. To determine the maximum pool sizes, you need to use a kernel debugger. If you are interested in further information about the system memory pools, I recommend that you look at Chapter 7 in the MSPress book Inside Windows 2000 by Solomon and Russinovich.

With this in mind, there is no recommendation on what size buffers you can use. The current and maximum size of the system pools are going to be varied from client to client. However, you probably should not go over 64k for each FileSystemWatcher / ReadDirectoryChangesW buffer. This stems from the fact that there is a 64k limitation with network access as documented in ReadDirectoryChangesW. But in the end you are going to have to test the application on a variety of expected target systems so that you can tune your buffer.

There is overhead associated with .Net applications and I imagine that a Win32 ReadDirectoryChangesW program might be able to achieve better performance with the same buffer size. However, with very fast and numerous file changes, buffer overruns will be inevitable and the developer is going to have to handle the case when an overrun occurs such as manually enumerating the directory to detect the changes.

In conclusion, FileSystemWatcher and ReadDirectoryChangesW are a lightweight file change detection mechanism that is going to have its limitations. Change Journals is another mechanism which we would consider a medium-weight solution, but would still have limitations:

http://msdn.microsoft.com/en-us/library/aa363798%28VS.85%29.aspx

Heavy-weight solutions would be to write a dedicated file system filter driver that sits in the file system stack and monitors file system changes. Of course this would be the most complex approach. Most virus scanners, backup software, and file system monitoring utilities such as filemon (www.sysinternals.com) implement a filter driver.

I hope above explanation helps you to understand the root cause of the issue you're experiencing. Please reply to let us know whether or not you need further information. Thank you.