FileSystemWatcher InternalBufferOverflow

Question

I am getting an exception System.IO.Internal.BufferOverflowException when I am trying to monitor a folder on network path(DFS - Distributed File System): To many changes at once . It works fine when FileSystemWatcher is monitoring local/network path that don't use this filesystem.

I am able to get an event from 1000 + files on local path and I am not getting BufferOverflow exception, however when I am copying file to folder that is on DFS I am not even able to get an event from one(To clarify this, I am getting an error event raised...).

I've already tried to set:

fileSystemWatcher.InternalBufferSize = 65536;

I am not sure if this will help you but the path look like this:

\\corpnet\cloud\\Network\testFolder\myFolderToMonitor

Edit: 1 I am not sure why there are two double slashes in path. I can monitor without a problem folder till the \corpnet\cloud path. I am getting errors once I am trying to monitor any folder that is starting from

...\\Network\...

Any hints from you appreciated.

Thanks

Answer 1

Sure, too many changes at once, this is a firehose problem. You've already increased the buffer size to the maximum allowed, Windows doesn't allow a bigger one. It is allocated in "precious" memory, the kernel memory pool.

This could be a highly active file server, but much more commonly this is caused by a problem in your code. You don't drink from the firehose fast enough. It is imperative that your event handlers return as quickly as possible so that the buffer is emptied fast enough and can keep up with the rate of changes on the file server.

That's very often fumbled, a typical implementation does something unwise like copying the file, reading it, looping until the file can get opened. Expensive things, the looping bug is a very common mistake, a file is very rarely usable when the event fires because whatever app that changes the file still has it opened. There's no upper limit either on how long it can keep a lock on the file. Clearly that will always cause a buffer overflow.

So a properly implemented FileSystemWatcher event handler does nothing but quickly put the passed file path into a thread-safe queue and does nothing else, that never ought to take more than a microsecond. And uses another thread to try to empty that queue again, dealing with the likelihood that the file can't be opened yet.

Answer 2

I got the same issue. There is no problem watching a local folder and dropping in 5 new files. But when I watch a network folder, I get the error "Too many changes at once in directory". It's only 5 files.

You already found a fix?

I can't really adjust the code so my temporary fix is to drop the files in a temp folder and poll that folder. When there are new files, I move them separately with 500ms delay