Is there a way to know how much buffer is left for FileSystemwatcher?

问题

When I query

fileSystemWatcher.InternalBufferSize

It will give the total Internal buffer size allocated to the Watcher. But I want to know (during debugging) how much buffer size for the Watcher is left and can be used and when I use the above statement in the Event handler method (say for the write operation) it always gives me the total buffer allocated size to the Watcher. Is there any way to obtain the remaining size of the buffer?

回答1:

No, you can't know how much buffer is left.

It is an implementation detail hidden in an internal class called FSWAsyncResult. If you get hold of an instance of that class and the buffer byte array it contains you still can't reliably answer how much space is left as that byte array only acts as reserved memory for the result of a call to ReadDirectoryChangesW

Find at the bottom of this answer a stripped down, reverse engineered version version of watching a folder for filechanges. Its logic and code matches what you'll find in the real FileSystemWatcher. I didn't bother to replace the magic constants with their proper meaning. It just works™. Don't forget to change the build setting unsafe as the code fiddles with pointers and native structures a lot. And I stripped out all error handling ...

If you follow below code, you'll notice that there is only one place where the byte[] buffer is created, and that only happens once. That same buffer is re-used. Reading the documentation, blogs and worker and I/O threads I understand that ReadDirectoryChangesW is used to issue a callback in an I/O completion fashion. It doesn't matter much for the managed world, that is just another thread.

The callback is scheduled on managed threadpool thread. Sometimes you'll get the same managed id you had before, when it is busy you get several. On that thread CompletionStatusChanged is executed. And that method is responsible for processing all the events that are present in the current byte buffer. Notice that I included a sizeused variable so you can see the actual size of valid data that was present in the buffer. For each event it found it raises/calls the subcribers of the events synchronously (so on the same thread). Once that is complete it calls Monitor again with the same byte[] buffer it just processed. Any file changes during the time CompletionStatusChanged is executing are kept by the OS and send the next time CompletionStatusChanged is called.

tl;dr; Here is a recap of the answers to your questions:

... I want to know (during debugging) how much buffer size for the Watcher is left and can be used

There is only one buffer used and it makes no sense to know how much is used or how much is left. Once your eventhandlers are called the buffer is reset and starts at 0 again. It raises an exception when there are more events then the byte buffer can handle.

Will the main thread wait to finish the processing of all the events?

The OS will issue an asynchronous callback via an IOCompletionPort but that will manifest itself as normal managed threadpool thread. That thread will handle all events in the current buffer and call the eventhandlers.

Which thread will clear the internal buffer associated with the Watcher and when?

The thread that executes the CompletionStatusChanged method. Notice in my testing the buffer was never cleared (as in filled with zeroes). Data was just overwritten.

I have read at lots of places that the handler method should take as minimum time as possible or we can get InternalBufferOverflow Exception. So, is it safe to assume that the Internal Buffer for the Watcher is only cleaned up when the thread(s) (I can't say one or all, but want to ask from you) which are processing the handler method has processed the method?

You should keep your processing as short as possible because there is only one thread that will call all eventhandlers and in the end it has to call ReadDirectoryChangesW again. During this time it will keep track of filechanges. When those filechange events don't fit in the buffer it will raise an InternalBufferOverflow the next time the completion method is called.

Setup

A simple console app, with a ReadLine to keep it running while waiting for events.

static object instance = new object(); // HACK
static SafeFileHandle hndl; // holds our filehandle (directory in this case)

static void Main(string[] args)
{

    // the folder to watch
    hndl = NativeMethods.CreateFile(@"c:\temp\delete", 1, 7, IntPtr.Zero, 3, 1107296256, new SafeFileHandle(IntPtr.Zero, false));
    // this selects IO completion threads in the ThreadPool
    ThreadPool.BindHandle(hndl);

    // this starts the actual listening
    Monitor(new byte[4096]);

    Console.ReadLine();

}

Monitor

This method is responsible for creating the Native structures and an instance of a helper class to act as IAsyncResult implementation.
This method also calls ReadDirectoryChangesW and it chooses the combination of parameters that sets it up for asynchronous completion, with IOCompletinPorts. More background on those options can be found in Understanding ReadDirectoryChangesW - Part 1

static unsafe void Monitor(byte[] buffer)
{

    Overlapped overlapped = new Overlapped();

    // notice how the buffer goes here as instance member on AsyncResult.
    // Arrays are still Reference types.      
    overlapped.AsyncResult = new AsyncResult { buffer = buffer };
    // CompletionStatusChanged is the method that will be called
    // when filechanges are detected
    NativeOverlapped* statusChanged = overlapped.Pack(new IOCompletionCallback(CompletionStatusChanged), buffer);

    fixed (byte* ptr2 = buffer)
    {
        int num;
        // this where the magic starts
        NativeMethods.ReadDirectoryChangesW(hndl, 
          new HandleRef(instance, (IntPtr)((void*)ptr2)), 
          buffer.Length, 
          1, 
          (int)(NotifyFilters.FileName | NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.Attributes), 
          out num, 
          statusChanged, 
          new HandleRef(null, IntPtr.Zero));
    }

}

CompletionStatusChanged

The CompletionStatusChanged method is called by the OS as soon as a filechange is detected. In the Overlapped structure we will find, after unpacking, our earlier ResultAsync instance with a filled buffer. The remainder of the method then decodes the byte array by reading the offset for any following events in the array as well as flags and the filename.

// this gets called by a ThreadPool IO Completion thread
static unsafe void CompletionStatusChanged(uint errorCode, uint numBytes, NativeOverlapped* overlappedPointer)
    {
        var sb = new StringBuilder();

        Overlapped overlapped = Overlapped.Unpack(overlappedPointer);
        var result = (AsyncResult) overlapped.AsyncResult;

        var position = 0;
        int offset;
        int flags;
        int sizeused = 0;
        string file;
        // read the buffer,
        // that can contain multiple events
        do
        {
            fixed (byte* ptr = result.buffer)
            {
                // process FILE_NOTIFY_INFORMATION
                // see https://msdn.microsoft.com/en-us/library/windows/desktop/aa364391(v=vs.85).aspx
                offset = ((int*)ptr)[position / 4];
                flags = ((int*)ptr + position / 4)[1];
                int len = ((int*)ptr + position / 4)[2];
                file = new string((char*)ptr + position / 2 + 6, 0, len / 2);
                sizeused = position + len + 14; 
            }
            sb.AppendFormat("#thread {0},  event: {1}, {2}, {3}, {4}\r\n", Thread.CurrentThread.ManagedThreadId, position, offset, flags, file);
            // in the real FileSystemWatcher here the several events are raised
            // so that uses the same thread this code is on.
            position += offset;
        } while (offset != 0);

        // my own logging
        sb.AppendFormat(" === buffer used: {0} ==== ", sizeused);

        Console.WriteLine(sb);

        // start again, reusing the same buffer:
        Monitor(result.buffer);
    }
}

Helper methods

The AsyncResult implements IAsyncResult (all empty) and holds the member to the byte array buffer.
The NativeMethods are exactly what they are called: entry points for Native calls into the WinAPI.

class AsyncResult : IAsyncResult
{
    internal byte[] buffer;
    // default implementation of the interface left out

    // removed default implementation for brevity
}

static class NativeMethods
{
    [DllImport("kernel32.dll", BestFitMapping = false, CharSet = CharSet.Auto)]
    public static extern SafeFileHandle CreateFile(string lpFileName, int dwDesiredAccess, int dwShareMode, IntPtr lpSecurityAttributes, int dwCreationDisposition, int dwFlagsAndAttributes, SafeFileHandle hTemplateFile);

    [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    public unsafe static extern bool ReadDirectoryChangesW(SafeFileHandle hDirectory, HandleRef lpBuffer, int nBufferLength, int bWatchSubtree, int dwNotifyFilter, out int lpBytesReturned, NativeOverlapped* overlappedPointer, HandleRef lpCompletionRoutine);
}

来源：https://stackoverflow.com/questions/45586630/is-there-a-way-to-know-how-much-buffer-is-left-for-filesystemwatcher

标签

multithreading

events

buffer

filesystemwatcher