问题
how to avoid the error of FileSystemWatcher in C#?
too many changes at once in directory
I have to detect all changes on a network share. The InternalBufferSize is increased to 8192 * 128
回答1:
There are two things you should do:
- Set
InternalBufferSizeto the maximum supported value (65536). Your attempt to set it to "8192 * 128" is larger than the maximum supported value listed in the documentation, so you may not have increased the buffer size at all. - Queue events from the
FileSystemWatcheronto a background thread for processing.
It's the second point here that isn't well understood, and really should be documented on MSDN. Internally, FileSystemWatcher is queuing change events into that internal buffer you set the size of above. Critically however, items are only removed from that buffer after your event handler returns. This means every cycle of overhead your event handlers introduce increases the possibility of the buffer filling up. What you should do is clear the limited queue of the FileSystemWatcher as quickly as possible, and move the events into your own infinite queue, to process at the rate you can handle, or discard if you care to do so, but with some intelligence around it.
Here's basically what I do in my code. First, I start my own dispatcher thread:
Dispatcher changeDispatcher = null;
ManualResetEvent changeDispatcherStarted = new ManualResetEvent(false);
Action changeThreadHandler = () =>
{
changeDispatcher = Dispatcher.CurrentDispatcher;
changeDispatcherStarted.Set();
Dispatcher.Run();
};
new Thread(() => changeThreadHandler()) { IsBackground = true }.Start();
changeDispatcherStarted.WaitOne();
Then I create the watcher. Note the buffer size being set. In my case, I only watch changes in the target directory, not subdirectories:
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.InternalBufferSize = 64 * 1024;
watcher.IncludeSubdirectories = false;
Now I attach my event handlers, but here I invoke them onto my dispatcher rather than running them synchronously in the watcher thread. Yes, the events will be processed in order by the dispatcher:
watcher.Changed += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnChanged(sender, e)));
watcher.Created += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnCreated(sender, e)));
watcher.Deleted += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnDeleted(sender, e)));
watcher.Renamed += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnRenamed(sender, e)));
And finally, after disposing of the FileSystemWatcher (you were doing that, right?), you need to shut down your dispatcher:
watcher.Dispose()
changeDispatcher.BeginInvokeShutdown(DispatcherPriority.Normal);
And that's it. I was getting this problem myself, both in network and local scenarios. After using this approach, I wasn't able to generate this error again, even when hammering out empty files to watched directories as fast as possible. If you did ever manage to somehow exhaust the buffer in this case (which I'm not sure is possible, the API upstream is probably slower), there's still further room for optimization here. As long as your dispatcher is over the "tipping point" though, where the sender can't post the events faster than you can dispatch them, you'll never get a backlog, and hence never blow the buffer. I believe this approach puts you into that safe area.
回答2:
I think I might have found a pattern that can help improve the the usage of the buffer considerably.
The issue with this class is that until the delegates for the events finish running, it cannot release the memory it uses to hold that information.
For the life of me I don't know why the maximum InternalBufferSize was set to 64Kb, but with this idea you'll use that small buffer much more effectively.
If instead of doing the operations inside the delegates you simply queue them and defer their execution for a background worker, the amount of memory used by it will be considerably smaller.
In fact, I've got no idea why the class in itself doesn't implement something like this in the first place.
The code below is just a sample piece of code for the idea, and shouldn't be used in production environments, but it increased the number of files I could copy and keep track of drastically.
using System;
using System.Collections.Concurrent;
using System.ComponentModel;
using System.IO;
using System.Threading;
using NUnit.Framework;
namespace Soundnet.Synchronisation.FileSystemWatcherTests
{
/// <summary>
///
/// </summary>
[TestFixture]
public class Tests
{
static readonly ConcurrentQueue<Change> ChangesQueue = new ConcurrentQueue<Change>();
const string Destination = @"c:\Destination";
const string Source = @"c:\Source";
/// <summary>
/// Tests this instance.
/// </summary>
[Test]
public void Test()
{
var changesBackgroundWorker = new BackgroundWorker();
changesBackgroundWorker.DoWork += ChangesBackgroundWorkerOnDoWork;
changesBackgroundWorker.RunWorkerAsync();
var fileSystemWatcher = new FileSystemWatcher
{
Path = Source,
EnableRaisingEvents = true,
IncludeSubdirectories = true,
NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.LastAccess | NotifyFilters.CreationTime | NotifyFilters.FileName | NotifyFilters.DirectoryName,
InternalBufferSize = 65536
};
fileSystemWatcher.Created += FileSystemWatcherOnCreated;
fileSystemWatcher.Deleted += FileSystemWatcherOnDeleted;
fileSystemWatcher.Renamed += FileSystemWatcherOnRenamed;
fileSystemWatcher.Error += FileSystemWatcherOnError;
while (true)
Thread.Sleep(1000000);
}
/// <summary>
/// Changeses the background worker configuration document work.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="doWorkEventArgs">The <see cref="DoWorkEventArgs"/> instance containing the event data.</param>
/// <exception cref="System.ArgumentOutOfRangeException"></exception>
private static void ChangesBackgroundWorkerOnDoWork(object sender, DoWorkEventArgs doWorkEventArgs)
{
while (true)
{
Change change;
if (ChangesQueue.TryDequeue(out change))
{
var backgroundWorker = new BackgroundWorker();
switch (change.ChangeType)
{
case WatcherChangeTypes.Created:
backgroundWorker.DoWork += (o, args) =>
{
var fileSystemType = GetFileSystemType(change.FullPath);
var newItem = Path.Combine(Destination, change.Name);
while (true)
{
try
{
switch (fileSystemType)
{
case FileSystemType.File:
File.Copy(change.FullPath, newItem, true);
break;
case FileSystemType.Directory:
var directorySecurity =
Directory.GetAccessControl(change.FullPath);
Directory.CreateDirectory(newItem, directorySecurity);
break;
case FileSystemType.NotExistant:
break;
}
return;
}
catch (IOException exception)
{
Thread.Sleep(100);
Console.WriteLine(exception.Message);
}
}
};
break;
case WatcherChangeTypes.Deleted:
backgroundWorker.DoWork += (o, args) =>
{
var itemToDelete = Path.Combine(Destination, change.Name);
var fileSystemType = GetFileSystemType(itemToDelete);
switch (fileSystemType)
{
case FileSystemType.File:
File.Delete(itemToDelete);
break;
case FileSystemType.Directory:
Directory.Delete(itemToDelete, true);
break;
}
};
break;
case WatcherChangeTypes.Changed:
backgroundWorker.DoWork += (o, args) =>
{
var fileSystemType = GetFileSystemType(change.FullPath);
var newItem = Path.Combine(Destination, change.Name);
switch (fileSystemType)
{
case FileSystemType.File:
File.Copy(change.FullPath, newItem, true);
break;
}
};
break;
case WatcherChangeTypes.Renamed:
backgroundWorker.DoWork += (o, args) =>
{
var fileSystemType = GetFileSystemType(change.FullPath);
var oldItem = Path.Combine(Destination, change.OldName);
var newItem = Path.Combine(Destination, change.Name);
switch (fileSystemType)
{
case FileSystemType.File:
if (File.Exists(oldItem))
File.Move(oldItem, newItem);
break;
case FileSystemType.Directory:
if (Directory.Exists(oldItem))
Directory.Move(oldItem, newItem);
break;
}
};
break;
case WatcherChangeTypes.All:
break;
default:
throw new ArgumentOutOfRangeException();
}
backgroundWorker.RunWorkerAsync();
}
}
}
/// <summary>
/// Files the system watcher configuration created.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="fileSystemEventArgs">The <see cref="FileSystemEventArgs"/> instance containing the event data.</param>
private static void FileSystemWatcherOnCreated(object sender, FileSystemEventArgs fileSystemEventArgs)
{
ChangesQueue.Enqueue(new Change
{
ChangeType = WatcherChangeTypes.Created,
FullPath = fileSystemEventArgs.FullPath,
Name = fileSystemEventArgs.Name
});
}
/// <summary>
/// Files the system watcher configuration deleted.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="fileSystemEventArgs">The <see cref="FileSystemEventArgs"/> instance containing the event data.</param>
private static void FileSystemWatcherOnDeleted(object sender, FileSystemEventArgs fileSystemEventArgs)
{
ChangesQueue.Enqueue(new Change
{
ChangeType = WatcherChangeTypes.Deleted,
FullPath = fileSystemEventArgs.FullPath,
Name = fileSystemEventArgs.Name
});
}
/// <summary>
/// Files the system watcher configuration error.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="errorEventArgs">The <see cref="ErrorEventArgs"/> instance containing the event data.</param>
private static void FileSystemWatcherOnError(object sender, ErrorEventArgs errorEventArgs)
{
var exception = errorEventArgs.GetException();
Console.WriteLine(exception.Message);
}
/// <summary>
/// Files the system watcher configuration renamed.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="fileSystemEventArgs">The <see cref="RenamedEventArgs"/> instance containing the event data.</param>
private static void FileSystemWatcherOnRenamed(object sender, RenamedEventArgs fileSystemEventArgs)
{
ChangesQueue.Enqueue(new Change
{
ChangeType = WatcherChangeTypes.Renamed,
FullPath = fileSystemEventArgs.FullPath,
Name = fileSystemEventArgs.Name,
OldFullPath = fileSystemEventArgs.OldFullPath,
OldName = fileSystemEventArgs.OldName
});
}
/// <summary>
/// Gets the type of the file system.
/// </summary>
/// <param name="fullPath">The full path.</param>
/// <returns></returns>
private static FileSystemType GetFileSystemType(string fullPath)
{
if (Directory.Exists(fullPath))
return FileSystemType.Directory;
if (File.Exists(fullPath))
return FileSystemType.File;
return FileSystemType.NotExistant;
}
}
/// <summary>
/// Type of file system object
/// </summary>
internal enum FileSystemType
{
/// <summary>
/// The file
/// </summary>
File,
/// <summary>
/// The directory
/// </summary>
Directory,
/// <summary>
/// The not existant
/// </summary>
NotExistant
}
/// <summary>
/// Change information
/// </summary>
public class Change
{
/// <summary>
/// Gets or sets the type of the change.
/// </summary>
/// <value>
/// The type of the change.
/// </value>
public WatcherChangeTypes ChangeType { get; set; }
/// <summary>
/// Gets or sets the full path.
/// </summary>
/// <value>
/// The full path.
/// </value>
public string FullPath { get; set; }
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the old full path.
/// </summary>
/// <value>
/// The old full path.
/// </value>
public string OldFullPath { get; set; }
/// <summary>
/// Gets or sets the old name.
/// </summary>
/// <value>
/// The old name.
/// </value>
public string OldName { get; set; }
}
}
回答3:
From MSDN;
The Windows operating system notifies your component of file changes in a buffer created by the
FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with theInternalBufferSizeproperty is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use theNotifyFilterandIncludeSubdirectoriesproperties so you can filter out unwanted change notifications.
回答4:
I have the experience that FileSystemWatcher is not always the most reliable thing to use. You could specify the filter to narrow down the files you are watching (NotifyFilter), or increase the buffer size.
But depending on your requirements you may also want to do it another way, like polling every x seconds to get the list of files. But then you might need to tell us more about your business case.
回答5:
SHChangeNotifyRegister can be used to get shell notifications.
回答6:
It should be fixed if you increase the buffer size but its not a practical solution. Because to make sure it always records everything you would need to make the buffer huge. And that will effect the performance greatlly. And i think performance issues could be fixed by implementing multi threading.
来源：https://stackoverflow.com/questions/15519089/avoid-error-too-many-changes-at-once-in-directory