Why does ReadDirectoryChangesW omit events?

Question

I use ReadDirectoryChangesW to watch a specified directory and update indexing structures whenever a change is detected. I use the following code (roughly)

v         


        

Answer 1

It seems to me that you are mixing the various ways to use ReadDirectoryChangesW(), you do both specify the FILE_FLAG_OVERLAPPED flag when opening the directory and provide a pointer to the lpOverlapped parameter, meaning you want to wait on the event in the structure and handle the asynchronous I/O; and at the same time you call ReadDirectoryChangesW() in a loop in a worker thread. I would first try again with lpOverlapped set to nil, as you have a dedicated thread and can use the synchronous mode.

In the documentation of the ReadDirectoryChangesW() API function the different ways to use it are described. Note that it is also possible that the buffer overflows, so change events can be lost anyway. Maybe you should rethink your strategy of relying solely on this function, comparing snapshots of directory contents could work as well.

Edit:

Your edited code looks better. In my tests however ReadDirectoryChangesW() did work as advertised, there were either several data entries in the returned buffer, or there were more than one buffer to process. This depends on timing, after hitting a breakpoint in Delphi I get several entries in one buffer.

For completeness I attach the test code, implemented using Delphi 5:

type
  TWatcherThread = class(TThread)
  private
    fChangeHandle: THandle;
    fDirHandle: THandle;
    fShutdownHandle: THandle;
  protected
    procedure Execute; override;
  public
    constructor Create(ADirectoryToWatch: string);
    destructor Destroy; override;

    procedure Shutdown;
  end;

constructor TWatcherThread.Create(ADirectoryToWatch: string);
const
  FILE_LIST_DIRECTORY = 1;
begin
  inherited Create(TRUE);
  fChangeHandle := CreateEvent(nil, FALSE, FALSE, nil);
  fDirHandle := CreateFile(PChar(ADirectoryToWatch),
    FILE_LIST_DIRECTORY or GENERIC_READ,
    FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE,
    nil, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS or FILE_FLAG_OVERLAPPED, 0);
  fShutdownHandle := CreateEvent(nil, FALSE, FALSE, nil);
  Resume;
end;

destructor TWatcherThread.Destroy;
begin
  if fDirHandle <> INVALID_HANDLE_VALUE then
    CloseHandle(fDirHandle);
  if fChangeHandle <> 0 then
    CloseHandle(fChangeHandle);
  if fShutdownHandle <> 0 then
    CloseHandle(fShutdownHandle);
  inherited Destroy;
end;

procedure TWatcherThread.Execute;
type
  PFileNotifyInformation = ^TFileNotifyInformation;
  TFileNotifyInformation = record
    NextEntryOffset: DWORD;
    Action: DWORD;
    FileNameLength: DWORD;
    FileName: WideChar;
  end;
const
  BufferLength = 65536;
var
  Filter, BytesRead: DWORD;
  InfoPointer: PFileNotifyInformation;
  Offset, NextOffset: DWORD;
  Buffer: array[0..BufferLength - 1] of byte;
  Overlap: TOverlapped;
  Events: array[0..1] of THandle;
  WaitResult: DWORD;
  FileName, s: string;
begin
  if fDirHandle <> INVALID_HANDLE_VALUE then begin
    Filter := FILE_NOTIFY_CHANGE_FILE_NAME or FILE_NOTIFY_CHANGE_DIR_NAME
      or FILE_NOTIFY_CHANGE_SIZE or FILE_NOTIFY_CHANGE_LAST_WRITE;

    FillChar(Overlap, SizeOf(TOverlapped), 0);
    Overlap.hEvent := fChangeHandle;

    Events[0] := fChangeHandle;
    Events[1] := fShutdownHandle;

    while not Terminated do begin
      if ReadDirectoryChangesW (fDirHandle, @Buffer[0], BufferLength, TRUE,
        Filter, @BytesRead, @Overlap, nil)
      then begin
        WaitResult := WaitForMultipleObjects(2, @Events[0], FALSE, INFINITE);
        if WaitResult = WAIT_OBJECT_0 then begin
          InfoPointer := @Buffer[0];
          Offset := 0;
          repeat
            NextOffset := InfoPointer.NextEntryOffset;
            FileName := WideCharLenToString(@InfoPointer.FileName,
              InfoPointer.FileNameLength);
            SetLength(FileName, StrLen(PChar(FileName)));
            s := Format('[%d] Action: %.8xh, File: "%s"',
               [Offset, InfoPointer.Action, FileName]);
            OutputDebugString(PChar(s));
            PByte(InfoPointer) := PByte(DWORD(InfoPointer) + NextOffset);
            Offset := Offset + NextOffset;
          until NextOffset = 0;
        end;
      end;
    end;
  end;
end;

procedure TWatcherThread.Shutdown;
begin
  Terminate;
  if fShutdownHandle <> 0 then
    SetEvent(fShutdownHandle);
end;

////////////////////////////////////////////////////////////////////////////////

procedure TForm1.FormCreate(Sender: TObject);
begin
  fThread := TWatcherThread.Create('D:\Temp');
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  if fThread <> nil then begin
    TWatcherThread(fThread).Shutdown;
    fThread.Free;
  end;
end;

Deleting a directory does indeed only return one change for it, nothing for the files contained in it. But it does make sense, as you are watching the handle of the parent directory only. If you need notifications for subdirectories you probably need to watch them as well.

Answer 2

We've had the same problem with losing events, especially if a lot of changes happens at the same time, ie. 500 files are copied to the monitored directory.

In the end we found Cromis and use the Directory watch. We have never looked back again.