Monitoring files - how to know when a file is complete

Question

We have several .NET applications that monitor a directory for new files, using FileSystemWatcher. The files are copied from another location, uploaded via FTP, etc. When th

Answer 1

If you are in control on the program that is writing the files into the directory, you can have the program write the files to a temporary directory and then move them into the watched directory. The move should be an atomic operation, so the watcher shouldn't see the file until it is fully in the directory.

If you are not in control of what is writing to the watched directory, you can set a time in the watcher where the file is considered complete when it has remained the same size for the given time. If immediate processing isn't a concern, setting this timer to something relatively large is a fairly safe way to know that either the file is complete or it never will be.

Answer 2

+1 for using a file.ext.end signaler if possible, where the contents of file.ext.end is a checksum for the larger file. This isn't for security so much as it is to make sure nothing got garbled along the way. If someone can insert their own file into the large stream they can replace the checksum as well.

Answer 3

Unless the contents of a file can be verified for completion (it has a verifiable format or includes a checksum of the contents) only the sender can verify that a whole file has arrived.

I have used a locking method for sending large files via FTP in the past.

File is sent with an alternative extension and is renamed once the sender is happy it is all there.

The above is obviously combined with a process which periodically tidies up old files with the temporary extension.

An alternative is to create a zero length file with the same name but with an additonal .lck extension. Once the real file is fully uploaded the lck file is deleted. The receiving process obviously ignores files which have the name of a lock file.

Without a system like this the receiver can never be sure that the whole file has arrived.

Checking for files that haven't been changed in x minutes is prone to all sorts of problems.