LINUX: how to detect that ftp file upload is finished

Question

In my project I have a file uploading feature. Files are uploaded via FTP. I need to configure a listener that will check for new files and invoke a script only when file up

Answer 1

Have a look at inotify

It doesn't automatically watch sub directories though, so if you need to monitor many ftp accounts (or the FTP client wants to create a sub dir and upload there) you'll need to handle this yourself.

Answer 2

I have used the HiddenStores feature of Proftpd. It keeps in-transit files hidden by prefixing them with a .in.filename.ext until they have finished uploading. Your process can then safely list the directories for completed files.

http://www.proftpd.org/docs/directives/linked/config_ref_HiddenStores.html

Answer 3

An old one but still worth adding good ideas to.

Basically, because of the nature of the internet it's pretty difficult to detect that a file is the entire file you want. Here's what I'd do:

Have clients perform the upload to a staging directory, let's call it "upload", once the put (upload) has finished the client should then rename the file to place it in another directory let's call it "ready". If the client has some issue mid transfer, the file will never end up in the "ready" directory. So no partial uploads.

Clients should handle the upload error and retry. Most clients will carry on uploading the file without needing to send the whole thing again.

On the server side you only need to act on the files in the "ready" directory and monitor for long lived files in the "upload" directory.

For extra assurance, the csender could generate a checksum file containing the sha256 hash of the contents of the file and send that with the file. That would allow you to validate the contents against the hash before doing anything with the file.

Answer 4

I'd try using inotify, event code IN_CLOSE_WRITE.

Answer 5

Apache "Mina" ftp server (java) might be able to do what you want, including detecting a failed upload, as mentioned here

Quote:

From Ftplet.afterCommand, you should be able to look at the reply. For those failed transfers that FtpServer can detect (that causes an SocketException or IOException) this should be something like 426 or 551.

Ftplet overview here, including response codes.

The afterCommand method signature:

FtpletResult afterCommand(FtpSession session, FtpRequest request, FtpReply reply)

You'd check reply.getCode() in your overriden method. You should subclass DefaultFtplet rather than implementing Ftplet interface from scratch.

Note that DefaultFtplet::afterCommand shows how to detect what client command is being responded to. You can check for STOR or STOU and reply-code 426 or 551 to detect failed uploads.

However, this may not detect an upload intentionally terminated by the client, if the client app decides to treat the transfer as though the file was just shorter than it is. In the case of a unintentionally broken connection, I think the reply-code check will work. A test could be to kill the client app, or bring down the client machine's network interface.

To handle successful uploads (your original question), you can look for the success reply-code instead, ie 226.

Answer 6

I was looking for the same thing and stumbled on pureftpd which has an upload script feature. Sounds like exactly what was needed. Found the details here: http://www.linuxbyexamples.net/2012/10/config-ftp-server-trigger-upload-file-to-call-external-script.html