FTP server using S3 as storage

Question

I am trying to create a ftp server ( using windows/linux/mac - no concern) which would have its storage as a Amazon S3 storage. Now note that S3 does not support FTP natively so

Answer 1

The answer really depends.

First, let me say FTP is a terrible and insecure protocol. Make sure you have a good reason before going down this route. There are plenty of user-friendly S3 tools.

Second, please note that none of these solutions will scale like S3 does. Each solution has arbitrary limits on how many files it can support, how large the files can be, and what happens if a file is updated frequently (i.e. it may save the wrong version). S3 Filesystems look neat at first, but when they have problems, they are hard to troubleshoot (They can only return generic filesystem error messages) and a harder to fix.

Some ideas:

• If you really just want cloud backup, consider using EBS instead of S3. Either attach an EBS drive to an EC2 box, or run Storage Gateway on your local box.

• depending on the read/write patterns and the delays, and the size of the files, etc, you might use something like s3sync instead. Have it download all your files, then do a bi-directional re-sync to S3 periodically to pickup any new files or delete any files that have been deleted in S3.

• If you only need to support uploads, just have a cron job that uploads new files to S3 periodically, then deletes them.