Keep Remote Directory Up-to-date

Question

I absolutely love the Keep Remote Directory Up-to-date feature in Winscp. Unfortunately, I can\'t find anything as simple to use in OS X or Linux. I know the same thing can

Answer 1

Well, I had the same kind of problem and it is possible using these together: rsync, SSH Passwordless Login, Watchdog (a Python sync utility) and Terminal Notifier (an OS X notification utility made with Ruby. Not needed, but helps to know when the sync has finished).

1. I created the key to Passwordless Login using this tutorial from Dreamhost wiki: http://cl.ly/MIw5

   1.1. When you finish, test if everything is ok… if you can't Passwordless Login, maybe you have to try afp mount. Dreamhost (where my site is) does not allow afp mount, but allows Passwordless Login. In terminal, type:

   ssh username@host.com You should login without passwords being asked :P

2. I installed the Terminal Notifier from the Github page: http://cl.ly/MJ5x

   2.1. I used the Gem installer command. In Terminal, type:

   gem install terminal-notifier

   2.3. Test if the notification works.In Terminal, type:

   terminal-notifier -message "Starting sync"

3. Create a sh script to test the rsync + notification. Save it anywhere you like, with the name you like. In this example, I'll call it ~/Scripts/sync.sh I used the ".sh extension, but I don't know if its needed.

   #!/bin/bash terminal-notifier -message "Starting sync" rsync -azP ~/Sites/folder/ user@host.com:site_folder/ terminal-notifier -message "Sync has finished"

   3.1. Remember to give execution permission to this sh script. In Terminal, type:

   sudo chmod 777 ~/Scripts/sync.sh 3.2. Run the script and verify if the messages are displayed correctly and the rsync actually sync your local folder with the remote folder.

4. Finally, I downloaded and installed Watchdog from the Github page: http://cl.ly/MJfb

   4.1. First, I installed the libyaml dependency using Brew (there are lot's of help how to install Brew - like an "aptitude" for OS X). In Terminal, type:

   brew install libyaml

   4.2. Then, I used the "easy_install command". Go the folder of Watchdog, and type in Terminal:

   easy_install watchdog

5. Now, everything is installed! Go the folder you want to be synced, change this code to your needs, and type in Terminal:

     watchmedo shell-command
         --patterns="*.php;*.txt;*.js;*.css" \
         --recursive \
         --command='~/Scripts/Sync.sh' \
         .
   

   It has to be EXACTLY this way, with the slashes and line breaks, so you'll have to copy these lines to a text editor, change the script, paste in terminal and press return.

   I tried without the line breaks, and it doesn't work!

   In my Mac, I always get an error, but it doesn't seem to affect anything:

   /Library/Python/2.7/site-packages/argh-0.22.0-py2.7.egg/argh/completion.py:84: UserWarning: Bash completion not available. Install argcomplete.

   Now, made some changes in a file inside the folder, and watch the magic!

Answer 2

How "real-time" do you want the syncing? I would still lean toward rsync since you know it is going to be fully supported on both platforms (Windows, too, with cygwin) and you can run it via a cron job. I have a super-simple bash file that I run on my system (this does not remove old files):

#!/bin/sh
rsync -avrz --progress --exclude-from .rsync_exclude_remote . remote_login@remote_computer:remote_dir    

# options
#   -a  archive
#   -v  verbose
#   -r  recursive
#   -z  compress 

Your best bet is to set it up and try it out. The -n (--dry-run) option is your friend!

Keep in mind that rsync (at least in cygwin) does not support unicode file names (as of 16 Aug 2008).

Answer 3

lsyncd seems to be the perfect solution. it combines inotify (kernel builtin function which watches for file changes in a directory trees) and rsync (cross platform file-syncing-tool).

lsyncd -rsyncssh /home remotehost.org backup-home/

Quote from github:

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.

Answer 4

You could always use version control, like SVN, so all you have to do is have the server run svn up on a folder every night. This runs into security issues if you are sharing your files publicly, but it works.

If you are using Linux though, learn to use rsync. It's really not that difficult as you can test every command with -n. Go through the man page, the basic format you will want is

rsync [OPTION...] SRC... [USER@]HOST:DEST

the command I run from my school server to my home backup machine is this

rsync -avi --delete ~ me@homeserv:~/School/ >> BackupLog.txt

This takes all of the files in my home directory (~) and uses rsync's archive mode (-a), verbosly (-v), lists all of the changes made (-i), while deleting any files that don't exist anymore (--delete) and puts the in the Folder /home/me/School/ on my remote server. All of the information it prints out (what was copied, what was deleted, etc.) is also appended to the file BackupLog.txt

I know that's a whirlwind tour of rsync, but I hope it helps.

Answer 5

The rsync solutions are really good, especially if you're only pushing changes one way. Another great tool is unison -- it attempts to syncronize changes in both directions. Read more at the Unison homepage.

Answer 6

Building off of icco's suggestion of SVN, I'd actually suggest that if you are using subversion or similar for source control (and if you aren't, you should probably start) you can keep the production environment up to date by putting the command to update the repository into the post-commit hook.

There are a lot of variables in how you'd want to do that, but what I've seen work is have the development or live site be a working copy and then have the post-commit use an ssh key with a forced command to log into the remote site and trigger an svn up on the working copy. Alternatively in the post-commit hook you could trigger an svn export on the remote machine, or a local (to the svn repository) svn export and then an rsync to the remote machine.

I would be worried about things that detect changes and push them, and I'd even be worried about things that ran every minute, just because of race conditions. How do you know it's not going to transfer the file at the very same instant it's being written to? Stumble across that once or twice and you'll lose all of the time-saving advantage you had by constantly rsyncing or similar.