UPDATED (2013-08-06): Added notes about requiring Xcode to compile rsync (if compiling is the route you choose).
I have a 3TB hard drive full of my RAW (.CR2) photography files that I replicate to another drive every time I Adobe Lightroom. Recently the main drive recently crashed but luckily I had my backup up and running and synced the night before! For those interested, the USB cord was pulled while in operation. I usually take time to recover my fallen hard drives, but because I had a backup, I quickly moved on.
My rsync Command
Now that my backup photography hard drive is my new main drive, I had to sync 3TB of data to a new backup drive. For this, I usually run a simple rsync command as such,
rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/
Easy enough, but the rsync command sat on “building file list…” for hours and hours and hours because 3TB and millions of files is a large “file list” for rsync to create.
Do not use the -z flag for local hard drive to hard drive data transfers as -z is the “compress” flag and can bring your computer to a crawl when you don’t need to use it in the first place.
You can add –delete to remove make sure files deleted on the SOURCE drive are deleted on the DESTINATION drive. e.g. If I delete /Volumes/Photography/old_folder/, it won’t delete it off of /Volumes/PhotographyBackup/ without using –delete.
rsync -av --delete /Volumes/Photography/ /Volumes/PhotographyBackup/
You can also view statistics of the transfer when it’s complete.
rsync -av --stats /Volumes/Photography/ /Volumes/PhotographyBackup/
rsync Out-of-Date on Mac OS X
As of this writing, Apple’s Mac OS X 10.8 Mountain Lion’s rsync is at version 2.6, while rsync has had 3.0+ released for some time now. Luckily rsync 3.0+ has a reduced memory footprint and creates a chunked (bit-by-bit) “file list”, rather than waiting for the entire list to be made first and then beginning transfer. This means moving to rsync 3.0+ will dramatically speed up the time before rsync begins its first transfer, e.g. fix the “building file list…” hang.
They state this explicitly in rsync 3.0+ by saying “sending incremental file list” rather than rsync 2.6′s “building file list…”.
Quick Note on Linux
I looked up Debians’s & Ubuntu’s, Fedora’s and SuSE’s package version for rsync and luckily they have been using 3.0+ for some time, so if you’re on a modern Linux distro, you don’t need to follow these instructions. If you’re on a Linux server or an old Linux desktop/laptop install, you may want to check for yourself.
Installing rsync 3.0+ on Mac OS X
Do not run all of these commands, you have a choice of which one you prefer.
A) Use Homebrew
If you use Homebrew, you can run this command,
brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb
B) Download & Compile
i) Download the Stable Version
Please visit the stable folder to find the most recent stable version as the version numbers may have changed.
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
tar -xzvf rsync-3.0.9.tar.gz
ii) Download the (in development) Nightly Version
curl -O https://rsync.samba.org/ftp/rsync/dev/nightly/rsync-HEAD.tar.gz
tar -xzvf rsync-HEAD.tar.gz
sudo make install
chmod +x ./rsync
If you receive,
configure.sh: error: no acceptable C compiler found in $PATH
-bash: make: command not found
It is because you don’t have the proper compiler tools installed. Apple requires Xcode to be downloaded and installed from the App Store and have the “Command Line Tools” installed to compile software.
Once the app has been downloaded and installed, to install “Command Line Tools” go to “Xcode” -> “Preferences” -> “Download” and install “Command Line Tools” there.
Running the Command
Staying inside the folder it was compiled, you can now run your rsync command similar to before,
./rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/
Or because rsync 3.0+ has a full-transfer progress indicator, rather than a file-by-file progress indicator, feel free to use –info=progress2 as an option.
./rsync -a --info=progress2 /Volumes/Photography/ /Volumes/PhotographyBackup/