UPDATED (2016-01-31): Mac OS X 10.11 El Capitan still uses rsync 2 so as of early 2016, this tutorial is still required to speed up rsync. Check your version via,
If the output says something like, rsync version 3.1.2, then great, but for most of you, without manually compiling an rsync binary or installing rsync via Homebrew, you’ll have to follow this tutorial…
UPDATED (2013-08-06): Added notes about requiring Xcode to compile rsync (if compiling is the route you choose).
I have a 3TB hard drive full of my RAW (.CR2) photography files that I replicate to another drive every week or so.
For this, I usually run a simple rsync command as such,
rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/
Easy enough, but the rsync command sat on “building file list…” for hours and hours and hours because 3TB and hundreds of thousands of files is a large “file list” for rsync to create.
A quick note on rsync flags
Do not use the -z flag for local hard drive to hard drive data transfers as -z is the “compress” flag and can bring your computer to a crawl when you don’t need to use it in the first place. This issue comes from copying and pasting commands without knowing what the flags were doing, so I wanted to point it out.
You can add –delete to remove make sure files deleted on the SOURCE drive are deleted on the DESTINATION drive. e.g. If I delete /Volumes/Photography/old_folder/, it won’t delete it off of /Volumes/PhotographyBackup/ without using –delete.
rsync -av --delete /Volumes/Photography/ /Volumes/PhotographyBackup/
You can also view statistics of the transfer when it’s complete.
rsync -av --stats /Volumes/Photography/ /Volumes/PhotographyBackup/
rsync Out-of-Date on Mac OS X
In Apple’s Mac OS X 10.8 Mountain Lion, 10.9 Mavericks and 10.10 Yosemite, rsync is at version 2.6, while rsync has had 3.0+ released for some time now. This is bad for performance reasons. rsync 3.0+ has a reduced memory footprint and creates a chunked (bit-by-bit) “file list”, rather than waiting for the entire list to be made first and then beginning transfer. This means moving to rsync 3.0+ will dramatically speed up the time before rsync begins its first transfer, e.g. fix the “building file list…” hang.
They state this explicitly in rsync 3.0+ by saying “sending incremental file list” rather than rsync 2.6’s “building file list…”.
Quick Note on Linux
I looked up Debians’s & Ubuntu’s, Fedora’s and SuSE’s package version for rsync and luckily they have been using 3.0+ for some time, so if you’re on a modern Linux distro, you don’t need to follow these instructions. If you’re on a Linux server or an old Linux desktop/laptop install, you may want to check for yourself using,
Installing rsync 3.0+ on Mac OS X 10.8, 10.9, 10.10 & 10.11
Do not run all of these commands, you have a choice of which one you prefer.
A) Use Homebrew
If you use Homebrew, you can run this command,
brew tap homebrew/dupes
brew install rsync
That’s it, you’re done. You can start using rsync 3.
B) Download & Compile
i) Download the Stable Version
Please visit the stable folder to find the most recent stable version as the version numbers may have changed.
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-3.1.2.tar.gz
tar -xzvf rsync-3.1.2.tar.gz
ii) Download the (in development) Nightly Version
curl -O https://rsync.samba.org/ftp/rsync/dev/nightly/rsync-HEAD.tar.gz
tar -xzvf rsync-HEAD.tar.gz
sudo make install
chmod +x ./rsync
If you receive,
configure.sh: error: no acceptable C compiler found in $PATH
-bash: make: command not found
It is because you don’t have the proper compiler tools installed. Apple requires Xcode to be downloaded and installed from the App Store and have the “Command Line Tools” installed to compile software.
Once the app has been downloaded and installed, to install “Command Line Tools” go to “Xcode” -> “Preferences” -> “Download” and install “Command Line Tools” there or use the command-line,
Running the Command
Staying inside the folder it was compiled, you can now run your rsync command similar to before,
./rsync -av /from/this/dir/ /to/this/dir/
Or because rsync 3.0+ has a full-transfer progress indicator, rather than a file-by-file progress indicator, feel free to use –info=progress2 as an option.
./rsync -a --info=progress2 /from/this/dir/ /to/this/dir/