How to speed up rsync on your Mac (and stop the dreaded “building file list…”)

UPDATED (2013-08-06): Added notes about requiring Xcode to compile rsync (if compiling is the route you choose).

Quick Story

I have a 3TB hard drive full of my RAW (.CR2) photography files that I replicate to another drive every time I Adobe Lightroom. Recently the main drive recently crashed but luckily I had my backup up and running and synced the night before! For those interested, the USB cord was pulled while in operation. I usually take time to recover my fallen hard drives, but because I had a backup, I quickly moved on.

My rsync Command

Now that my backup photography hard drive is my new main drive, I had to sync 3TB of data to a new backup drive. For this, I usually run a simple rsync command as such,

rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/

Screen Shot 2013-08-05 at 3.14.55 PM

Easy enough, but the rsync command sat on “building file list…” for hours and hours and hours because 3TB and millions of files is a large “file list” for rsync to create.

Quick Aside

Do not use the -z flag for local hard drive to hard drive data transfers as -z is the “compress” flag and can bring your computer to a crawl when you don’t need to use it in the first place.

You can add –delete to remove make sure files deleted on the SOURCE drive are deleted on the DESTINATION drive. e.g. If I delete /Volumes/Photography/old_folder/, it won’t delete it off of /Volumes/PhotographyBackup/ without using –delete.

rsync -av --delete /Volumes/Photography/ /Volumes/PhotographyBackup/

You can also view statistics of the transfer when it’s complete.

rsync -av --stats /Volumes/Photography/ /Volumes/PhotographyBackup/

rsync Out-of-Date on Mac OS X

As of this writing, Apple’s Mac OS X 10.8 Mountain Lion’s rsync is at version 2.6, while rsync has had 3.0+ released for some time now. Luckily rsync 3.0+ has a reduced memory footprint and creates a chunked (bit-by-bit) “file list”, rather than waiting for the entire list to be made first and then beginning transfer. This means moving to rsync 3.0+ will dramatically speed up the time before rsync begins its first transfer, e.g. fix the “building file list…” hang.

They state this explicitly in rsync 3.0+ by saying “sending incremental file list” rather than rsync 2.6’s “building file list…”.

Quick Note on Linux

I looked up Debians’s & Ubuntu’s, Fedora’s and SuSE’s package version for rsync and luckily they have been using 3.0+ for some time, so if you’re on a modern Linux distro, you don’t need to follow these instructions. If you’re on a Linux server or an old Linux desktop/laptop install, you may want to check for yourself.

Installing rsync 3.0+ on Mac OS X

Do not run all of these commands, you have a choice of which one you prefer.

A) Use Homebrew

If you use Homebrew, you can run this command,

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

B) Download & Compile

i) Download the Stable Version

Please visit the stable folder to find the most recent stable version as the version numbers may have changed.

cd ~/Downloads/
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
tar -xzvf rsync-3.0.9.tar.gz
cd rsync-3.0.9

Or…

ii) Download the (in development) Nightly Version

cd ~/Downloads/
curl -O https://rsync.samba.org/ftp/rsync/dev/nightly/rsync-HEAD.tar.gz
tar -xzvf rsync-HEAD.tar.gz
cd rsync-HEAD*

Compiling

./configure
make
sudo make install
chmod +x ./rsync

Possible Errors

If you receive,

configure.sh: error: no acceptable C compiler found in $PATH

or,

-bash: make: command not found

It is because you don’t have the proper compiler tools installed. Apple requires Xcode to be downloaded and installed from the App Store and have the “Command Line Tools” installed to compile software.

Once the app has been downloaded and installed, to install “Command Line Tools” go to “Xcode” -> “Preferences” -> “Download” and install “Command Line Tools” there.

Running the Command

Staying inside the folder it was compiled, you can now run your rsync command similar to before,

./rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/

Or because rsync 3.0+ has a full-transfer progress indicator, rather than a file-by-file progress indicator, feel free to use –info=progress2 as an option.

./rsync -a --info=progress2 /Volumes/Photography/ /Volumes/PhotographyBackup/

Screen Shot 2013-08-05 at 4.05.33 PM

  • Roger

    Apple intensely dislike GPL3. The result is that Apple users are stuck with very old versions of tools like rsync (v3 changed license to GPL3), bash, gcc, samba (now removed) etc. Here is a random article about it http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/

    Whatever your opinion about Apple and GPL3, it is negatively affecting Apple users.

    • brettalton

      I’m not a big fan of copyleft licenses like the GPL, but as much as I don’t agree with Apple’s aberrant choices, I don’t think it’s affecting too many users, especially because there are easy alternatives like Homebrew or compiling you own software. If Apple stopped you from compiling your own software, _then_ we’d have a problem.

  • paales

    Easier installation: brew install rsync
    Delete old rsync from /usr/bin/rsync

    • http://infinite-sushi.com _ado

      But only if you did `brew tap homebrew/dupes`

  • Pingback: A Photographers Guide to Backing Up Using RSYNC | Kuljit Athwal – Photographer

  • billtheamerican

    Very good article on rsync. Here is one from around 1999 but still the very best rsync tutorial I’ve ever read. Explains it very well IMO: http://tinyurl.com/l37guv8

    • http://eatingrichly.com/ Eric J

      This is a spam link?

    • brettalton

      My apologies. I deleted it some time back, thanks for letting me know.

  • Pingback: rsync braucht unter Mac OS X ewig für “building file list…” « think eMeidi

  • Pingback: Brew, rsync and command line errors . . . - Mac-Forums Discussions for Apple Products & Services

  • Jolly D’Bugger

    Oh jezz, what did i do?
    I have homebrew installed and rsync 3.1.0
    rsync fails on …”–info=progress2″… and reports ver 2.6.9
    I, mv /usr/bin/rsync /usr/bin/rsync.old to get the old installed version out of the way.
    run brew install rsync
    “rsync-3.1.0 already installed”
    crap, where did I install it?
    any help is apprcd

    Should I just install rsync 3.0.9?

    • http://blog.mihros.de RobertS

      Hi Jolly,

      the installation is on my system in the following folder:
      /usr/local/Cellar/rsync/3.1.0/bin/

      So I moved/backuped the rsync from the /usr/bin/ like you and then I made a symlink to the installation binary.
      sudo ln -s /usr/local/Cellar/rsync/3.1.0/bin/rsync /usr/bin/rsync

      Check with the following command:
      rsync –version

    • Christopher Thomas

      a better idea might have been to edit your .bash_profile and put “export PATH=/usr/local/….etc..:${PATH}” in there so when you open a terminal, it’ll find your brew stuff first, but without messing about with your base filesystem, thats what I just did and works fine and keeps the default stuff intact

    • Jolly D’Bugger

      Thanks RobertS
      Successfully installed rsync 3.0.9, but still couldn’t get the “–info=progress2″ option (as illustrated about) to work or find any documentation on it. Just used –progress instead, and it works but i have to watch the output line for line.
      Where can I read up on –info ? if it’s in 3.0.9.

  • MrPerfect

    This is a great article. Thank you one million times. You sped up a script of mine by a LOT. :)

  • Andrew Zimmerman

    Thanks for the article.