How to speed up rsync on your Mac (and stop the dreaded “building file list…”)

UPDATED (2016-01-31): Mac OS X 10.11 El Capitan still uses rsync 2 so as of early 2016, this tutorial is still required to speed up rsync. Check your version via,

rsync --version
If the output says something like, rsync version 3.1.2, then great, but for most of you, without manually compiling an rsync binary or installing rsync via Homebrew, you’ll have to follow this tutorial…

UPDATED (2013-08-06): Added notes about requiring Xcode to compile rsync (if compiling is the route you choose).

Quick Story

I have a 3TB hard drive full of my RAW (.CR2) photography files that I replicate to another drive every week or so.

For this, I usually run a simple rsync command as such,

rsync -av /Volumes/Photography/ /Volumes/PhotographyBackup/

Screen Shot 2013-08-05 at 3.14.55 PM

Easy enough, but the rsync command sat on “building file list…” for hours and hours and hours because 3TB and hundreds of thousands of files is a large “file list” for rsync to create.

A quick note on rsync flags

Do not use the -z flag for local hard drive to hard drive data transfers as -z is the “compress” flag and can bring your computer to a crawl when you don’t need to use it in the first place. This issue comes from copying and pasting commands without knowing what the flags were doing, so I wanted to point it out.

You can add –delete to remove make sure files deleted on the SOURCE drive are deleted on the DESTINATION drive. e.g. If I delete /Volumes/Photography/old_folder/, it won’t delete it off of /Volumes/PhotographyBackup/ without using –delete.

rsync -av --delete /Volumes/Photography/ /Volumes/PhotographyBackup/

You can also view statistics of the transfer when it’s complete.

rsync -av --stats /Volumes/Photography/ /Volumes/PhotographyBackup/

rsync Out-of-Date on Mac OS X

In Apple’s Mac OS X 10.8 Mountain Lion, 10.9 Mavericks and 10.10 Yosemite, rsync is at version 2.6while rsync has had 3.0+ released for some time now. This is bad for performance reasons. rsync 3.0+ has a reduced memory footprint and creates a chunked (bit-by-bit) “file list”, rather than waiting for the entire list to be made first and then beginning transfer. This means moving to rsync 3.0+ will dramatically speed up the time before rsync begins its first transfer, e.g. fix the “building file list…” hang.

They state this explicitly in rsync 3.0+ by saying “sending incremental file list” rather than rsync 2.6’s “building file list…”.

Quick Note on Linux

I looked up Debians’s & Ubuntu’s, Fedora’s and SuSE’s package version for rsync and luckily they have been using 3.0+ for some time, so if you’re on a modern Linux distro, you don’t need to follow these instructions. If you’re on a Linux server or an old Linux desktop/laptop install, you may want to check for yourself using,
rsync --version

Installing rsync 3.0+ on Mac OS X 10.8, 10.9, 10.10 & 10.11

Do not run all of these commands, you have a choice of which one you prefer.

A) Use Homebrew

If you use Homebrew, you can run this command,

brew tap homebrew/dupes
brew install rsync

That’s it, you’re done. You can start using rsync 3.

B) Download & Compile

i) Download the Stable Version

Please visit the stable folder to find the most recent stable version as the version numbers may have changed.

cd ~/Downloads/
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-3.1.2.tar.gz
tar -xzvf rsync-3.1.2.tar.gz
cd rsync-3.1.2

Or…

ii) Download the (in development) Nightly Version

cd ~/Downloads/
curl -O https://rsync.samba.org/ftp/rsync/dev/nightly/rsync-HEAD.tar.gz
tar -xzvf rsync-HEAD.tar.gz
cd rsync-HEAD*

Then, Compile…

./configure
make
sudo make install
chmod +x ./rsync

Possible Errors

If you receive,

configure.sh: error: no acceptable C compiler found in $PATH

or,

-bash: make: command not found

It is because you don’t have the proper compiler tools installed. Apple requires Xcode to be downloaded and installed from the App Store and have the “Command Line Tools” installed to compile software.

Once the app has been downloaded and installed, to install “Command Line Tools” go to “Xcode” -> “Preferences” -> “Download” and install “Command Line Tools” there or use the command-line,

xcode-select --install

Running the Command

Staying inside the folder it was compiled, you can now run your rsync command similar to before,

./rsync -av /from/this/dir/ /to/this/dir/

Or because rsync 3.0+ has a full-transfer progress indicator, rather than a file-by-file progress indicator, feel free to use –info=progress2 as an option.

./rsync -a --info=progress2 /from/this/dir/ /to/this/dir/

Screen Shot 2013-08-05 at 4.05.33 PM

  • Roger

    Apple intensely dislike GPL3. The result is that Apple users are stuck with very old versions of tools like rsync (v3 changed license to GPL3), bash, gcc, samba (now removed) etc. Here is a random article about it http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/

    Whatever your opinion about Apple and GPL3, it is negatively affecting Apple users.

    • brettalton

      I’m not a big fan of copyleft licenses like the GPL, but as much as I don’t agree with Apple’s aberrant choices, I don’t think it’s affecting too many users, especially because there are easy alternatives like Homebrew or compiling you own software. If Apple stopped you from compiling your own software, _then_ we’d have a problem.

    • IanK

      It’s not that apple dislikes GPL3 it’s that GPL3 was deliberately setup to stop companies like apple using the software unless they ship the source code to their entire system. Initially this was done to prevent Microsoft from using the results of GPL software but it also hits companies like apple too. That’s the reason Samba was never updated, it’s also the same for rsync which is a Samba product. Instead of sticking with an increasingly unreliable old version of samba apple chose to create their own SMB solution.

  • paales

    Easier installation: brew install rsync
    Delete old rsync from /usr/bin/rsync

    • http://infinite-sushi.com _ado

      But only if you did `brew tap homebrew/dupes`

    • numpty

      Homebrew or not, it’s really not recommended to ever delete anything from /usr/bin on OS X.

  • Pingback: A Photographers Guide to Backing Up Using RSYNC | Kuljit Athwal – Photographer()

  • billtheamerican

    Very good article on rsync. Here is one from around 1999 but still the very best rsync tutorial I’ve ever read. Explains it very well IMO: http://tinyurl.com/l37guv8

    • http://eatingrichly.com/ Eric J

      This is a spam link?

    • brettalton

      My apologies. I deleted it some time back, thanks for letting me know.

  • Pingback: rsync braucht unter Mac OS X ewig für “building file list…” « think eMeidi()

  • Pingback: Brew, rsync and command line errors . . . - Mac-Forums Discussions for Apple Products & Services()

  • Jolly D’Bugger

    Oh jezz, what did i do?
    I have homebrew installed and rsync 3.1.0
    rsync fails on …”–info=progress2″… and reports ver 2.6.9
    I, mv /usr/bin/rsync /usr/bin/rsync.old to get the old installed version out of the way.
    run brew install rsync
    “rsync-3.1.0 already installed”
    crap, where did I install it?
    any help is apprcd

    Should I just install rsync 3.0.9?

    • http://blog.mihros.de RobertS

      Hi Jolly,

      the installation is on my system in the following folder:
      /usr/local/Cellar/rsync/3.1.0/bin/

      So I moved/backuped the rsync from the /usr/bin/ like you and then I made a symlink to the installation binary.
      sudo ln -s /usr/local/Cellar/rsync/3.1.0/bin/rsync /usr/bin/rsync

      Check with the following command:
      rsync –version

    • Christopher Thomas

      a better idea might have been to edit your .bash_profile and put “export PATH=/usr/local/….etc..:${PATH}” in there so when you open a terminal, it’ll find your brew stuff first, but without messing about with your base filesystem, thats what I just did and works fine and keeps the default stuff intact

    • Jolly D’Bugger

      Thanks RobertS
      Successfully installed rsync 3.0.9, but still couldn’t get the “–info=progress2” option (as illustrated about) to work or find any documentation on it. Just used –progress instead, and it works but i have to watch the output line for line.
      Where can I read up on –info ? if it’s in 3.0.9.

  • MrPerfect

    This is a great article. Thank you one million times. You sped up a script of mine by a LOT. :)

  • Andrew Zimmerman

    Thanks for the article.

  • Tom Stratton

    If you want to save some complexity and headache you can get a precompiled binary of rsync 3 at http://rudix.org/packages/rsync.html

    • http://brettalton.com brett

      Rsync 3 has also been included in the latest Mac OSes, like OS X 10.11 El Capitan and 10.10 Yosemite.

  • Pingback: Alton Roberts Photography | Great Photos For You And Me()

  • mjpaci

    I just did a rebuild of my Mac – fresh install of 10.11.3 on a wiped disk and the version of rsync is 2.6.9.

    • baaaeh

      That’s very interesting to note. I didn’t do a fresh install of 10.11 and so I must have had a Homebrew compiled binary for rsync 3, when the vanilla binary is still 2.6.9. Thanks for the info.

  • IanK

    rsync 3 isn’t included by default in 10.11, you should remove that piece from the article as it’s very confusing.

    • baaaeh

      Edited for error. Thanks. It was noted below by another reader but I had yet to change the article.

  • http://www.linkedin.com/in/josemedeiros Jose F Medeiros

    Good article. What’s your opinion on using ArRsync for simple Mac & Linux users such as myself?

    • baaaeh

      I’ve never tried it! And I won’t have access to a laptop for another month (travelling) so you’ll have to wait on my review, but let me know how you like it!

  • Patric

    Thanks a lot! It really helped me out!