Articles

Archive for September, 2010

Performance Issues and rsync

Posted at 7:30 am by brett on September 10th, 2010

Under categories: Computers,Ubuntu

I'm upgrading my external hard drive from 1.5 TB to 2.0 TB. Not a big jump, no, but it's a good opportunity to throw the 1.5 TB in the closet for backup/storage.

My 1.5 TB external was hooked up to my Dell Mini 9 netbook and used as a media server as I found it far too small (dimensions-wise) to be used for programming or school. Since the old external was already attached, I threw on the new external, formatted it to ext4 using gparted and started transferring the data.

Here's the command I used:
sudo rsync -avrz /media/external/ /media/external_/

D'oh! Can anyone catch what I did?

My Dell Mini 9, equipped with a less-than-powerful Intel Atom N270 was gobbled up by the command, almost making the netbook unusable.

You see, I'm used to using the tar command tar -xvzf filename.tar.gz which extracts an archive of files.

For tar, '-z' means

-z, --gzip, --gunzip --ungzip

meaning that the file is compressed with gzip.

For rsync however, '-z' means

-z, --compress compress file data during the transfer

Yikes! My little Intel Atom N270 was trying to compress all my files before transferring them! Not good nor efficient for local transfer.

If you're transferring MB or GB of data over a network (and have the time and CPU power to compress files), then by all means, use the '-z' with rsync. But don't use the '-z' flag when copying data from one local hard drive to another, especially when using a netbook. That's reserved for people by the name of Brett Alton.

So i switched the transfer over to my new latop, sporting a Core i5 430m, transferring 1.5 TB of data without sweating

CPU breakdown
CPU CPU-usage Cores/Threads Speed Cache (L2/L3) rsync flags
Intel Atom N270 97% 1/2 1.6 GHz 512 kB / none -avrz
Intel Core i5 430m 3% 2/4 2.26 GHz 2x256 kB / 3 MB -av

Here is the suggested command to use (changing the path names of course):
sudo rsync -av /media/external/ /media/external_/

Good luck!