Rsync takes long time to finish

Stier, Matthew Matthew.Stier at us.fujitsu.com
Thu Apr 12 13:29:03 MDT 2012


The first clause should read "does not parallelize".


-----Original Message-----
From: rsync-bounces at lists.samba.org [mailto:rsync-bounces at lists.samba.org] On Behalf Of Stier, Matthew
Sent: Thursday, April 12, 2012 3:07 PM
To: Kevin Korb; rsync at lists.samba.org
Subject: RE: Rsync takes long time to finish

And, although rsync does parallelize, nothing stops you from running multiple instances of rsync.

I had to transfer files from system A to system B, and being limited by the processing power of a single thread of rsync, I drilled down one level, and ran rsync's against each the first level file and subdirectory.  This put more threads/cores/processors to work made better use of the network bandwidth to get the job done.

When all the rsync's finished, I ran a single root level rsync to catch the stragglers.

If you have the processing power, use it.


-----Original Message-----
From: rsync-bounces at lists.samba.org [mailto:rsync-bounces at lists.samba.org] On Behalf Of Kevin Korb
Sent: Thursday, April 12, 2012 2:44 PM
To: rsync at lists.samba.org
Subject: Re: Rsync takes long time to finish

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Several suggestions...

Add a lockfile to your cron job so it doesn't run two instances at the
same time and you don't have to predict the run time.

Make sure you are running rsync version 3+ on both systems.  It has
significant performance benefits over version 2.

Run a job manually and add --itemize-changes and --progress.  Try to
figure out where most of the time is spent.  Looking for something to
transfer, transferring new files, or updating changed files.

If it is mostly looking for something to transfer then you need
filesystem optimizations.  Such as directory indexing.  You didn't
specify the OS or anything but if you are on Linux this is where an
ext3 > ext4 conversion would be helpful.

If it is mostly transferring new files then look at the network
transfer rate.  If it is low then try optimizing the ssh portion.  Try
using -e 'ssh -c arcfour' or try using the hpn version of openssh.  If
encryption isn't important you could also setup rsyncd.

If it is mostly updating existing files check the itemize output to
see if the files really need updating.  For instance if something is
screwing with your timestamps that will create a bunch of extra work
for rsync.  Also, --inplace might help performance but be sure to read
about it.

On 04/12/12 14:29, vijay patel wrote:
> Hi Friends,
> 
> I am using rsync to copy data from Production File Server to
> Disaster Recovery file server. I have 100Mbps link setup between
> these two servers. Folder structure is very deep. It is having path
> like /reports/folder1/date/folder2/file.tx, where we have 1600
> directories like 'folder1',  daily folders since last year in date
> folder and 2 folders for each date folder like folder2  which
> ultimately will contain the file. Files are not too big but just
> design of folder structure is complex. Folder structure design is
> done by application and we can't change it at the moment. I am
> using following command in cron to run rsync.
> 
> rsync -avh --delete --exclude-from 'ex_file.txt' /reports/ 
> 10.10.10.100:/reports/ | tee /tmp/rsync_report.out >> 
> /tmp/rsync_report.out.$today
> 
> Initially we were running it every 5 mins then we increased it to
> every 30 mins since one instance was not getting finished in 5
> mins. Now we have made it to run every 8 hours because of lots of
> folders. Is there a way i can improve performance of my rsync??
> 
> 
> Regards, Vijay
> 
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+HIoMACgkQVKC1jlbQAQddkACeOljjKSj/NVpc4dj6+Hjm946j
9IsAoPNV4DrbTtH5Yj8Zk7p/2O8JacE3
=LsDJ
-----END PGP SIGNATURE-----
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


More information about the rsync mailing list