Rsync takes long time to finish

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yes, you have the feature in your filesystem.  Good.  If it is ext3
then converting it to ext4 would still help assuming your distro
supports it.

You are using rsync over ssh.  This is my preference as well for
security reasons.  Using rsyncd would be faster because it would
remove the encryption overhead but that shouldn't be a big deal on
only 100Mbits.  It would make no difference in the indexing.

Have you checked your version yet?  Run rsync --version on both
systems.  If it isn't 3.0.something upgrade.  That will make a big
difference.

On 04/12/12 17:16, vijay patel wrote:
> I am getting following thing in 'tune2fs -l' :
> 
> Filesystem features:      has_journal resize_inode dir_index
> filetype needs_recovery sparse_super large_file
> 
> Does this mean it is set? One more thing i am not using rsync as
> daemon (Because i am confused with its usage at the moment), will
> it make any difference?
> 
> 
>> Date: Thu, 12 Apr 2012 17:05:06 -0400 From: kmk at sanitarium.net 
>> To: rsync at lists.samba.org Subject: Re: Rsync takes long time to
>> finish
>> 
> And make sure both systems are running rsync v3. It indexes in 
> parallel to the copying.
> 
> On 04/12/12 16:59, Karl O. Pinc wrote:
>> On 04/12/2012 03:28:18 PM, vijay patel wrote:
> 
>>> Thanks friends. We are using Redhat Linux 5.8 on Production
>>> and Disaster Recovery side. By drilling down we have found out
>>> it is taking lot of time to check what has changed while data 
>>> tranfer is very fast. As i mentioned data in these folders is 
>>> very less (hardly 40GB) and whenever new file is created, it
>>> is of max 30KB.
> 
>>> Since we have to sync production environment to DR every 10
>>> mins as per Business requirement i have to schedule it via
>>> cron. This already distributed folder structure i am using. I
>>> already have another rsync job which runs every 5 mins on
>>> another folder structure. It is running fine. Is there any
>>> option i can use with rsync to make this folder check fast?
> 
>> No. Per the response below you need to look at your filesystems.
> 
>> Use "tune2fs -l" and see if the dir_index option is on. If not, 
>> then turn it on using tune2fs. This probably won't fix the 
>> existing directories. If this is the problem you'll have to do a 
>> backup/restore, or a move of all the files into a new directory 
>> hierarchy and then replace the old hierarchy, or something else
>> to fix all the existing directories. (I don't think e2fsck will
>> help, but I've not looked. As I say, there may also be some
>> other approach.)
> 
>>>> If it is mostly looking for something to transfer then you 
>>>> need filesystem optimizations. Such as directory indexing.
>>>> You didn't specify the OS or anything but if you are on Linux
>>>> this is where an ext3 > ext4 conversion would be helpful.
> 
> 
>> Karl <kop at meme.com> Free Software: "You don't pay back, you pay 
>> forward." -- Robert A. Heinlein
> 
> 
>> -- Please use reply-all for most replies to avoid omitting the
>> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
>> http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+HRvMACgkQVKC1jlbQAQfSRgCg/unUPvt3pX+fbQf7qCQktWQc
kJoAn3ENigLu05Molf5iijT4VhJ1OoVU
=gQ9y
-----END PGP SIGNATURE-----