Rsync takes long time to finish

[vijay patel]

yes both servers are having rsync 3.0.6.
 

> Date: Thu, 12 Apr 2012 17:19:47 -0400
> From: kmk at sanitarium.net
> To: rsync at lists.samba.org
> Subject: Re: Rsync takes long time to finish
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Yes, you have the feature in your filesystem. Good. If it is ext3
> then converting it to ext4 would still help assuming your distro
> supports it.
> 
> You are using rsync over ssh. This is my preference as well for
> security reasons. Using rsyncd would be faster because it would
> remove the encryption overhead but that shouldn't be a big deal on
> only 100Mbits. It would make no difference in the indexing.
> 
> Have you checked your version yet? Run rsync --version on both
> systems. If it isn't 3.0.something upgrade. That will make a big
> difference.
> 
> On 04/12/12 17:16, vijay patel wrote:
> > I am getting following thing in 'tune2fs -l' :
> > 
> > Filesystem features: has_journal resize_inode dir_index
> > filetype needs_recovery sparse_super large_file
> > 
> > Does this mean it is set? One more thing i am not using rsync as
> > daemon (Because i am confused with its usage at the moment), will
> > it make any difference?
> > 
> > 
> >> Date: Thu, 12 Apr 2012 17:05:06 -0400 From: kmk at sanitarium.net 
> >> To: rsync at lists.samba.org Subject: Re: Rsync takes long time to
> >> finish
> >> 
> > And make sure both systems are running rsync v3. It indexes in 
> > parallel to the copying.
> > 
> > On 04/12/12 16:59, Karl O. Pinc wrote:
> >> On 04/12/2012 03:28:18 PM, vijay patel wrote:
> > 
> >>> Thanks friends. We are using Redhat Linux 5.8 on Production
> >>> and Disaster Recovery side. By drilling down we have found out
> >>> it is taking lot of time to check what has changed while data 
> >>> tranfer is very fast. As i mentioned data in these folders is 
> >>> very less (hardly 40GB) and whenever new file is created, it
> >>> is of max 30KB.
> > 
> >>> Since we have to sync production environment to DR every 10
> >>> mins as per Business requirement i have to schedule it via
> >>> cron. This already distributed folder structure i am using. I
> >>> already have another rsync job which runs every 5 mins on
> >>> another folder structure. It is running fine. Is there any
> >>> option i can use with rsync to make this folder check fast?
> > 
> >> No. Per the response below you need to look at your filesystems.
> > 
> >> Use "tune2fs -l" and see if the dir_index option is on. If not, 
> >> then turn it on using tune2fs. This probably won't fix the 
> >> existing directories. If this is the problem you'll have to do a 
> >> backup/restore, or a move of all the files into a new directory 
> >> hierarchy and then replace the old hierarchy, or something else
> >> to fix all the existing directories. (I don't think e2fsck will
> >> help, but I've not looked. As I say, there may also be some
> >> other approach.)
> > 
> >>>> If it is mostly looking for something to transfer then you 
> >>>> need filesystem optimizations. Such as directory indexing.
> >>>> You didn't specify the OS or anything but if you are on Linux
> >>>> this is where an ext3 > ext4 conversion would be helpful.
> > 
> > 
> >> Karl <kop at meme.com> Free Software: "You don't pay back, you pay 
> >> forward." -- Robert A. Heinlein
> > 
> > 
> >> -- Please use reply-all for most replies to avoid omitting the
> >> mailing list. To unsubscribe or change options:
> > https://lists.samba.org/mailman/listinfo/rsync
> >> Before posting, read:
> >> http://www.catb.org/~esr/faqs/smart-questions.html
> > 
> > 
> 
> - -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk+HRvMACgkQVKC1jlbQAQfSRgCg/unUPvt3pX+fbQf7qCQktWQc
> kJoAn3ENigLu05Molf5iijT4VhJ1OoVU
> =gQ9y
> -----END PGP SIGNATURE-----
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20120413/295a91fa/attachment.html>