<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
Thanks friends. We are using Redhat Linux 5.8 on Production and Disaster Recovery side. By drilling down we have found out it is taking lot of time to check what has changed while data tranfer is very fast. As i mentioned data in these folders is very less (hardly 40GB) and whenever new file is created, it is of max 30KB. <BR>
<BR>
Since we have to sync production environment to DR every 10 mins as per Business requirement i have to schedule it via cron. This already distributed folder structure i am using. I already have another rsync job which runs every 5 mins on another folder structure. It is running fine. Is there any option i can use with rsync to make this folder check fast?<BR>
<BR>
Regards,<BR>
Vijay<BR>
<BR><BR> <BR>
<DIV>
<DIV id=SkyDrivePlaceholder></DIV>> From: Matthew.Stier@us.fujitsu.com<BR>> To: kmk@sanitarium.net; rsync@lists.samba.org<BR>> Subject: RE: Rsync takes long time to finish<BR>> Date: Thu, 12 Apr 2012 19:29:03 +0000<BR>> <BR>> The first clause should read "does not parallelize".<BR>> <BR>> <BR>> -----Original Message-----<BR>> From: rsync-bounces@lists.samba.org [mailto:rsync-bounces@lists.samba.org] On Behalf Of Stier, Matthew<BR>> Sent: Thursday, April 12, 2012 3:07 PM<BR>> To: Kevin Korb; rsync@lists.samba.org<BR>> Subject: RE: Rsync takes long time to finish<BR>> <BR>> And, although rsync does parallelize, nothing stops you from running multiple instances of rsync.<BR>> <BR>> I had to transfer files from system A to system B, and being limited by the processing power of a single thread of rsync, I drilled down one level, and ran rsync's against each the first level file and subdirectory. This put more threads/cores/processors to work made better use of the network bandwidth to get the job done.<BR>> <BR>> When all the rsync's finished, I ran a single root level rsync to catch the stragglers.<BR>> <BR>> If you have the processing power, use it.<BR>> <BR>> <BR>> -----Original Message-----<BR>> From: rsync-bounces@lists.samba.org [mailto:rsync-bounces@lists.samba.org] On Behalf Of Kevin Korb<BR>> Sent: Thursday, April 12, 2012 2:44 PM<BR>> To: rsync@lists.samba.org<BR>> Subject: Re: Rsync takes long time to finish<BR>> <BR>> -----BEGIN PGP SIGNED MESSAGE-----<BR>> Hash: SHA1<BR>> <BR>> Several suggestions...<BR>> <BR>> Add a lockfile to your cron job so it doesn't run two instances at the<BR>> same time and you don't have to predict the run time.<BR>> <BR>> Make sure you are running rsync version 3+ on both systems. It has<BR>> significant performance benefits over version 2.<BR>> <BR>> Run a job manually and add --itemize-changes and --progress. Try to<BR>> figure out where most of the time is spent. Looking for something to<BR>> transfer, transferring new files, or updating changed files.<BR>> <BR>> If it is mostly looking for something to transfer then you need<BR>> filesystem optimizations. Such as directory indexing. You didn't<BR>> specify the OS or anything but if you are on Linux this is where an<BR>> ext3 > ext4 conversion would be helpful.<BR>> <BR>> If it is mostly transferring new files then look at the network<BR>> transfer rate. If it is low then try optimizing the ssh portion. Try<BR>> using -e 'ssh -c arcfour' or try using the hpn version of openssh. If<BR>> encryption isn't important you could also setup rsyncd.<BR>> <BR>> If it is mostly updating existing files check the itemize output to<BR>> see if the files really need updating. For instance if something is<BR>> screwing with your timestamps that will create a bunch of extra work<BR>> for rsync. Also, --inplace might help performance but be sure to read<BR>> about it.<BR>> <BR>> On 04/12/12 14:29, vijay patel wrote:<BR>> > Hi Friends,<BR>> > <BR>> > I am using rsync to copy data from Production File Server to<BR>> > Disaster Recovery file server. I have 100Mbps link setup between<BR>> > these two servers. Folder structure is very deep. It is having path<BR>> > like /reports/folder1/date/folder2/file.tx, where we have 1600<BR>> > directories like 'folder1', daily folders since last year in date<BR>> > folder and 2 folders for each date folder like folder2 which<BR>> > ultimately will contain the file. Files are not too big but just<BR>> > design of folder structure is complex. Folder structure design is<BR>> > done by application and we can't change it at the moment. I am<BR>> > using following command in cron to run rsync.<BR>> > <BR>> > rsync -avh --delete --exclude-from 'ex_file.txt' /reports/ <BR>> > 10.10.10.100:/reports/ | tee /tmp/rsync_report.out >> <BR>> > /tmp/rsync_report.out.$today<BR>> > <BR>> > Initially we were running it every 5 mins then we increased it to<BR>> > every 30 mins since one instance was not getting finished in 5<BR>> > mins. Now we have made it to run every 8 hours because of lots of<BR>> > folders. Is there a way i can improve performance of my rsync??<BR>> > <BR>> > <BR>> > Regards, Vijay<BR>> > <BR>> > <BR>> > <BR>> <BR>> - -- <BR>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<BR>> Kevin Korb Phone: (407) 252-6853<BR>> Systems Administrator Internet:<BR>> FutureQuest, Inc. Kevin@FutureQuest.net (work)<BR>> Orlando, Florida kmk@sanitarium.net (personal)<BR>> Web page: http://www.sanitarium.net/<BR>> PGP public key available on web site.<BR>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<BR>> -----BEGIN PGP SIGNATURE-----<BR>> Version: GnuPG v2.0.17 (GNU/Linux)<BR>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/<BR>> <BR>> iEYEARECAAYFAk+HIoMACgkQVKC1jlbQAQddkACeOljjKSj/NVpc4dj6+Hjm946j<BR>> 9IsAoPNV4DrbTtH5Yj8Zk7p/2O8JacE3<BR>> =LsDJ<BR>> -----END PGP SIGNATURE-----<BR>> -- <BR>> Please use reply-all for most replies to avoid omitting the mailing list.<BR>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync<BR>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html<BR>> -- <BR>> Please use reply-all for most replies to avoid omitting the mailing list.<BR>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync<BR>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html<BR>> -- <BR>> Please use reply-all for most replies to avoid omitting the mailing list.<BR>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync<BR>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html<BR></DIV> </div></body>
</html>