Rsync takes long time to finish

[Karl O. Pinc]

On 04/12/2012 03:28:18 PM, vijay patel wrote:
> 
> Thanks friends.  We are using Redhat Linux 5.8 on Production and
> Disaster Recovery side.  By drilling down we have found out it is
> taking lot of time to check what has changed while data tranfer is
> very fast.  As i mentioned data in these folders is very less (hardly
> 40GB) and whenever new file is created, it is of max 30KB. 
>  
> Since we have to sync production environment to DR every 10 mins as
> per Business requirement i have to schedule it via cron. This already
> distributed folder structure i am using. I already have another rsync
> job which runs every 5 mins on another folder structure. It is 
> running
> fine. Is there any option i can use with rsync to make this folder
> check fast?

No.  Per the response below you need to look at your filesystems.

Use "tune2fs -l" and see if the dir_index option is on.  If not,
then turn it on using tune2fs.  This probably won't fix the existing
directories.  If this is the problem you'll have to do a
backup/restore, or a move of all the files into a new directory
hierarchy and then replace the old hierarchy, or 
something else to fix all the existing directories.
(I don't think e2fsck will help, but I've not looked.  As I say,
there may also be some other approach.)

> > If it is mostly looking for something to transfer then you need
> > filesystem optimizations. Such as directory indexing. You didn't
> > specify the OS or anything but if you are on Linux this is where an
> > ext3 > ext4 conversion would be helpful.


Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein