<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18828">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT size=2 face=Arial>Hi,</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>In my situation I'm using rsync to backup a
server with (currently) about 570,000 files.</FONT></DIV>
<DIV><FONT size=2 face=Arial>These are all little files and maybe .1% of them
change or new ones are added in</FONT></DIV>
<DIV><FONT size=2 face=Arial>any 15 minute period.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>I've split the main tree up so rsync can run on sub
sub directories of the main tree. </FONT></DIV>
<DIV><FONT size=2 face=Arial>It does each </FONT><FONT size=2 face=Arial>of
these sub sub directories sequentially. I would have liked to run </FONT></DIV>
<DIV><FONT size=2 face=Arial>some of these </FONT><FONT size=2 face=Arial>in
parallel, but that seems to increase i/o on the main server too
much.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Today I tried the following:</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>For all subsub directories</FONT></DIV>
<DIV><FONT size=2 face=Arial> a) Fork a "du -s
subsubdirectory" on the destination subsubdirectory</FONT></DIV>
<DIV><FONT size=2 face=Arial> b) Run rsync on the
subsubdirectory</FONT></DIV>
<DIV><FONT size=2 face=Arial> c) repeat untill
done</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Seems to have improved the time it takes by about
25-30%. It looks like the du can</FONT></DIV>
<DIV><FONT size=2 face=Arial>run ahead of the rsync...so that while rsync is
building its file list, the du is warming up</FONT></DIV>
<DIV><FONT size=2 face=Arial>the file cache on the destination. Then when rsync
looks to see what it needs to do</FONT></DIV>
<DIV><FONT size=2 face=Arial>on the destination, it can do this more
efficiently.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Looks like a keeper so far. Any other suggestions?
(was thinking of a previous</FONT></DIV>
<DIV><FONT size=2 face=Arial>suggestion of setting <FONT
size=3>/proc/sys/vm/vfs_cache_pressure to a low value).</FONT></FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>Thanks,</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>Mike</FONT></DIV></BODY></HTML>