Long time needed for "Building file list" Any suggestions ?

[Clint Byrum]

On Mon, 2004-03-22 at 04:42, Hergaarden, Marcel wrote:
> We're running rsync 2.5.7 on a Windows2000 server, in combination with
> cygwin/ssh. The server who receives the data is a Linux server.
> 
> The amount of data from the Windows server is about 100 Gb. Represented
> by 532.000 files of different nature. Mostly doc, ppt and xls files.
> 
> It takes about 2 hours to create only the file list. 
> 
> Is the amount of data/files to big, should I segmentate the backupfiles
> or is something else the cause of this long duration.
> 

2 hours for 530,000+ files sounds about right to me. 2 things..

1) how much does the data change per day?
2) how fast is the network link between the two boxes?

We had a situtation recently where a backup via rsync that used to take
1 hour total, suddenly ballooned to 3.5 hours. This wasn't acceptable as
it was loading the server down. We had recently upgraded to gigabit
ethernet, so we were a bit perplexed. Then we realized that the number
of files being rsynced had gone up by a factor of 5.

We switched to just doing a simple tar backup of all the files. It only
takes an hour again. rsync is great (REALLY GREAT), but remember, its
mostly about maximizing bandwidth by only sending whats changed. If 80%
of your data changes every day, and you have a gigabit link... rsync
isn't really meant for that anyway.

This does bring up one point though. Is there any way to optimize file
list building? It seems like that turns into a huge bottleneck in the
"lots of files" situation.