rsync limit to file size/file count

jw schultz jw at pegasys.ws
Thu Jun 12 07:59:35 EST 2003


On Wed, Jun 11, 2003 at 12:28:47PM -0700, Eric Chen wrote:
> I left the rsync running since last night, and it is still running
> 
> receiving file list ...
> expand file_list to 8000 bytes, did move
> expand file_list to 16000 bytes, did move
> expand file_list to 32000 bytes, did move
> expand file_list to 64000 bytes, did move
> expand file_list to 128000 bytes, did move
> expand file_list to 256000 bytes, did move
> expand file_list to 512000 bytes, did move
> expand file_list to 1024000 bytes, did move
> expand file_list to 2048000 bytes, did move
> expand file_list to 4096000 bytes, did move
> expand file_list to 8192000 bytes, did move
> 
> will this file-list ever be processed? what does the "expand file_list to X
> bytes, did move" supposed to mean?

The file list will be processed when it is complete.  This
is one of rsync's biggest weaknesses it has to build the
whole file list, complete with stat(2) info before it even
starts processing.  While not a significant speed impediment
it is often a problem in terms of memory footprint.

Surprisingly, it means exactly what it says.  The file list
was expanded to X bytes and in these cases that meant moving
it.  The file list is an array of pointers to file structs.
When it runs out of space it realloc is used to expand it.
"did move" is reported if realloc expands it to new space.

You have more than 1 million files,  It shouldn't have taken
overnight to get to that point so i hope this is truncated.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt



More information about the rsync mailing list