rsync limit to file size/file count

Eric Chen echen at ateonix.com
Thu Jun 12 05:28:47 EST 2003


I left the rsync running since last night, and it is still running

receiving file list ...
expand file_list to 8000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 8192000 bytes, did move

will this file-list ever be processed? what does the "expand file_list to X
bytes, did move" supposed to mean?

-----Original Message-----
From: rsync-bounces+echen=ateonix.com at lists.samba.org
[mailto:rsync-bounces+echen=ateonix.com at lists.samba.org]On Behalf Of jw
schultz
Sent: Tuesday, June 10, 2003 5:30 PM
To: Rsync List
Subject: Re: rsync limit to file size/file count


On Tue, Jun 10, 2003 at 04:46:25PM -0700, Eric Chen wrote:
> Hi,
>
> What are the limits to file size and file count when doing a rsync
transfer
> using 2.5.6?

File sizes somewhere between 300MB and 1GB 2.5.6 begins
having noticeable issues.  This is not a limit but merely
pushing the capacity of truncated digests. This problem
impacts performance only. It can be mitigated somewhat by
using a larger block size.  These issues have been fixed in
CVS so 2.5.7 and later will be fine with huge files.

> I was trying to rsync about 500 GB of data with many files and many
> directories, but it has been stuck building the file list for several
hours.
> First of all, is it possible to transfer 500 GB of data? Secondly, what

Yes.

> would the limit for file count be when doing a rsync transfer?

There is no hard limit on file count you could hit.
Each file requires approximately 70 + path-length bytes so it
is a question of how much process addressable memory you have.

--
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html




More information about the rsync mailing list