Max number of files to transfer

[Adrian Cooke]

Wayne Davison wrote:
> On Thu, Jul 06, 2006 at 01:59:54PM -0400, Bob Bagwill wrote:
>> You probably already considered this, but you could also do something
>> like this: [...]
> 
> That divides the entire file-list up into 50-file chunks, whereas I
> believe that the original poster wanted to allow 50 differing files to
> be updated before quitting (out of 50+N files checked for being
> up-to-date).

Yes this is correct. The link that is being used is the limiting factor 
and the time that the link is available is also a limiting factor. So I 
want to be able to quickly startup rsync(by limiting the file scanning) 
and then transfer those in one go, and not create and terminate rsync 
sessions as I go.

> If the file-system scan is not the limiting factor that is slowing
> things down, then running rsync with --dry-run could be used to output a
> list of the files that need to be updated, and then that list could be
> divided into groups of 50 and iterated over.
> 
> ..wayne..
I dont think this wont really work because it will take to long to run 
because the link is to slow. This is because with --dry-run rsync still 
has to talk to the remote host if I understand it correctly and I do not 
want this.

For the remote to local transfer the problem should be solved by using a 
command to be passed to the remote shell as described in 
http://rsync.samba.org/examples.html "Fancy footwork with remote file 
lists". For the local to remote transfer I will take a look at the src 
or will just keep two directories and transfer the correct amount of 
files into the directory that I am going to rsync before sending.

Once the files are placed in the directory they will not change. In a 
previous post Matt McCutchen said that I can use --append to improve 
this start up time.

In the man page it says that this assumes that if the destination file 
is shorter than the source file it will then assume that the rest of the 
file must simply be appended to the file in the destination. Will this 
always be the case here because the source file never changes? It says 
that it implies --inplace but I need to used --partial-dir, will this 
cause a conflict?

Cheers,

Adrian