Count of files to be transferred

[Robert DuToit]

Hi All,

On Apr 30, 2010, at 5:29 PM, Matt McCutchen wrote:

> On Fri, 2010-04-30 at 14:21 +0700, Ryan Joseph wrote:
>> On Apr 30, 2010, at 12:54 PM, Matt McCutchen wrote:
>>> Rsync determines which of the files in the file list need a data
>>> transfer as it goes, so the only way to get the count in advance is to
>>> do a separate dry run.  Then you could take the "Number of files
>>> transferred" from --stats output.
>> 
>> What do you mean "dry run" and wouldn't that take as much time as the
>> real backup thus effectively doubling the time? If so that's probably
>> not a good option.
> 
> Rsync has a --dry-run option to run without actually making the changes
> to the destination.  The source and destination are scanned as normal,
> but transmission of the contents of files is skipped.  See the man page
> for more information.
> 
>>> Alternatively, you could base the progress bar on the total number of
>>> entries in the file list.  The --progress option does this, but it won't
>>> be accurate if incremental recursion is enabled.
>> 
>> I tried the --progress option but it only seemed to show the progress
>> for each file which really isn't helpful for my case. 
> 
> See the "to-check" values at the end of each progress line, for example:
> 
> $ rsync -r --progress src2/ dest2/
> sending incremental file list
> created directory dest2
> ./
> filea
>           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
> fileb
>           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=0/3)
> 
> sent 126 bytes  received 53 bytes  358.00 bytes/sec
> total size is 0  speedup is 0.00
> 
> Alternatively, if you use a development snapshot of rsync 3.1.0, the
> --info=progress2 option will give a size and percentage that refer to
> the entire set of source files, not the current file.
> 
>> How do I get the file list then? If I could read a list of files that
>> will be transferred (rsync says it builds this list so it must be
>> somewhere)

I don't know if this is what you are looking for but I have a GUI for rsync (Backuplist+) and now use unix "find directoryPath | wc -l" and it is very fast and gives you the total file count which reflects the rsync count in stats at the end of the run, almost exactly. I  then just use the total count and the "to check" number to easily get the # of files transferred of total, for updating a progress bar. You could get fancy and use "find' with options to reflect any options for rsync that may exclude certain files etc...  

This is on OSX but would apply anywhere I would think. For 1 million files it takes about 2.5 minutes for the initial "find" count, on my old Macbook Pro laptop, and then rsync goes about it's business. This is much faster then running rsync non-recursively.

FWIW... 

yours,  Rob 



> 
> What rsync is "building" is the "file list", which contains /all/ source
> files.  As I said, rsync determines as it goes which of those files need
> a data transfer.
> 
> -- 
> Matt
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html