using rsync 3.0.0 CVS version

[Paul Slootman]

On Tue 30 Jan 2007, Wayne Davison wrote:
> On Tue, Jan 30, 2007 at 02:38:42PM +0100, Paul Slootman wrote:
> > It also did the sync in under 2 hours, whereas the non-incremental
> > version would take up to 12 hours, and use up to 500MB of memory.
> 
> Awesome!  That's an even more dramatic improvement than what I was
> anticipating for large hierarchies.  Thanks for your help testing

It now only has to keep 1% of the total list of files in memory (there
are 100 subdirectories which don't have any further subdirs).

However, I may have exaggerated the 12 hours, according to the logs it
used to be around 5 hours, so the improvement is still pretty good, and
the memory usage reduction is very useful as well.

Here's some output from a log that's generated;
the "File list generation time" and "File list transfer time" is perhaps
less useful nowadays :)

20070130-113432 doing images-17

Number of files: 3766788
Number of files transferred: 194
Total file size: 31635945072 bytes
Total transferred file size: 2453786 bytes
Literal data: 633820 bytes
Matched data: 1819966 bytes
File list size: 58092428
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 22425
Total bytes received: 58745190

sent 22425 bytes  received 58745190 bytes  9103.50 bytes/sec
total size is 31635945072  speedup is 538.32

20070130-132207 doing images-18

Number of files: 3843863
Number of files transferred: 74
Total file size: 32515860221 bytes
Total transferred file size: 1017500 bytes
Literal data: 381837 bytes
Matched data: 635663 bytes
File list size: 59415970
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 9884
Total bytes received: 59804946

sent 9884 bytes  received 59804946 bytes  9102.16 bytes/sec
total size is 32515860221  speedup is 543.61

20070130-151138 doing ...



It used to be like this:

20070105-035712 doing images-16

Number of files: 4338185
Number of files transferred: 472
Total file size: 36888804858 bytes
Total transferred file size: 3467730 bytes
Literal data: 2254171 bytes
Matched data: 1213559 bytes
File list size: 71031147
File list generation time: 5601.976 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33097
Total bytes received: 73313978

sent 33097 bytes  received 73313978 bytes  4857.90 bytes/sec
total size is 36888804858  speedup is 502.93

20070105-080853 doing images-17

Number of files: 3766442
Number of files transferred: 682
Total file size: 31632304810 bytes
Total transferred file size: 5235606 bytes
Literal data: 3479744 bytes
Matched data: 1755862 bytes
File list size: 61852761
File list generation time: 6170.866 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 52753
Total bytes received: 65373727

sent 52753 bytes  received 65373727 bytes  3546.05 bytes/sec
total size is 31632304810  speedup is 483.48

20070105-131639 ...


> this, and please keep us apprised of how things are working out.

Still going strong :)


Paul Slootman