rsync 2 million files

[Martin Strand]

On Tue, 12 Feb 2008 10:52:00 +0100, Giuliano Gavazzi <dev+lists at humph.com>  
wrote:

>
> On 12Feb, 2008, at 09:53 , Martin Strand wrote:
>
>> I'm trying to use rsync to get a live backup of 2 million files, about  
>> 50 gb, max depth of 5 directory levels.
>> I'm on a gigabit lan so I'm passing -W, but it's still incredibly slow.  
>> What else can I do to speed things up?
>> Perhaps there's a good way to filter out files older than X so only  
>> newer files are checked?.
>> Will rsync 3.0.0 make a big difference for large trees?
>
> I say yes, although you did not quantify your "slow".
> The following was not on a gigabit, but on a firewire (tcp/ip over  
> firewire), and I suspect that it would have worked better on a gigabit  
> (better drivers). It was reported some time ago using rsync-3.0.0pre7:
>
> a new tranfer of about 30GB worth of data in about 2 hours:
>
>
> Number of files: 407317
>
> Total file size: 34860439736 bytes
> Total transferred file size: 34860261591 bytes
> Literal data: 34860261591 bytes
> Matched data: 0 bytes
> File list size: 10391700
> File list generation time: 0.003 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 35084947192
> Total bytes received: 7332354
>
> sent 35084947192 bytes  received 7332354 bytes  4927652.82 bytes/sec
>
> A subsequent re-rsync took about 5 minutes (little data changes).
>
> I would say that you can look at about the same sort of figures, or even  
> a bit less, with better hardware. This was after all just a PowerBook G4  
> backed up to an iMac Intel.
>
> Giuliano

I just tried it with rsync 3.0.0pre8 and it was significantly faster.
Thanks!