rsync 2 million files

[Giuliano Gavazzi]

On 12Feb, 2008, at 09:53 , Martin Strand wrote:

> I'm trying to use rsync to get a live backup of 2 million files,  
> about 50 gb, max depth of 5 directory levels.
> I'm on a gigabit lan so I'm passing -W, but it's still incredibly  
> slow. What else can I do to speed things up?
> Perhaps there's a good way to filter out files older than X so only  
> newer files are checked?.
> Will rsync 3.0.0 make a big difference for large trees?

I say yes, although you did not quantify your "slow".
The following was not on a gigabit, but on a firewire (tcp/ip over  
firewire), and I suspect that it would have worked better on a gigabit  
(better drivers). It was reported some time ago using rsync-3.0.0pre7:

a new tranfer of about 30GB worth of data in about 2 hours:


Number of files: 407317

Total file size: 34860439736 bytes
Total transferred file size: 34860261591 bytes
Literal data: 34860261591 bytes
Matched data: 0 bytes
File list size: 10391700
File list generation time: 0.003 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 35084947192
Total bytes received: 7332354

sent 35084947192 bytes  received 7332354 bytes  4927652.82 bytes/sec

A subsequent re-rsync took about 5 minutes (little data changes).

I would say that you can look at about the same sort of figures, or  
even a bit less, with better hardware. This was after all just a  
PowerBook G4 backed up to an iMac Intel.

Giuliano