rsync 2 million files
do.not.eat.yellow.snow at gmail.com
Tue Feb 12 14:49:25 GMT 2008
On Tue, 12 Feb 2008 10:52:00 +0100, Giuliano Gavazzi <dev+lists at humph.com>
> On 12Feb, 2008, at 09:53 , Martin Strand wrote:
>> I'm trying to use rsync to get a live backup of 2 million files, about
>> 50 gb, max depth of 5 directory levels.
>> I'm on a gigabit lan so I'm passing -W, but it's still incredibly slow.
>> What else can I do to speed things up?
>> Perhaps there's a good way to filter out files older than X so only
>> newer files are checked?.
>> Will rsync 3.0.0 make a big difference for large trees?
> I say yes, although you did not quantify your "slow".
> The following was not on a gigabit, but on a firewire (tcp/ip over
> firewire), and I suspect that it would have worked better on a gigabit
> (better drivers). It was reported some time ago using rsync-3.0.0pre7:
> a new tranfer of about 30GB worth of data in about 2 hours:
> Number of files: 407317
> Total file size: 34860439736 bytes
> Total transferred file size: 34860261591 bytes
> Literal data: 34860261591 bytes
> Matched data: 0 bytes
> File list size: 10391700
> File list generation time: 0.003 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 35084947192
> Total bytes received: 7332354
> sent 35084947192 bytes received 7332354 bytes 4927652.82 bytes/sec
> A subsequent re-rsync took about 5 minutes (little data changes).
> I would say that you can look at about the same sort of figures, or even
> a bit less, with better hardware. This was after all just a PowerBook G4
> backed up to an iMac Intel.
I just tried it with rsync 3.0.0pre8 and it was significantly faster.
More information about the rsync