Speeding Up Rsync for Large File Sets

Steven Hartland killing at multiplay.co.uk
Fri Nov 30 17:09:51 MST 2012


zfs send receive should work perfectly for this.

If the destination has changes you don't mind loosing e.g. just permission
changes you can have the receive roll back to the last known source snapshot
before replaying changes. The result being two identical copies of everything.

If there are other changes you also want you could maybe write a script
combining zfs diff with rsync to only send the known changed files.

    Regards
    Steve

----- Original Message ----- 
From: "Tim Gustafson" <tjg at soe.ucsc.edu>
To: <rsync at lists.samba.org>
Sent: Friday, November 30, 2012 5:33 PM
Subject: Speeding Up Rsync for Large File Sets


> We have a particular file system that we're trying to keep in sync
> between two FreeBSD/ZFS servers using Rsync.
> 
> The file system has many millions of files, and about 4TB of data
> total.  Rsync takes HOURS to run, even when there are no files to
> transfer.  Just the comparison itself takes hours.
> 
> Is there any way to speed up the transfer?  The command line I'm using is:
> 
> /usr/local/bin/rsync --stats --rsh=/usr/bin/rsh --recursive --delete
> --one-file-system --xattrs --links --hard-links --times --perms
> --owner --group --exclude=.zfs /foo/bar root at remote:/foo/bar


================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.



More information about the rsync mailing list