Speeding Up Rsync for Large File Sets
kmk at sanitarium.net
Fri Nov 30 10:40:16 MST 2012
-----BEGIN PGP SIGNED MESSAGE-----
If you are using zfs then why not use zfs send and zfs receive?
Rsync has to stat every file on both ends which can take a long time
with millions of files. The zfs tools don't have to do any of that.
On 11/30/12 12:33, Tim Gustafson wrote:
> We have a particular file system that we're trying to keep in sync
> between two FreeBSD/ZFS servers using Rsync.
> The file system has many millions of files, and about 4TB of data
> total. Rsync takes HOURS to run, even when there are no files to
> transfer. Just the comparison itself takes hours.
> Is there any way to speed up the transfer? The command line I'm
> using is:
> /usr/local/bin/rsync --stats --rsh=/usr/bin/rsh --recursive
> --delete --one-file-system --xattrs --links --hard-links --times
> --perms --owner --group --exclude=.zfs /foo/bar
> root at remote:/foo/bar
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
-----END PGP SIGNATURE-----
More information about the rsync