Maximum file size with rsync?

[Steve Bonds]

On Tue, 12 Aug 2003, Jeff Frost J.Frost-at-fantastic.com |Rsync List| wrote:

> I was thinking to use the whole-file option instead of letting it
> attempt to chew through the files for diffs, as I had problems with
> rsync dealing with files in the 12-15gb range in the past, though it
> worked great when using --whole-file.  Would there be a more efficient
> program to use for just moving the bits over there?

I suspect that just about any program is going to be network limited on a
large transfer like this.  A couple things to consider:

1) If you encrypt your data by using ssh as the rsync transport, it can no
longer be compressed.  If you are going over any WAN links that use
transparent compression, this can really reduce your throughput

2) If your CPUs are much faster than your network, using "--compress".
Usually it just makes things slower, but it's worth trying if you use ssh
as your transport, since you'll then be encrypting less data, which can be
faster.

3) If you really want speed and don't care a whit about security, look at
"netcat".  This avoids all that slow protocol overhead but doesn't do
things like preserve permissions, filenames, etc.

4) Benchmark the transfer compared with a basic FTP or rcp.

  -- Steve