Transfering very large files / and restarting failures

[Wayne Davison]

On Wed, Jul 27, 2005 at 01:50:39PM -0700, Todd Papaioannou wrote:
> where both theFile and /path/to/dest are local drives. [...]
> rsync -u --no-whole-file --progress theFile /path/to/dest

When using local drives, the rsync protocol (--no-whole-file) slows
things down, so you don't want to use it (the rsync protocol's purpose
is to trade disk I/O and CPU cycles to reduce network bandwidth, so it
doesn't help when the transfer bandwidth is very high, as it is in a
local copy).

Note also that you're not preserving the file times, which makes rsync
less efficient (which forces you to use the -u option to avoid a
retransfer) -- you're usually better off using -t (--times) unless you
have some overriding reason to omit it.

> However, the stats shown during the progress seem to imply that the
> whole transfer is starting again. 

Yes, that's what rsync does.   It retransfers the whole file, but it
uses the local data to make the amount of data flowing over the socket
(or pipe) smaller.  The already-sent data is thus coming from the
original, partially-transferred file rather than coming from the
sender (which would lower the network bandwidth if this were a
remote connection).

> In the future /path/to/dest will be an NFS mount.

You don't want to do that unless you're network speed is higher than
your disk speed -- with slower net speeds you are better off rsyncing
directly to the remote machine that is the source of the NFS mount so
that rsync can reduce the amount of data it is sending.  With higher net
speeds you're better off just transferring the data via --whole-file and
not using --partial.  One other possibility is the --append option from
the patch named patches/append.diff -- this implements a more efficient
append mode for incremental transfers (I'm considering adding this to
the next version of rsync).

..wayne..