What is it doing?

[Brian K. White]

On 1/13/2014 5:34 PM, Perry Smith wrote:
> Ok.  I can get the Mac up to version 3 but I'm wondering if
> I need to rethink my whole strategy.  Since the source is on NFS,
> doing a stat on all the files each run may cost me too much
> time.
>
> I might need to split it into smaller pieces and then rotate
> through the pieces via a script.  Do you have any suggestions
> for this type of situation?

There is no avoiding at least a stat on every file no matter what tool 
you use other than say a dd of the whole underlying partition or drive.

One of the main differences rsync 3 will make is if both sides are 3.0+ 
then rsync will stat a small list of files and start comparing and 
transferring those right away, and work in an unlimited number of those 
chunks sequentially until the job is done, instead of 
parsing/expanding/building the entire job in memory before doing 
anything else.

So I don't think breaking up the job into pieces yourself will help any 
more than letting rsync 3 do it internally. It might help if you 
couldn't upgrade both sides to 3.

Definitely I would try to rsync directly between the nfs server and the 
other target instead of rsyncing from a point that accesses the files 
via nfs, if that's possible.

If that's not possible, at least I would hope that the rsync client and 
the nfs server were at least on a good lan together. Even 100Mb lan 
connection to nfs should be good enough to keep your wan connection 
saturated.

Or better yet, test the difference nfs makes. Construct a reasonable 
test job right on the Mac, copy the same files to nfs, and try the same 
exact job both ways:
1) rsync client on the Mac, source from the Macs local drive
2) rsync client on the Mac, source from the nfs mount
You may be nowhere near saturating your wan either way so the point is 
to see if nfs is actually hurting very much relative to pure rsync, and 
then don't worry about it if it's not.

And you can always add --progress and more -vvv's to see what it's 
working on while it's busy.

-- 
bkw