[clug] rsync question

[Brett Worth]

On 22/06/13 20:03, Eyal Lebedinsky wrote:
> I am moving data between two arrays. I use this command
>     rsync -aSv --progress rsync://root@e7/root/data1/ /data1
> to pull the data from 'e7' to a local disk (both arrays mounted on /data1 and
> 'root' is a module that exports e7:/, created just for this purpose).
>
> Not all the data seems to arrive. The copy did not complete but one of the large
> subdirectories shows less files in the target (71k) than the source (82k).

That's a worry.  If your receiving rsync didn't error then there must have been an error 
on the rsyncd.  It's probably logging to syslog.

If the target ran out of inodes you should have had errors.

> Also, am I correct that rsync does *not* encrypt the data, and since the
> two arrays can do ~500MB/s I should expect the wire to be the limiting
> factor? The only CPU work is the checksum calcs.

I have never used ext4 on a filesystem larger than 16TB.  Under CentOS there's been some 
issue with the extutils not being up to the version that can handle big filesystems even 
though ext4 can.

Another thing I've used for a speedup is to run several rsyncs in parallel after carving 
up the subdirs into sort of even  chunks. That's when I've had to use encryption though it 
may help with the checksumming.

Brett


-- 
   /) _ _ _/_/ / / /  _ _//
  /_)/</= / / (_(_/()/< ///