[clug] rsync question

[Eyal Lebedinsky]

With the background RAID resync done I do get 80+MB/s transfers so this is as expected,
I will not be able to get much more out of this wire.

The main problem is the missing files. No errors were logged anywhere. I will try to xfer
just one of these dirs and see if it arrived complete.

Eyal

On 06/23/13 08:55, Brett Worth wrote:
> On 22/06/13 20:03, Eyal Lebedinsky wrote:
>> I am moving data between two arrays. I use this command
>>     rsync -aSv --progress rsync://root@e7/root/data1/ /data1
>> to pull the data from 'e7' to a local disk (both arrays mounted on /data1 and
>> 'root' is a module that exports e7:/, created just for this purpose).
>>
>> Not all the data seems to arrive. The copy did not complete but one of the large
>> subdirectories shows less files in the target (71k) than the source (82k).
>
> That's a worry.  If your receiving rsync didn't error then there must have been an error on the rsyncd.  It's probably logging to syslog.
>
> If the target ran out of inodes you should have had errors.
>
>> Also, am I correct that rsync does *not* encrypt the data, and since the
>> two arrays can do ~500MB/s I should expect the wire to be the limiting
>> factor? The only CPU work is the checksum calcs.
>
> I have never used ext4 on a filesystem larger than 16TB.  Under CentOS there's been some issue with the extutils not being up to the version that can handle big filesystems even though ext4 can.
>
> Another thing I've used for a speedup is to run several rsyncs in parallel after carving up the subdirs into sort of even  chunks. That's when I've had to use encryption though it may help with the checksumming.
>
> Brett
>
>

-- 
Eyal Lebedinsky	(eyal at eyal.emu.id.au)