Aw: Re: strange behavior of --inplace on ZFS

[devzero at web.de]

Hi Pavel, 
could you try if --inplace --no-whole-file makes a difference?

normally, if using rsync --inplace on local host without network in betweens makes rsync switch to --whole-file, and  i assume, that for some reason your rsync is rewriting the whole file inplace. and any rewritten block on zfs means newly allocated block, which quickly eats up your space.

-W, --whole-file
              With this option rsyncâs delta-transfer algorithm is not used and the whole file  is  sent  as-is  instead.
              The  transfer  may  be  faster if this option is used when the bandwidth between the source and destination
              machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked  filesys-
              tem).   This  is the default when both the source and destination are specified as local paths, but only if
              no batch-writing option is in effect.

i came across this wile trying to recreate your issue - i tested this on localhost and while scratching my head abourt the weird growth i remembered that rsync is "too intelligent" when not run via network....

regards
Roland


> Gesendet: Freitag, 07. März 2014 um 00:50 Uhr
> Von: "Pavel Herrmann" <morpheus.ibis at gmail.com>
> An: "Hendrik Visage" <hvjunk at gmail.com>
> Cc: devzero at web.de, "rsync at lists.samba.org" <rsync at lists.samba.org>
> Betreff: Re: strange behavior of --inplace on ZFS
>
> Hi
> 
> apologies for reordering the message parts, and for the long bit at the end
> 
> On Thursday 06 March 2014 23:45:22 Hendrik Visage wrote:
> > Question: The source and destination folder host OS and are both sides ZFS?
> 
> no, the remote (source) was ext4. However, i plan to use it against at least 
> one NTFS machine as well
> 
> > I'd like to see some stats that rsync said it transfered, also add the
> > -S flag as an extra set of tests.
> 
> --sparse had the opposite result, minimal size for zeroed file, no space saved 
> for random file.
> combination of --sparse and --inplace is not supported
> 
> > When you are on Solaris, also see the impact of a test case using
> > mkfile and not dd if=/dev/zero.
> 
> sadly, I have no Solaris in my environment, all test were done on Linux
> 
> > 
> > On Thu, Mar 6, 2014 at 11:17 PM,  <devzero at web.de> wrote:
> > > Hi Pavel,
> > > 
> > > maybe that´s related to zfs compression ?
> > > 
> > > on compressed zfs filesystem, zeroes are not written to disk.
> 
> compression was not turned on on the volume (unless this is enabled even if 
> compression is set to off)
> 
> > > 
> > > # dd if=/dev/zero of=test.dat bs=1024k count=100
> > > 
> > > /zfspool # ls -la
> > > total 8
> > > drwxr-xr-x  3 root root         4 Feb 26 10:18 .
> > > drwxr-xr-x 27 root root      4096 Mar 29  2013 ..
> > > drwxr-xr-x 25 root root        25 Mar 29  2013 backup
> > > -rw-r--r--  1 root root 104857600 Feb 26 10:18 test.dat
> > > 
> > > /zfspool # du -k test.dat
> > > 1       test.dat
> > > 
> > > /zfspool # du -k --apparent-size test.dat
> > > 102400  test.dat
> > > 
> > > despite that, space calculation on compressed fs is a difficult thing...
> > > 
> 
> space was as reported by 'zfs list', on a volume that was created specifically 
> for this test. I would assume that is the most reliable way to get space usage
> 
> 
> > The other Question that would be interested (both with and without -S)
> > is when you use the dd if=/dev/urandom created file, but change some
> > places with dd =/dev/zero (ie the reverse of the A test case, creatin
> > with dd if=/dev/zero and changes with dd if=/dev/urandom)
> 
> I just rerun all the tests with --sparse and --inplace, results follow 
> (cleanup is done after each 'zfs list', not shown)
> 
> 
> thanks
> Pavel Herrmann
> 
> 
> 
> 
> 
> zero-inited file
> 
> remote runs:
> # dd if=/dev/zero of=testfile bs=1024 count=102400
> # dd if=/dev/urandom of=testfile count=1 bs=1024 seek=854 conv=notrunc
> # dd if=/dev/urandom of=testfile count=1 bs=1024 seek=45368 conv=notrunc
> # dd if=/dev/urandom of=testfile count=50 bs=1024 seek=9647 conv=notrunc
> 
> 
> # rsync -aHAv --stats --delete --sparse root at remote:/test/ .
> receiving incremental file list
> ./
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 1 (reg: 1)
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 33
> Total bytes received: 104,870,500
> 
> sent 33 bytes  received 104,870,500 bytes  41,948,213.20 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --sparse root at remote:/test/ .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,301
> 
> sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
> total size is 104,857,600  speedup is 852.42
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --sparse root at remote:/test/ .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,301
> 
> sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
> total size is 104,857,600  speedup is 852.42
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --sparse root at remote:/test/ .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 61,440 bytes
> Matched data: 104,796,160 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 102,489
> 
> sent 71,710 bytes  received 102,489 bytes  116,132.67 bytes/sec
> total size is 104,857,600  speedup is 601.94
> 
> # zfs list
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                         1.43M  3.44T   616K  legacy
> 
> 
> 
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 104,870,497
> 
> sent 71,710 bytes  received 104,870,497 bytes  29,983,487.71 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,301
> 
> sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
> total size is 104,857,600  speedup is 852.42
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,301
> 
> sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
> total size is 104,857,600  speedup is 852.42
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 61,440 bytes
> Matched data: 104,796,160 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 102,489
> 
> sent 71,710 bytes  received 102,489 bytes  69,679.60 bytes/sec
> total size is 104,857,600  speedup is 601.94
> # zfs list
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                          346M  3.44T   100M  legacy
> 
> 
> random-init
> 
> remote runs:
> # dd if=/dev/urandom of=testfile count=102400 bs=1024
> # dd if=/dev/urandom of=testfile count=1 bs=1024 seek=854 conv=notrunc
> # dd if=/dev/urandom of=testfile count=1 bs=1024 seek=45368 conv=notrunc
> # dd if=/dev/urandom of=testfile count=50 bs=1024 seek=9647 conv=notrunc
> 
> 
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> ./
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 1 (reg: 1)
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 33
> Total bytes received: 104,870,500
> 
> sent 33 bytes  received 104,870,500 bytes  41,948,213.20 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,297
> 
> sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
> total size is 104,857,600  speedup is 852.45
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,297
> 
> sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
> total size is 104,857,600  speedup is 852.45
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 61,440 bytes
> Matched data: 104,796,160 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 102,481
> 
> sent 71,710 bytes  received 102,481 bytes  69,676.40 bytes/sec
> total size is 104,857,600  speedup is 601.97
> 
> # zfs list 
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                          401M  3.44T   100M  legacy
> 
> 
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> ./
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 1 (reg: 1)
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 33
> Total bytes received: 104,870,500
> 
> sent 33 bytes  received 104,870,500 bytes  69,913,688.67 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --inplace root at test:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,297
> 
> sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
> total size is 104,857,600  speedup is 852.45
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 10,240 bytes
> Matched data: 104,847,360 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 51,297
> 
> sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
> total size is 104,857,600  speedup is 852.45
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 61,440 bytes
> Matched data: 104,796,160 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 102,481
> 
> sent 71,710 bytes  received 102,481 bytes  116,127.33 bytes/sec
> total size is 104,857,600  speedup is 601.97
> # zfs list
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                          101M  3.44T   100M  legacy
> 
> 
> 
> random init, zeroed holes
> 
> remote runs:
> # dd if=/dev/urandom of=testfile count=102400 bs=1024
> # dd if=/dev/zero of=testfile count=20000 bs=1024 seek=10 conv=notrunc
> # dd if=/dev/zero of=testfile count=20000 bs=1024 seek=20010 conv=notrunc
> # dd if=/dev/zero of=testfile count=20000 bs=1024 seek=40010 conv=notrunc
> # dd if=/dev/zero of=testfile count=20000 bs=1024 seek=60010 conv=notrunc
> # dd if=/dev/zero of=testfile count=20000 bs=1024 seek=80010 conv=notrunc
> 
> 
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> ./
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 1 (reg: 1)
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 33
> Total bytes received: 104,870,500
> 
> sent 33 bytes  received 104,870,500 bytes  29,963,009.43 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 20,480,000 bytes
> Matched data: 84,377,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 20,515,557
> 
> sent 71,710 bytes  received 20,515,557 bytes  8,234,906.80 bytes/sec
> total size is 104,857,600  speedup is 5.09
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  25,059.33 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at d
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  75,178.00 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at e
> # rsync -aHAv --stats --delete --sparse root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  75,178.00 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs list
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                          309M  3.44T  2.72M  legacy
> 
> 
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> ./
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 1 (reg: 1)
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 104,857,600 bytes
> Matched data: 0 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 33
> Total bytes received: 104,870,500
> 
> sent 33 bytes  received 104,870,500 bytes  29,963,009.43 bytes/sec
> total size is 104,857,600  speedup is 1.00
> # zfs snapshot zraid/test at a
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 20,480,000 bytes
> Matched data: 84,377,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 20,515,557
> 
> sent 71,710 bytes  received 20,515,557 bytes  13,724,844.67 bytes/sec
> total size is 104,857,600  speedup is 5.09
> # zfs snapshot zraid/test at b
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at c
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at d
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> 
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
> total size is 104,857,600  speedup is 929.86
> # zfs snapshot zraid/test at e
> # rsync -aHAv --stats --delete --inplace root at remote:/test/  .
> receiving incremental file list
> testfile
> z
> Number of files: 2 (reg: 1, dir: 1)
> Number of created files: 0
> Number of regular files transferred: 1
> Total file size: 104,857,600 bytes
> Total transferred file size: 104,857,600 bytes
> Literal data: 0 bytes
> Matched data: 104,857,600 bytes
> File list size: 50
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 71,710
> Total bytes received: 41,057
> 
> sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
> total size is 104,857,600  speedup is 929.86
> zalohovadlo recover # zfs list
> NAME                                USED  AVAIL  REFER  MOUNTPOINT
> <snip>
> zraid/test                          198M  3.44T   100M  legacy
> 
> 
> 
> 
> 
> 
> 
> 
>