strange behavior of --inplace on ZFS

Pavel Herrmann morpheus.ibis at gmail.com
Thu Mar 6 16:50:47 MST 2014


Hi

apologies for reordering the message parts, and for the long bit at the end

On Thursday 06 March 2014 23:45:22 Hendrik Visage wrote:
> Question: The source and destination folder host OS and are both sides ZFS?

no, the remote (source) was ext4. However, i plan to use it against at least 
one NTFS machine as well

> I'd like to see some stats that rsync said it transfered, also add the
> -S flag as an extra set of tests.

--sparse had the opposite result, minimal size for zeroed file, no space saved 
for random file.
combination of --sparse and --inplace is not supported

> When you are on Solaris, also see the impact of a test case using
> mkfile and not dd if=/dev/zero.

sadly, I have no Solaris in my environment, all test were done on Linux

> 
> On Thu, Mar 6, 2014 at 11:17 PM,  <devzero at web.de> wrote:
> > Hi Pavel,
> > 
> > maybe that´s related to zfs compression ?
> > 
> > on compressed zfs filesystem, zeroes are not written to disk.

compression was not turned on on the volume (unless this is enabled even if 
compression is set to off)

> > 
> > # dd if=/dev/zero of=test.dat bs=1024k count=100
> > 
> > /zfspool # ls -la
> > total 8
> > drwxr-xr-x  3 root root         4 Feb 26 10:18 .
> > drwxr-xr-x 27 root root      4096 Mar 29  2013 ..
> > drwxr-xr-x 25 root root        25 Mar 29  2013 backup
> > -rw-r--r--  1 root root 104857600 Feb 26 10:18 test.dat
> > 
> > /zfspool # du -k test.dat
> > 1       test.dat
> > 
> > /zfspool # du -k --apparent-size test.dat
> > 102400  test.dat
> > 
> > despite that, space calculation on compressed fs is a difficult thing...
> > 

space was as reported by 'zfs list', on a volume that was created specifically 
for this test. I would assume that is the most reliable way to get space usage


> The other Question that would be interested (both with and without -S)
> is when you use the dd if=/dev/urandom created file, but change some
> places with dd =/dev/zero (ie the reverse of the A test case, creatin
> with dd if=/dev/zero and changes with dd if=/dev/urandom)

I just rerun all the tests with --sparse and --inplace, results follow 
(cleanup is done after each 'zfs list', not shown)


thanks
Pavel Herrmann





zero-inited file

remote runs:
# dd if=/dev/zero of=testfile bs=1024 count=102400
# dd if=/dev/urandom of=testfile count=1 bs=1024 seek=854 conv=notrunc
# dd if=/dev/urandom of=testfile count=1 bs=1024 seek=45368 conv=notrunc
# dd if=/dev/urandom of=testfile count=50 bs=1024 seek=9647 conv=notrunc


# rsync -aHAv --stats --delete --sparse root at remote:/test/ .
receiving incremental file list
./
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33
Total bytes received: 104,870,500

sent 33 bytes  received 104,870,500 bytes  41,948,213.20 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --sparse root at remote:/test/ .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,301

sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
total size is 104,857,600  speedup is 852.42
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --sparse root at remote:/test/ .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,301

sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
total size is 104,857,600  speedup is 852.42
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --sparse root at remote:/test/ .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 61,440 bytes
Matched data: 104,796,160 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 102,489

sent 71,710 bytes  received 102,489 bytes  116,132.67 bytes/sec
total size is 104,857,600  speedup is 601.94

# zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                         1.43M  3.44T   616K  legacy



# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 104,870,497

sent 71,710 bytes  received 104,870,497 bytes  29,983,487.71 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,301

sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
total size is 104,857,600  speedup is 852.42
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,301

sent 71,710 bytes  received 51,301 bytes  82,007.33 bytes/sec
total size is 104,857,600  speedup is 852.42
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 61,440 bytes
Matched data: 104,796,160 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 102,489

sent 71,710 bytes  received 102,489 bytes  69,679.60 bytes/sec
total size is 104,857,600  speedup is 601.94
# zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                          346M  3.44T   100M  legacy


random-init

remote runs:
# dd if=/dev/urandom of=testfile count=102400 bs=1024
# dd if=/dev/urandom of=testfile count=1 bs=1024 seek=854 conv=notrunc
# dd if=/dev/urandom of=testfile count=1 bs=1024 seek=45368 conv=notrunc
# dd if=/dev/urandom of=testfile count=50 bs=1024 seek=9647 conv=notrunc


# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
./
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33
Total bytes received: 104,870,500

sent 33 bytes  received 104,870,500 bytes  41,948,213.20 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,297

sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
total size is 104,857,600  speedup is 852.45
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,297

sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
total size is 104,857,600  speedup is 852.45
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 61,440 bytes
Matched data: 104,796,160 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 102,481

sent 71,710 bytes  received 102,481 bytes  69,676.40 bytes/sec
total size is 104,857,600  speedup is 601.97

# zfs list 
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                          401M  3.44T   100M  legacy


# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
./
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33
Total bytes received: 104,870,500

sent 33 bytes  received 104,870,500 bytes  69,913,688.67 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --inplace root at test:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,297

sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
total size is 104,857,600  speedup is 852.45
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 10,240 bytes
Matched data: 104,847,360 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 51,297

sent 71,710 bytes  received 51,297 bytes  82,004.67 bytes/sec
total size is 104,857,600  speedup is 852.45
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 61,440 bytes
Matched data: 104,796,160 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 102,481

sent 71,710 bytes  received 102,481 bytes  116,127.33 bytes/sec
total size is 104,857,600  speedup is 601.97
# zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                          101M  3.44T   100M  legacy



random init, zeroed holes

remote runs:
# dd if=/dev/urandom of=testfile count=102400 bs=1024
# dd if=/dev/zero of=testfile count=20000 bs=1024 seek=10 conv=notrunc
# dd if=/dev/zero of=testfile count=20000 bs=1024 seek=20010 conv=notrunc
# dd if=/dev/zero of=testfile count=20000 bs=1024 seek=40010 conv=notrunc
# dd if=/dev/zero of=testfile count=20000 bs=1024 seek=60010 conv=notrunc
# dd if=/dev/zero of=testfile count=20000 bs=1024 seek=80010 conv=notrunc


# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
./
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33
Total bytes received: 104,870,500

sent 33 bytes  received 104,870,500 bytes  29,963,009.43 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 20,480,000 bytes
Matched data: 84,377,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 20,515,557

sent 71,710 bytes  received 20,515,557 bytes  8,234,906.80 bytes/sec
total size is 104,857,600  speedup is 5.09
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  25,059.33 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at d
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  75,178.00 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at e
# rsync -aHAv --stats --delete --sparse root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  75,178.00 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                          309M  3.44T  2.72M  legacy


# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
./
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33
Total bytes received: 104,870,500

sent 33 bytes  received 104,870,500 bytes  29,963,009.43 bytes/sec
total size is 104,857,600  speedup is 1.00
# zfs snapshot zraid/test at a
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 20,480,000 bytes
Matched data: 84,377,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 20,515,557

sent 71,710 bytes  received 20,515,557 bytes  13,724,844.67 bytes/sec
total size is 104,857,600  speedup is 5.09
# zfs snapshot zraid/test at b
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at c
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at d
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
total size is 104,857,600  speedup is 929.86
# zfs snapshot zraid/test at e
# rsync -aHAv --stats --delete --inplace root at remote:/test/  .
receiving incremental file list
testfile
z
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 0 bytes
Matched data: 104,857,600 bytes
File list size: 50
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71,710
Total bytes received: 41,057

sent 71,710 bytes  received 41,057 bytes  45,106.80 bytes/sec
total size is 104,857,600  speedup is 929.86
zalohovadlo recover # zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
<snip>
zraid/test                          198M  3.44T   100M  legacy










More information about the rsync mailing list