file contents cause rsync to fail (with certains args and dir structure)

Mon Mar 5 10:14:36 UTC 2018

Problem was introduced with this commit:

commit f3873b3d88b61167b106e7b9227a20147f8f6197

Author: Wayne Davison <wayned at samba.org>

Date:   Mon Oct 10 11:49:50 2016 -0700

    Support --sparse combined with --preallocate or --inplace.

    The new code tries to punch holes in the destination file using newer

    Linux fallocate features. It also supports a --whole-file + --sparse +

    --inplace copy on any filesystem by truncating the destination file.

.Dave.

Sent using Zoho Mail

---- On Sun, 04 Mar 2018 23:17:23 +0100 Dave Gordon via rsync <rsync at lists.samba.org> wrote ----

Quite strange at first sight that the failure should depend on the files containing NULs!

But I've reproduced it on both Ubuntu and OpenSUSE with d73762e "Preparing for release of 3.1.3".

The problem remains even if you drop the --checksum or --delay-updates options from the command line, but goes away if you don't use both "--sparse" and "--preallocate" together.

So it looks like a bad interaction of these options, probably with the sparse handling being affected by the NULs in the files.

HTH,

.Dave.

Sent using Zoho Mail

---- On Sun, 04 Mar 2018 14:57:24 +0100 xftroxgpx via rsync <rsync at lists.samba.org> wrote ----

-- 

Please use reply-all for most replies to avoid omitting the mailing list. 

To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync 

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

script to reproduce: 

#!/bin/bash 

#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 

#tested to work ok : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 

if test "$1" == "clean"; then 

rm -vrf destdir sourcedir sourcedir2 sourcedir3 

exit 0 

fi 

echo '!! test 1:' 

mkdir -p destdir 

mkdir -p sourcedir/a 

#one \0 followed by a non-\0 (so, using a space) required: 

echo -ne '\0 ' > sourcedir/a/b 

#non zero size file required: 

echo -ne 'c' > sourcedir/c 

echo 'sourcedir' >/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ 

#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b": No such file or directory (2) 

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 

# ^ this happens first and any subsequent times! 

echo '!! test 2:' 

mkdir -p sourcedir2/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' > sourcedir2/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' > sourcedir2/c 

#in order to see this error: (for file 'c') 

#non-zero file size required and it must be prefixed by '.' aka dot 

#echo -ne '1' > sourcedir2/.d 

#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go 

#otherwise, the error is for file "a/b" 

echo 'sourcedir2' >/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ 

#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b": No such file or directory (2) 

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1] 

echo '!! test 3:' #same as 2 but an extra file '.d' exists! 

mkdir -p sourcedir3/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' > sourcedir3/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' > sourcedir3/c 

#non-zero file size required and it must be prefixed by '.' aka dot 

echo -ne '1' > sourcedir3/.d 

echo 'sourcedir3' >/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ 

#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c": No such file or directory (2) 

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 

A pristine copy of this script can be found here: https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go 

Just run ./go to start all 3 tests. 

All 3 tests should fail with: 

ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 

All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 

(aka rsync version 3.1.2 protocol version 31) 

Cheers! 

Sent with ProtonMail Secure Email. 

-- 

Please use reply-all for most replies to avoid omitting the mailing list. 

To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync 

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20180305/ec6a83d9/attachment.html>