From samba-bugs at samba.org Sat Oct 1 13:58:26 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 01 Oct 2016 13:58:26 +0000
Subject: [Bug 12305] New: --fallocate and --sparse works wrong
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
Bug ID: 12305
Summary: --fallocate and --sparse works wrong
Product: rsync
Version: 3.1.1
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: socketpair at gmail.com
QA Contact: rsync-qa at samba.org
1.If I specify --fallocate without --sparse:
Sparse areas of SRC become fallocated on DST. This is not what I expect.
2. If I specify --sparse without --fallocate:
Fallocated areas of SRC become sparse areas on DST, meaning that very important
"attribute" is copied.
3. If I specify both --sparse and --fallocate:
Fallocate will win. see 1.
4. If I specify none of these flags:
sparse/preallocate information will be lost on DST.
This all is very sad.
So, during synchronisation, rsync should send type of the each area, which
should be on of:
1) containig user-data
2) sparse (contains zeroes)
3) preallocated (contain zeroes)
And receiver should apply that information on target file.
Note, that plain file may have preallocated space AFTER END OF FILE -- it is
not a bug, and should be transferred.
Also, receiver side may not handle all that features, so it should be
controllable what rsync receiver will do if making sparse or preallocated area
fails with EOPNOTSUPP or so.
Detecting holes and preallocated areas may give wrong information since some
FSes fake answer. So it should be controllable
see:
man 2 fallocate for possible modern operations.
man 2 lseek: detecting that areas (SEEK_HOLE/SEEK_DATA)
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 1 14:00:02 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 01 Oct 2016 14:00:02 +0000
Subject: [Bug 12305] --fallocate and --sparse work wrong
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
--- Comment #1 from Коренберг Марк ---
typo: very important "attribute" is NOT copied
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 1 19:20:30 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 01 Oct 2016 19:20:30 +0000
Subject: [Bug 12305] --fallocate and --sparse work wrong
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
Andrey Gursky changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |andrey.gursky at e-mail.ua
--- Comment #2 from Andrey Gursky ---
Hi Марк,
I stumbled over this almost a year ago. See
https://bugzilla.samba.org/show_bug.cgi?id=11588
Regards,
Andrey
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 1 22:07:29 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 01 Oct 2016 22:07:29 +0000
Subject: [Bug 12305] --fallocate and --sparse work wrong
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
Wayne Davison changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |DUPLICATE
--- Comment #3 from Wayne Davison ---
*** This bug has been marked as a duplicate of bug 11588 ***
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 1 22:07:29 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 01 Oct 2016 22:07:29 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
Wayne Davison changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |socketpair at gmail.com
--- Comment #1 from Wayne Davison ---
*** Bug 12305 has been marked as a duplicate of this bug. ***
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 2 07:33:16 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 02 Oct 2016 07:33:16 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #2 from Коренберг Марк ---
Bug 12305 has much more detailed description
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 2 09:45:58 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 02 Oct 2016 09:45:58 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #3 from Andrey Gursky ---
(In reply to Коренберг Марк from comment #2)
Марк, nevertheless it is still the same issue. You're welcome to add the
details here. Please, be sure, to fix the typo: instead of --fallocate it
should be --preallocate.
Thanks,
Andrey
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Mon Oct 3 19:27:52 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Mon, 03 Oct 2016 19:27:52 +0000
Subject: [Bug 12336] New: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE
Tech Support number
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12336
Bug ID: 12336
Summary: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE Tech
Support number
Product: rsync
Version: 3.1.3
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: erajasthan778 at gmail.com
QA Contact: rsync-qa at samba.org
❶❽667698127 SKYPE phone number 18667698127 SKYPE customer service phone number
18667698127
SKYPE desktop support, SKYPE laptop customer service, contact SKYPE support,
SKYPE pc
support, SKYPE laptop customer care number 18667698127, SKYPE support for ,
SKYPE customer care, SKYPE
customer care phone number 18667698127, hewlett packard help, phone number
18667698127 for SKYPE, SKYPE online help, SKYPE laptop
customer care, SKYPE helpline phone number 18667698127, SKYPE customer support,
SKYPE technical support chat,
SKYPE computer help, SKYPE support number 18667698127s, SKYPE technical support
contact number 18667698127, SKYPE telephone
number 18667698127, SKYPE technical support phone number 18667698127, SKYPE
helpline, SKYPE support ,
SKYPE support online, SKYPE contact number 18667698127, SKYPE help phone number
18667698127, SKYPE customer care
18667698127SKYPE helpline, SKYPE telephone support, SKYPE online support, SKYPE
support contact,SKYPE chat support, hewlett packard phone number, SKYPE
18667698127
customer service, SKYPE tech support number,18667698127 SKYPE product support,
hewlett
packard customer service phone number18667698127,SKYPE computer support number,
SKYPE support
contact number,18667698127 SKYPE support , SKYPE computer support, SKYPE tech
support chat,18667698127
1888 468 0321SKYPE helpline number, 18667698127 SKYPE laptop support,
hewlett packard tech support, SKYPE online
chat, hewlett packard technical support, SKYPE help line, phone number for
SKYPE support, hewlett packard support phone number, SKYPE technical support,
hewlett packard customer service number,1888 468 0321 SKYPE service number,
hewlett packard 18667698127
helpline, SKYPE customer care no, SKYPE customer service number, SKYPE help
number, 18667698127 SKYPE customer service phone number, SKYPE number, SKYPE
support
phone, SKYPE support line, hewlett packard contact number, SKYPE tech support
phone number, SKYPE customer support phone number, SKYPEs help, call
SKYPE support, ##SKYPE support## chat, hewlett packard support number,
hewlett packard tech support number, SKYPE support telephone number, hewlett
packard tech support phone number, call SKYPE, SKYPE contact support, hewlett
packard
technical support phone number, SKYPE support centre, hewlett packard customer
support, SKYPE desktop support, SKYPE laptop customer service, contact SKYPE
support, SKYPE pc support, SKYPE laptop customer care number, SKYPE support for
s, SKYPE customer care, SKYPE customer care phone number, hewlett
packard help, phone number for SKYPE, SKYPE online help, SKYPE laptop customer
care, SKYPE
helpline phone number, SKYPE customer support, SKYPE technical support chat,
SKYPE computer help, SKYPE support numbers, SKYPE technical support contact
number, SKYPE
telephone number, SKYPE technical support phone number, SKYPE
helpline, SKYPE support s, SKYPE support online, SKYPE contact number, SKYPE
help phone number, SKYPE customer care number, contact hewlett packard by
phone, SKYPE phone support, hewlett packard s support, SKYPE tech
support phone, SKYPE technical help, SKYPE laptop tech support number, contact
SKYPE by
phone, SKYPE support call, SKYPE computers support, hewlett packard customer
service
telephone number, phone number for hewlett packard, SKYPE online support chat,
SKYPE
laptop customer service number, SKYPE online chat support, SKYPEs customer
service, hewlett packard customer service phone, SKYPE laptop tech support,
SKYPE
service phone number, hewlett packard help, phone number for SKYPE
s, SKYPE troubleshooting phone number, SKYPE number,http://bit.ly/2bwGOcj
hewlett packard technical support number, contact SKYPE support phone, phone
number for SKYPE support, SKYPE customer support chat, SKYPE help and support
number, contact hewlett packard, SKYPE laptop support phone number, SKYPEs
customer service phone number, SKYPE laptop customer service phone number,
SKYPE
computer support phone number, SKYPE pavilion support, SKYPE computer customer
service, SKYPE customer services, hewlett packard telephone number, SKYPE
helpline
no, SKYPE help desk number, contact SKYPE support phone number, hewlett packard
contact, SKYPE phone numbers, SKYPEs customer care number, SKYPE help
and support, contact SKYPE technical support, SKYPE contact numbers, contact
SKYPE
support chat, call SKYPE tech support, SKYPE customer service phone, SKYPE help
support,
SKYPE computer tech support, SKYPE assistance phone number, SKYPE customer
service
telephone number, hewlett packard support phone number, SKYPE contact
support number, SKYPE support center phone number, SKYPE support phone numbers,
tech
support for SKYPE, SKYPE it support, SKYPE laptop helpline, SKYPE technical,
SKYPE laptop
technical support number, SKYPEs tech support phone number, SKYPEs
support phone number, hewlett packard help desk phone number, SKYPE computer
tech
support phone number, SKYPE customer service number for laptop, SKYPE
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Fri Oct 7 12:20:17 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 07 Oct 2016 12:20:17 +0000
Subject: [Bug 12367] New: temporary lines in --progress output are not cleared
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12367
Bug ID: 12367
Summary: temporary lines in --progress output are not cleared
Product: rsync
Version: 3.1.2
Hardware: All
OS: All
Status: NEW
Severity: minor
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: paul at debian.org
QA Contact: rsync-qa at samba.org
A bug report received by Debian
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749165). I can verify this.
emit_filelist_progress() in flist.c needs something like a
"output_needs_newline_maybe" variable... I can't quite see how to implement
that cleanly.
>>---
Some temporary lines output by --progress are not cleared, so that
the output looks strange. Here's a testcase.
------------------------------------------------------------
#!/bin/sh
set -e
export LC_ALL=C
t=$HOME/rsync-test
mkdir "$t"
cd "$t"
if [ ! -d a1 ]; then
mkdir a1
touch a1/foo
fi
if [ ! -d a2 ]; then
mkdir a2
for i in `seq 1 250`
do
mkdir -p a2/dir$i
for j in `seq 1 20`
do
mktemp --tmpdir=a2/dir$i > /dev/null
done
done
fi
mkdir b
for i in 1 2
do
mkdir -p c/a$i
ln -ns ../c/a$i b
done
rs()
{
rsync -KRt "$@" a2/*/* a1/foo "localhost:$t/b/"
}
rs
rs --progress
rm -r "$t"
------------------------------------------------------------
I get:
a1/00 files...
a2/
(sometimes without the "a2/" line).
If I redirect the output to some file, I can see with "less":
0 files...^M 400 files...^M 2500 files...^M 4600 files...^Ma1/
a2/
The problem is that when "a1/" is output, the previous " 4600 files..."
line was not cleared and one can still see the end of it.
<<---
I can verify this.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 00:19:05 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 00:19:05 +0000
Subject: [Bug 12367] temporary lines in --progress output are not cleared
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12367
--- Comment #1 from Andrey Gursky ---
Hi Paul,
you reported something similar on the list
(https://lists.samba.org/archive/rsync/2015-December/030478.html). I've
instantly recalled that, because this issue also annoys me. If I redirect rsync
output via pipe and tee the resulting file is polluted with percentage progress
info.
Regards,
Andrey
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 16:12:14 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 16:12:14 +0000
Subject: [Bug 12305] --fallocate and --sparse work wrong
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
--- Comment #4 from Carson Gaspar ---
Please define what you mean by 'fallocated' in (1) and (2). Please also specify
how you're determining that something has been 'fallocated'. I agree that (3)
is a bug, and as the only real one that I can see, and is correctly marked a s
a dup.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 16:44:44 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 16:44:44 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #4 from Carson Gaspar ---
rsync currently just has the receiver turn "long" sequences of zeroes into
sparse regions when --sparse is specified. If --preallocate is also specified,
what would you like rsync to do?
No wire protocol change required (please pick which behaviour you prefer even
if you'd rather a protocol bump, as we may negotiate an older wire protocol):
1) Emit an error (as with --in-place and --sparse)
2) Disable one of the options (which one?)
3) Pre-allocate the file, but when zero regions are detected then ftruncate()
it and create the sparse region. Reallocate the rest of the file space after
creating the sparse region or not? (IFF receiver is on Linux and on a supported
filesystem, fallocate(,FALLOC_FL_PUNCH_HOLE,...) could be used to create sparse
regions without truncating the file)
Wire protocol change required, have the sender determine the sparse regions,
using SEEK_HOLE if available, otherwise scanning for all-zero regions:
A) Preallocate 1st data region, create 1st sparse region, preallocate 2nd data
region, ...
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 16:53:06 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 16:53:06 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #5 from Carson Gaspar ---
(In reply to Carson Gaspar from comment #4)
Actually, you never want the sender to scan for zero regions if SEEK_HOLE isn't
supported, as performance would then be terrible. And a given filesystem may
not support SEEK_HOLE, even if lseek() does. So we're really back to picking
(1), (2), or (3), as I don't see unreliable sender sparse mapping as sensible
(although using SEEK_HOLE to save source file read time and provide a hint to
the receiver may be nice).
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 17:09:20 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 17:09:20 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #6 from Carson Gaspar ---
For punching holes, Solaris and UnixWare support F_FREESP(64) in fcntl().
Windows supports both reporting and punching holes, but I don't know if cygwin
(or any other rsync on windows platform) implements it.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 18:18:13 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 18:18:13 +0000
Subject: [Bug 11588] missing option: preallocate for all files except for
sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #7 from Wayne Davison ---
Created attachment 12556
--> https://bugzilla.samba.org/attachment.cgi?id=12556&action=edit
Preliminary patch to support punching holes
In my testing, using both a pre-allocate call on a file followed by a
hole-punch call has no effect on the allocation of the blocks (though it does
zero them).
I tested --sparse and --inplace with this, and it worked fine on one system
(with a new enough linux kernel). There are cases where the sparseness will be
lost, though, depending on OS & filesystem. I'm thinking we just update the
docs to mention that if you combine --sparse with --inplace (and/or
--preallocate) that you might not get the sparseness preserved.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 19:44:32 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 19:44:32 +0000
Subject: [Bug 12305] --fallocate and --sparse work wrong
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12305
--- Comment #5 from Коренберг Марк ---
Fallocated: areas of the file that has been fallocate()d, but stillnot written.
Technically, on sender, even written parts that was written, but contain zeroes
may be considered as fallocated areas. I mean that receiver should call
fallocate() on that region instead writing zeroes.
How to determine: How to determine if area is alocated: lseek() +
SEEK_DATA/SEEK_HOLE. And after that, check if non-sparse area contains zeroes.
Another way - is to examine fiemap (
https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt )
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 20:10:52 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 20:10:52 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #8 from Andrey Gursky ---
(In reply to Wayne Davison from comment #7)
Wayne,
since this bug made rsync unusable for me, I fixed that and implemented
additional checks needed for ext4 a month or two after I reported this bug and
saw no reaction at all. Now a couple of people got interested and you also, so
I can share my work. It's not tiny.
> In my testing, using both a pre-allocate call on a file followed by a
> hole-punch call has no effect on the allocation of the blocks (though it does
> zero them).
Yes, this is tricky. Hole-punch works only for full filesystem blocks (e.g.,
default 4K). Issuing few partial hole-punch requests wouldn't work, even if
they cover the whole block.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 21:03:22 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 21:03:22 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #9 from Wayne Davison ---
> Hole-punch works only for full filesystem blocks
That has nothing to do with it. If you fallocate() the full file length and
then (on the same file handle) try to punch out parts of the allocated file, no
blocks change away from becoming allocated. Looks like a bug in Linux to me.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 21:05:32 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 21:05:32 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #10 from Wayne Davison ---
> ... I can share my work.
Sounds interesting! Looking forward to seeing what you've come up with.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 21:28:49 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 21:28:49 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #11 from Theodore Ts'o ---
Re: #9. I'm not able to reproduce the described behavior. If you want to
follow up on what you think is a kernel bug, please send a simple repro program
or script and what version of the kernel you are using to the linux-ext4
mailing list. Thanks!!
Cheers,
{/usr/projects/docker/dropbox} (master)
1009% fallocate -o 0 -l 128M test.file
{/usr/projects/docker/dropbox} (master)
1010% filefrag -v test.file
Filesystem type is: ef53
File size of test.file is 134217728 (32768 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 4095: 55990272.. 55994367: 4096: unwritten
1: 4096.. 8191: 56025088.. 56029183: 4096: 55994368: unwritten
2: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten
3: 10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten
4: 14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten
5: 16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten
6: 20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten
7: 28672.. 32767: 56352768.. 56356863: 4096: 56313856:
last,unwritten,eof
test.file: 8 extents found
{/usr/projects/docker/dropbox} (master)
1011% xfs_io -c "fpunch 65536 65536" test.file
{/usr/projects/docker/dropbox} (master)
1012% filefrag -v test.file
Filesystem type is: ef53
File size of test.file is 134217728 (32768 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 15: 55990272.. 55990287: 16: unwritten
1: 32.. 4095: 55990304.. 55994367: 4064: unwritten
2: 4096.. 8191: 56025088.. 56029183: 4096: 55994368: unwritten
3: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten
4: 10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten
5: 14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten
6: 16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten
7: 20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten
8: 28672.. 32767: 56352768.. 56356863: 4096: 56313856:
last,unwritten,eof
test.file: 8 extents found
{/usr/projects/docker/dropbox} (master)
1013% uname -a
Linux callcc 4.8.0-00041-gecd2f69 #3 SMP Mon Oct 3 02:56:05 EDT 2016 x86_64
GNU/Linux
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 22:32:09 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 22:32:09 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #12 from Andrey Gursky ---
(In reply to Wayne Davison from comment #9)
>> Hole-punch works only for full filesystem blocks
> That has nothing to do with it.
Wayne,
OK, might be. I haven't tested it the exactly way you're doing it now, since I
did it other way. But this was important to take care in order to fix the bugs
related to sparse with inplace [1], [2]. But due to missing response I've hold
back the fix (not yet throughly tested) for me.
[1] About data/token send/receive protocol part and more
https://lists.samba.org/archive/rsync/2015-December/030471.html
[2] Status of --inplace and --sparse in rsync or alternative?
https://lists.samba.org/archive/rsync/2015-December/030472.html
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 22:36:56 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 22:36:56 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #13 from Andrey Gursky ---
(In reply to Theodore Ts'o from comment #11)
Theo,
I believe "on the same file handle" is the unusual prerequisite to trigger the
behavior described by Wayne. Or such a test is already contained in e2fsprogs
(in a C test program, not a shell script)?
Regards,
Andrey
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 23:30:23 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 23:30:23 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #14 from Theodore Ts'o ---
>I believe "on the same file handle" is the unusual prerequisite to trigger the >behavior described by Wayne.
I was fairly sure that was a red herring, so I was trying to save myself some
time, but no, it doesn't replicate even if you use the same file descriptor....
{/usr/projects/linux/ext4} (origin)
1009% strace /tmp/test-fallocate
execve("/tmp/test-fallocate", ["/tmp/test-fallocate"], [/* 64 vars */]) = 0
brk(NULL) = 0x187e000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f20e7d2a000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=101609, ...}) = 0
mmap(NULL, 101609, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f20e7d11000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\3\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1685264, ...}) = 0
mmap(NULL, 3791264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f20e776d000
mprotect(0x7f20e7902000, 2093056, PROT_NONE) = 0
mmap(0x7f20e7b01000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x194000) = 0x7f20e7b01000
mmap(0x7f20e7b07000, 14752, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f20e7b07000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f20e7d0f000
arch_prctl(ARCH_SET_FS, 0x7f20e7d0f700) = 0
mprotect(0x7f20e7b01000, 16384, PROT_READ) = 0
mprotect(0x7f20e7d2d000, 4096, PROT_READ) = 0
munmap(0x7f20e7d11000, 101609) = 0
open("test-file", O_WRONLY|O_CREAT|O_TRUNC, 0700) = 3
fallocate(3, 0, 0, 1048576) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 32768, 32768) = 0
close(3) = 0
exit_group(0) = ?
+++ exited with 0 +++
{/usr/projects/linux/ext4} (origin)
1010% filefrag -v test-file
Filesystem type is: ef53
File size of test-file is 1048576 (256 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 7: 62130432.. 62130439: 8: unwritten
1: 16.. 255: 62130448.. 62130687: 240:
last,unwritten,eof
test-file: 1 extent found
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 23:31:03 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 23:31:03 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #15 from Theodore Ts'o ---
Created attachment 12557
--> https://bugzilla.samba.org/attachment.cgi?id=12557&action=edit
Test program to show that fallocate followed by punch hole works just fine....
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 8 23:47:03 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 08 Oct 2016 23:47:03 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #16 from Andrey Gursky ---
(In reply to Theodore Ts'o from comment #15)
Theo, thanks for taking time to test it! This works for me too (Debian Testing
4.7.4-2, ext4):
$ /usr/sbin/filefrag -v test-file
Filesystem type is: ef53
File size of test-file is 1048576 (256 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 7: 14482176.. 14482183: 8: unwritten
1: 16.. 255: 14482192.. 14482431: 240:
last,unwritten,eof
test-file: 1 extent found
$
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 04:15:11 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 04:15:11 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #17 from Wayne Davison ---
Take the test program and change the SYS_fallocate to use the
FALLOC_FL_KEEP_SIZE flag (don't forget to "rm test-file") and it will fail.
Rsync always pre-allocates with FALLOC_FL_KEEP_SIZE when the flag is available.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 04:16:40 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 04:16:40 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #18 from Wayne Davison ---
FYI, I tested on Linux 4.2.0 and 3.10.0 (I don't have a newer kernel running
here to try).
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 04:27:27 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 04:27:27 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #19 from Wayne Davison ---
Also, to be more like rsync would do you can follow the hole-punch with a seek
and a write so that the file ends up with a non-zero size. Apparently if I
change the order to do the seek & the write first and THEN punch hole it works.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 13:15:38 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 13:15:38 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #20 from Andrey Gursky ---
(In reply to Wayne Davison from comment #17)
>From what I see, it doesn't fail, since the file is not preallocated at all
with FALLOC_FL_KEEP_SIZE, but just a fully sparse file is created (consisting
of only one big hole):
$ ls -ls test-file
1024 -rwx------ 1 andrey andrey 0 Oct 9 14:47 test-file
$ /usr/sbin/filefrag -v test-file
Filesystem type is: ef53
File size of test-file is 0 (0 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 255: 14828800.. 14829055: 256:
last,unwritten,eof
test-file: 1 extent found
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 13:53:57 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 13:53:57 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #21 from Andrey Gursky ---
(In reply to Andrey Gursky from comment #20)
Sorry, it is indeed preallocated. Other still holds: hole-punch doesn't fail
because the file already consists of only hole. Such file I would call a
preallocated data-sparse. In opposite to the usual not preallocated
space-sparse with truncate:
$ truncate -s 1048576 test-sparse
$ ls -ls test-sparse
0 -rw-r--r-- 1 andrey andrey 1048576 Oct 9 15:34 test-sparse
$ /usr/sbin/filefrag -v test-sparse
Filesystem type is: ef53
File size of test-sparse is 1048576 (256 blocks of 4096 bytes)
test-sparse: 0 extents found
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 14:04:07 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 14:04:07 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #22 from Andrey Gursky ---
(In reply to Andrey Gursky from comment #21)
Continuing to think aloud. It's not really a hole, it's already reserved space.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sun Oct 9 14:54:48 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sun, 09 Oct 2016 14:54:48 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #23 from Wayne Davison ---
> Continuing to think aloud. It's not really a hole, it's already reserved space.
Exactly, and it's impossible to punch holes in that allocation. I'm changing my
patch to give the file a size to deal with this anomaly.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Mon Oct 10 14:38:24 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Mon, 10 Oct 2016 14:38:24 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #24 from Theodore Ts'o ---
So a simple workaround would be to use fallocate with KEEP_SIZE at first, then
use punch whole, write the blocks, etc., and then use either truncate to set
i_size, or seek to the desired size minus one and write a single byte.
Seeking to the desired size minus one is more portable, but if you want to
avoid allocating an extra 4k block, you could try using truncate, and if that
doesn't set i_size (it's not guaranteed by POSIX, but I believe all Linux file
systems will set i_size), seeking to size-1 and writing a single zero byte is
guaranteed to work.
That being said, I agree that ext4 should allow punch hole to work beyond
i_size, if there are blocks allocated using fallocate(2). We'll fix that for
the future, but for now, the workaround suggested above is probably the
simplest way to work around the issue in a way that's compatible with both the
current and future behavior.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Mon Oct 10 15:02:33 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Mon, 10 Oct 2016 15:02:33 +0000
Subject: [Bug 11588] better handling for --preallocate with --sparse
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #25 from Коренберг Марк ---
What about fallcate()d area beyond file size ? Will they be synchronized ? Just
curious.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From kip at thevertigo.com Mon Oct 10 16:24:18 2016
From: kip at thevertigo.com (Kip Warner)
Date: Mon, 10 Oct 2016 09:24:18 -0700
Subject: rsync: connection unexpectedly closed
Message-ID: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Hey list,
I am having problems as of late with my rsync backup. On the client
side I am using the following:
OPTS="-avvvrz
--compress-level=9
--itemize-changes
--delete
--delete-excluded
--human-readable
--files-from=$FILES
--include-from=$INCLUDES
--exclude-from=$EXCLUDES
--partial
--progress
--owner
--perms
--progress
--timeout=0
--times
--stats"
sudo rsync -e "ssh -i ${IDENTITY_FILE} -v -p ${REMOTE_PORT}" $OPTS / $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH
Note that I have temporarily disabled timeouts and added extra
verbosity. The transfer to the remote host via SSH works fine, up until
it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the
way through, hangs, and then times out. On the client side I see the
following:
...
rsync: connection unexpectedly closed (3542035 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1]
[sender] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255)
On the server side if I attach to the rsync process via strace, I see
the following:
$ strace -f -p 3095
...
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971})
3095 write(1, "\3\0\0\7\1\0\0", 7) = 7
3095 gettimeofday({1476036967, 673095}, NULL) = 0
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999970})
3095 write(1, "H\0\0\trecv_files(home/kip/.Virtual"..., 76) = 76
3095 gettimeofday({1476036967, 680312}, NULL) = 0
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 1 (in [3], left {40, 364402})
3095 read(3, "B\0\0\trecv_files(home/kip/.Virtual"..., 8184) = 160
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999973})
3095 write(1, "B\0\0\trecv_files(home/kip/.Virtual"..., 70) = 70
3095 gettimeofday({1476037227, 506412}, NULL) = 0
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971})
3095 write(1, "V\0\0\trecv mapped home/kip/.Virtua"..., 90) = 90
3095 gettimeofday({1476037227, 512591}, NULL) = 0
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
... a couple hundred times or so repeats ...
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}
Note that it looks like the select() call is timing out for what I
presume is a regular file descriptor (4 since stdin, stdout, and stderr
are 0-3 respectively). This could have nothing to do with rsync at all
and could be a file system issue, but I figured I'd ask.
The server the data is being uploaded to with the strace running on it
has rsync version:
$ rsync --version
rsync version 3.0.9 protocol version 30
The client reported:
$ rsync --version
rsync version 3.1.1 protocol version 31
Any help appreciated.
Regards,
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
URL:
From kip at thevertigo.com Sun Oct 9 22:07:06 2016
From: kip at thevertigo.com (Kip Warner)
Date: Sun, 09 Oct 2016 15:07:06 -0700
Subject: rsync: connection unexpectedly closed
Message-ID: <1476050826.1424.19.camel@thevertigo.com>
Hey list,
I am having problems as of late with my rsync backup. On the client
side I am using the following:
OPTS="-avvvrz
--compress-level=9
--itemize-changes
--delete
--delete-excluded
--human-readable
--files-from=$FILES
--include-from=$INCLUDES
--exclude-from=$EXCLUDES
--partial
--progress
--owner
--perms
--progress
--timeout=0
--times
--stats"
sudo rsync -e "ssh -i ${IDENTITY_FILE} -v -p ${REMOTE_PORT}" $OPTS / $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH
Note that I have temporarily disabled timeouts and added extra
verbosity. The transfer to the remote host via SSH works fine, up until
it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the
way through, hangs, and then times out. On the client side I see the
following:
...
rsync: connection unexpectedly closed (3542035 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1]
[sender] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255)
On the server side if I attach to the rsync process via strace, I see
the following:
$ strace -f -p 3095
...
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971})
3095 write(1, "\3\0\0\7\1\0\0", 7) = 7
3095 gettimeofday({1476036967, 673095}, NULL) = 0
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999970})
3095 write(1, "H\0\0\trecv_files(home/kip/.Virtual"..., 76) = 76
3095 gettimeofday({1476036967, 680312}, NULL) = 0
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}) = 1 (in [3], left {40, 364402})
3095 read(3, "B\0\0\trecv_files(home/kip/.Virtual"..., 8184) = 160
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999973})
3095 write(1, "B\0\0\trecv_files(home/kip/.Virtual"..., 70) = 70
3095 gettimeofday({1476037227, 506412}, NULL) = 0
3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971})
3095 write(1, "V\0\0\trecv mapped home/kip/.Virtua"..., 90) = 90
3095 gettimeofday({1476037227, 512591}, NULL) = 0
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
... a couple hundred times or so repeats ...
3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout)
3095 select(4, [3], [], NULL, {60, 0}
Note that it looks like the select() call is timing out for what I
presume is a regular file descriptor (4 since stdin, stdout, and stderr
are 0-3 respectively). This could have nothing to do with rsync at all
and could be a file system issue, but I figured I'd ask.
The server the data is being uploaded to with the strace running on it
has rsync version:
$ rsync --version
rsync version 3.0.9 protocol version 30
The client reported:
$ rsync --version
rsync version 3.1.1 protocol version 31
Any help appreciated.
Regards,
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From henri.shustak at gmail.com Wed Oct 12 00:30:03 2016
From: henri.shustak at gmail.com (Henri Shustak)
Date: Wed, 12 Oct 2016 13:30:03 +1300
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Message-ID:
> I am having problems as of late with my rsync backup.
Have you tried performing a copy to a known good local device? If a local copy fails, then I would start checking the file system of the source and also the hardware of that system.
----------------------------
HTRAX : http://www.htrax.xyz
From paul+rsync at wurtel.net Wed Oct 12 06:36:54 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Wed, 12 Oct 2016 08:36:54 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Message-ID: <20161012063654.GA11197@msgid.wurtel.net>
On Mon 10 Oct 2016, Kip Warner wrote:
>
> The server the data is being uploaded to with the strace running on it
> has rsync version:
>
> $ rsync --version
> rsync version 3.0.9 protocol version 30
>
> The client reported:
>
> $ rsync --version
> rsync version 3.1.1 protocol version 31
As always it's best to first upgrade to the current version (3.1.3) if
at all possible, as there's always the chance that the cause of your
problems has already been fixed.
Paul
From kip at thevertigo.com Wed Oct 12 16:48:18 2016
From: kip at thevertigo.com (Kip Warner)
Date: Wed, 12 Oct 2016 09:48:18 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To:
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Message-ID: <1476290898.27406.119.camel@thevertigo.com>
On Wed, 2016-10-12 at 13:30 +1300, Henri Shustak wrote:
> Have you tried performing a copy to a known good local device? If a
> local copy fails, then I would start checking the file system of the
> source and also the hardware of that system.
That's a good idea. I just tried that and it copied no problem.
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From kip at thevertigo.com Thu Oct 13 03:30:56 2016
From: kip at thevertigo.com (Kip Warner)
Date: Wed, 12 Oct 2016 20:30:56 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161012063654.GA11197@msgid.wurtel.net>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
Message-ID: <1476329456.27406.168.camel@thevertigo.com>
On Wed, 2016-10-12 at 08:36 +0200, Paul Slootman wrote:
> As always it's best to first upgrade to the current version (3.1.3)
> if at all possible, as there's always the chance that the cause of
> your problems has already been fixed.
Good call, but I believe I may have ruled this out. I didn't upgrade to
3.1.3, but both sides are running 3.1.1 protocol version 31 now. Same
problem.
I think the key insight was in the strace log which showed the select()
call was timed out. If I knew what type of file descriptor it was being
fed, I might have a clue. It might have been a socket or something on
disk. I don't know.
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From paul+rsync at wurtel.net Thu Oct 13 05:51:32 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Thu, 13 Oct 2016 07:51:32 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476329456.27406.168.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
Message-ID: <20161013055132.GA18767@msgid.wurtel.net>
On Wed 12 Oct 2016, Kip Warner wrote:
>
> I think the key insight was in the strace log which showed the select()
> call was timed out. If I knew what type of file descriptor it was being
> fed, I might have a clue. It might have been a socket or something on
> disk. I don't know.
You can use lsof -p $pid to show what files that process has opened.
On linux you can also use 'ls -l /proc/$pid/fd'.
Paul
From dhoworth at mrc-lmb.cam.ac.uk Thu Oct 13 09:09:04 2016
From: dhoworth at mrc-lmb.cam.ac.uk (Dave Howorth)
Date: Thu, 13 Oct 2016 10:09:04 +0100
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Message-ID:
On 2016-10-10 17:24, Kip Warner wrote:
> Note that I have temporarily disabled timeouts and added extra
> verbosity. The transfer to the remote host via SSH works fine, up until
> it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the
> way through, hangs, and then times out.
I have a similar but different problem. I make a regular download from a
site that always errors out on a particular large file. However, my
rsync error symptoms are different.
Unfortunately, the server admins seem to be the strong, silent types who
have repeatedly changed their minds about what they think is wrong and
who may or may not be attempting to solve the problem in isolation -
I've failed to get any meaningful communication going with them.
Fortunately, excluding the problem file is a reasonable workaround for me.
Have you tried excluding the problem file from the transfer?
One possibility is that the problem is not caused directly by rsync but
because of some underlying filesystem glitch. What OS & filesystems are
you using?
Cheers, Dave
From samba-bugs at samba.org Thu Oct 13 15:45:14 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Thu, 13 Oct 2016 15:45:14 +0000
Subject: [Bug 12378] New: why i cannot exclude dir/files if using option
"--files-from"
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
Bug ID: 12378
Summary: why i cannot exclude dir/files if using option
"--files-from"
Product: rsync
Version: 3.1.2
Hardware: x64
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: isaja at sissa.it
QA Contact: rsync-qa at samba.org
I don't understand if this is intended or not, but when I use "--files-from"
and "--exclude" options together all files are synced anyhow.
eg.:
rsync -av -c --delete --relative --delete-excluded --exclude=".cache"
--exclude="mydirToExclude" /home .
it correctly doesn't copy the .cache folder.
but:
rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt
--exclude=".cache" --exclude="mydirToExclude" /home .
does copy everything.
instead using a pattern such as --exclude="**myDirToExclude**"
doesn't copy all lines with that pattern
myFileList.txt is something like:
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json
userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe
userName/.cache/winetricks/dotnet30/dotnetfx3.exe
userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe
userName/.cache/winetricks/msls31/InstMsiW.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html
Thanks,
alessio
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Thu Oct 13 16:24:43 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Thu, 13 Oct 2016 16:24:43 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
--- Comment #1 from Kevin Korb ---
It did not copy the directory you excluded it copied the files within that
directory that you explicitly told it to copy and created the appropriate
directories to allow that to happen.
IOW, .cache is not relative to
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Thu Oct 13 22:04:15 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Thu, 13 Oct 2016 22:04:15 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
--- Comment #2 from Alessio ---
(In reply to Kevin Korb from comment #1)
Excuse me, but i don't get it, what's the difference between excluding
(matching) the pattern from a --files-from and a remote host?
debugging the rsync transfer from the remote host I get:
[sender] hiding directory home/userName/.cache because of pattern .cache
in the other case nothing is being hidden instead but .cache is inside my
path:
userName/.cache/winetricks/msls31/InstMsiW.exe
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Thu Oct 13 22:13:46 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Thu, 13 Oct 2016 22:13:46 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
--- Comment #3 from Kevin Korb ---
I didn't say anything specific about a remote host.
You excluded .cache then you told it to copy specific files some of which are
in .cache. If there was a file named .cache inside of a directory that you
told it to copy then it would be excluded. If you excluded
userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded.
Exclude is not patern matched against parts of things you explicitly tell rsync
to copy...
kmk at dementia[1%]> cd /tmp
kmk at dementia[2%]> mkdir test
kmk at dementia[3%]> cd !$
cd test
kmk at dementia[4%]> mkdir src dst
kmk at dementia[5%]> cd src
kmk at dementia[6%]> mkdir -p a b c d e cache/c
kmk at dementia[7%]> cd ..
kmk at dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" > list
kmk at dementia[9%]> cat list
a
b
c
d
e
cache/c
kmk at dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ cache/
cd+++++++++ cache/c/
cd+++++++++ d/
cd+++++++++ e/
sent 157 bytes received 37 bytes 388.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
kmk at dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ d/
cd+++++++++ e/
sent 106 bytes received 28 bytes 268.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
kmk at dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ d/
cd+++++++++ e/
sent 121 bytes received 31 bytes 304.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
Cache isn't really being copied it is just being created to make a place for
cache/c to be stored when cache/c or c is not excluded.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Thu Oct 13 23:51:07 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Thu, 13 Oct 2016 23:51:07 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
Wayne Davison changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |WORKSFORME
Status|NEW |RESOLVED
--- Comment #4 from Wayne Davison ---
Command args (which includes names inside a files-from file) are never excluded
by an exclude directive. You told rsync to copy it, so it copies it. Excludes
only affect matching of files that rsync finds inside directories that you told
it to copy. It is expected that you already trimmed any unwanted files/dirs
from your args before you called rsync, since those names are under your
control.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From wayned at samba.org Thu Oct 13 23:58:49 2016
From: wayned at samba.org (Wayne Davison)
Date: Thu, 13 Oct 2016 16:58:49 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476329456.27406.168.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
Message-ID:
On Wed, Oct 12, 2016 at 8:30 PM, Kip Warner wrote:
> I think the key insight was in the strace log which showed the
> select() call was timed out.
No, that's totally expected. While select() is waiting for I/O to arrive,
it returns to rsync every 60 seconds to allow it to decide if it wants to
continue waiting or do something else. You have to find the process that is
exiting first to discover why the connection is closing. This assumes that
it's not a network issue, where all of the programs get a closed-connection
error.
..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From kmk at sanitarium.net Fri Oct 14 00:08:43 2016
From: kmk at sanitarium.net (Kevin Korb)
Date: Thu, 13 Oct 2016 20:08:43 -0400
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476403504.18135.31.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
<1476403504.18135.31.camel@thevertigo.com>
Message-ID: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
I don't remember whether or not you said you were running rsync over ssh
but if you are you can also debug the ssh layer. You can even do it at
both ends....
On the server run a debugging sshd on an alternate port with:
/usr/sbin/sshd -dDp222
(note that this will only accept 1 connection, debug to the terminal,
then exit)
then on the client use rsync -e "ssh -vp222" to go verbose and use the
debugging port.
On 10/13/2016 08:05 PM, Kip Warner wrote:
> On Thu, 2016-10-13 at 16:58 -0700, Wayne Davison wrote:
>> No, that's totally expected. While select() is waiting for I/O to
>> arrive, it returns to rsync every 60 seconds to allow it to decide if
>> it wants to continue waiting or do something else. You have to find
>> the process that is exiting first to discover why the connection is
>> closing. This assumes that it's not a network issue, where all of the
>> programs get a closed-connection
>> error.
>
> Hey Wayne,
>
> I took Paul's advice and tried monitoring with lsof. This is what I
> saw:
>
> $ lsof -p 3104
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> rsync 3104 kip cwd DIR 0,19 4096 214695958 /home/kip/Disk_Backups/kip-desktop/yakkety
> rsync 3104 kip rtd DIR 179,2 4096 2 /
> rsync 3104 kip txt REG 179,2 460584 7817 /usr/bin/rsync
> rsync 3104 kip mem REG 179,2 42692 3974 /lib/arm-linux-gnueabihf/libnss_files-2.13.so
> rsync 3104 kip mem REG 179,2 71624 3971 /lib/arm-linux-gnueabihf/libnsl-2.13.so
> rsync 3104 kip mem REG 179,2 38608 3976 /lib/arm-linux-gnueabihf/libnss_nis-2.13.so
> rsync 3104 kip mem REG 179,2 26484 3972 /lib/arm-linux-gnueabihf/libnss_compat-2.13.so
> rsync 3104 kip mem REG 179,2 2953776 18967 /usr/lib/locale/locale-archive
> rsync 3104 kip mem REG 179,2 1216624 3965 /lib/arm-linux-gnueabihf/libc-2.13.so
> rsync 3104 kip mem REG 179,2 130448 41964 /lib/arm-linux-gnueabihf/libgcc_s.so.1
> rsync 3104 kip mem REG 179,2 43024 10745 /lib/arm-linux-gnueabihf/libpopt.so.0.0.0
> rsync 3104 kip mem REG 179,2 26240 1840 /lib/arm-linux-gnueabihf/libacl.so.1.1.0
> rsync 3104 kip mem REG 179,2 17904 1854 /lib/arm-linux-gnueabihf/libattr.so.1.1.0
> rsync 3104 kip mem REG 179,2 10170 28719 /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
> rsync 3104 kip mem REG 179,2 126236 3961 /lib/arm-linux-gnueabihf/ld-2.13.so
> rsync 3104 kip 1w FIFO 0,10 0t0 8455 pipe
> rsync 3104 kip 2w FIFO 0,10 0t0 8456 pipe
> rsync 3104 kip 3u unix 0xdbda86c0 0t0 8466 socket
>
> And on the strace...
>
> ...
> 3104 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left {59, 999978})
> 3104 write(1, "h\340\0\7\0y@\312\227\241\236\255\367\312\3637\323X\206\314\250\372\362P1\202\374\"i\265c\16"..., 57452) = 57452
> 3104 select(4, [3], [], [3], {60, 0}) = 0 (Timeout)
> (repeats above line until client (sender) reports connection timed out)
>
> Sender reports the following until it times out...
>
> ...
> match at 32949141504 last_match=32949141504 j=251382 len=131072 n=0
> 32.95G 96% 83.55MB/s 0:00:12
>
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL:
From kip at thevertigo.com Fri Oct 14 01:27:42 2016
From: kip at thevertigo.com (Kip Warner)
Date: Thu, 13 Oct 2016 18:27:42 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To:
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
Message-ID: <1476408462.18135.35.camel@thevertigo.com>
On Thu, 2016-10-13 at 10:09 +0100, Dave Howorth wrote:
> Have you tried excluding the problem file from the transfer?
Hey Dave. All the other files appear to sync, up until it gets to that
one large file. Then it stalls, and finally times out. I could tell it
to exclude that important file, but that would defeat the purpose of my
backup.
> One possibility is that the problem is not caused directly by rsync
> but because of some underlying filesystem glitch. What OS &
> filesystems are you using?
That could well be, but how to know?
Client side:
$ lsb_release -a
LSB Version: security-9.20160110ubuntu5-amd64:security-9.20160110ubuntu5-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety
Server side:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.11 (wheezy)
Release: 7.11
Codename: wheezy
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From samba-bugs at samba.org Fri Oct 14 09:20:04 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 14 Oct 2016 09:20:04 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
--- Comment #5 from Alessio ---
(In reply to Wayne Davison from comment #4)
Well if it's by design I will not go further. But for me it's a missing
"feature" because one has to prepare the list beforehand while rsync could do
the job "while it's there".
It's like the impossibility to sync files newer than specific date, which I
really miss, but it's another story. Thanks for your help!
(In reply to Kevin Korb from comment #3)
Thanks for the detailed example and explanation it's exactly what happens to
me.
And if you remove the "--files-from" you see that "cache" folder is really
excluded from the copy.
I was wondering why the behavior is different applying the "--files-from" or
not and unfortunately WayneD said it's by design.
Thanks for your help
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Fri Oct 14 16:57:33 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 14 Oct 2016 16:57:33 +0000
Subject: [Bug 12378] why i cannot exclude dir/files if using option
"--files-from"
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12378
--- Comment #6 from Kevin Korb ---
If you want to sync files newer than say 3 days ago that is what --files-from
is for...
cd /source
find . -mtime -3 -print | rsync -vai --files-from=- . /target
The primary purpose of --files-from is to give rsync the power of find.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From kip at thevertigo.com Sat Oct 15 07:02:16 2016
From: kip at thevertigo.com (Kip Warner)
Date: Sat, 15 Oct 2016 00:02:16 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
<1476403504.18135.31.camel@thevertigo.com>
<86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
Message-ID: <1476514936.31856.8.camel@thevertigo.com>
On Thu, 2016-10-13 at 20:08 -0400, Kevin Korb wrote:
> I don't remember whether or not you said you were running rsync over
> ssh but if you are you can also debug the ssh layer. You can even do
> it at both ends....
Yes, I'm doing this over an SSH tunnel.
> On the server run a debugging sshd on an alternate port with:
> /usr/sbin/sshd -dDp222
> (note that this will only accept 1 connection, debug to the terminal,
> then exit) then on the client use rsync -e "ssh -vp222" to go verbose
> and use the debugging port.
This is a great idea except that I only have one port forwarded to the
server (beyond my control) and that port is for the current SSH session
I would need to spawn another one on some other inaccessible port.
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From kip at thevertigo.com Sun Oct 16 02:54:41 2016
From: kip at thevertigo.com (Kip Warner)
Date: Sat, 15 Oct 2016 19:54:41 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
<1476403504.18135.31.camel@thevertigo.com>
<86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
Message-ID: <1476586481.31856.31.camel@thevertigo.com>
On Thu, 2016-10-13 at 20:08 -0400, Kevin Korb wrote:
> I don't remember whether or not you said you were running rsync over
> ssh but if you are you can also debug the ssh layer. You can even do
> it at both ends....
>
> On the server run a debugging sshd on an alternate port with:
> /usr/sbin/sshd -dDp222
> (note that this will only accept 1 connection, debug to the terminal,
> then exit) then on the client use rsync -e "ssh -vp222" to go verbose
> and use the debugging port.
Hey Kevin,
I managed to get a chance to try your suggestion. I noticed something
very interesting. On the server side, there's actually two processes of
rsync that appear to be spawned by the client's connection.
Running the ssh daemon on the server side, I notice that long after the
server dies like so...
...
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Read error from remote host : Connection reset by peer
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
Sessions still open, not unmounting
debug1: PAM: deleting credentials
...and the client rsync dies...
match at 32949010432 last_match=32949010432 j=251381 len=131072 n=0
match at 32949141504 last_match=32949141504 j=251382 len=131072 n=0
32.95G 96% 74.13MB/s 0:00:14 packet_write_wait: Connection to port 22223: Broken pipe
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.1]
[sender] _exit_cleanup(code=10, file=io.c, line=820): about to call exit(255)
...one of the server side rsync processes according to strace is still
alive...
write(1, "h\340\0\7\310\376\\\233\227\241\236\255VC\31\351\323X\206\314F$\337\0321\202\374\"\320(d\271"..., 57452) = 57452
select(4, [3], [], [3], {60, 0}) = 0 (Timeout)
(last line repeats a couple hundred times)
...and apparently the second rsync server side process is actually
still performing reads and writes more than an hour after the
connection died...
...
read(1, "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\252d'h\351\371\0ny"..., 262144) = 262144
write(3, "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\252d'h\351\371\0ny"..., 262144) = 262144
...
This makes me wonder if this has something to do with memory buffers
for file i/o being exhausted on the server and this server side
activity long after the connection died is an effort to flush a file
buffer to disk. The file this always breaks on is 30+GB, but the server
hardware is a small embedded system.
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From kip at thevertigo.com Sun Oct 16 04:55:58 2016
From: kip at thevertigo.com (Kip Warner)
Date: Sat, 15 Oct 2016 21:55:58 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476586481.31856.31.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<20161012063654.GA11197@msgid.wurtel.net>
<1476329456.27406.168.camel@thevertigo.com>
<1476403504.18135.31.camel@thevertigo.com>
<86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net>
<1476586481.31856.31.camel@thevertigo.com>
Message-ID: <1476593758.31856.33.camel@thevertigo.com>
On Sat, 2016-10-15 at 19:54 -0700, Kip Warner wrote:
> ...and apparently the second rsync server side process is
> actually
> still performing reads and writes more than an hour after the
> connection died...
>
> ...
> read(1,
> "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\2
> 52d'h\351\371\0ny"..., 262144) = 262144
> write(3,
> "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\2
> 52d'h\351\371\0ny"..., 262144) = 262144
> ...
And it finally just terminated:
...
select(5, [0], [4], [0], {60, 0}) = 2 (in [0], out [4], left {59, 999979})
read(0, "\373\375\372\343\276M\337\374'\201\234o\306\34\313\255\274b\16\314\206\262\10\304\7{z-\316N\347\242"..., 16384) = 16384
write(4, "B\0\0\trecv_files(home/kip/.Virtual"..., 160) = 160
select(1, [0], [], [0], {60, 0}) = 1 (in [0], left {59, 999978})
--- SIGUSR1 (User defined signal 1) @ 0 (0) ---
rt_sigaction(SIGUSR1, {SIG_IGN, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0
rt_sigaction(SIGUSR2, {SIG_IGN, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0
close(1) = 0
write(3, "\236\244\355{\356$\237\377\223\237\34\366\275\244\350\t\253\206<\266\3634\371\376\214\377\f\212\211|\310\263"..., 164033) = 164033
close(3) = 0
lstat64("home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", {st_mode=S_IFREG|0600, st_size=32949305537, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0
utimensat(AT_FDCWD, "home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", {UTIME_NOW, {0, 854158541}}, AT_SYMLINK_NOFOLLOW) = 0
rename("home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", "home/kip/.VirtualBox/Machines/Woe64 8.1/Woe64 8.1.vdi") = 0
gettimeofday({1476587334, 151625}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
gettimeofday({1476587334, 255616}, NULL) = 0
exit_group(19) = ?
Process 30619 detached
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From bs27975 at gmail.com Sun Oct 16 20:17:14 2016
From: bs27975 at gmail.com (B. S.)
Date: Sun, 16 Oct 2016 16:17:14 -0400
Subject: Links repeatedly being copied???
In-Reply-To:
References:
Message-ID: <5803E04A.40201@gmail.com>
(Kubuntu 12.04.5 LTS rsync version 3.0.9 protocol version 30)
I rsync the data ('data master') partition of my main machine to the
global ('local global repository') partition, and this global partition
to two other remote ('remote global repository') partitions, nightly.
local -> global, global -> 2 x remotes. This is done via cifs - i.e. no
daemons involved. (mount //remote/dest /mnt/remote/dest, rsync /src
/mnt/remote/dest)
I have for the longest time been seeing '.L..t......'s in my rsync log,
despite no changes to such in quite some time, and in trying to track it
down, been baffled.
Only yesterday did I notice (man rsync):
(1) -l, --links copy symlinks as symlinks
(2) -l, --links
When symlinks are encountered, recreate the symlink on
the destination.
The aha moment was tracking the word 'recreate'!
Sanity check please: rsync DOES NOT COPY links just as it would any
other file, it recreates them (at each creation / change detection)?
If so, could 'man rsync', (1) above, please s/copy/recreate/? I might
have picked up on things rather earlier than I did. [i.e. rsync mirror
... doesn't. At least for how I think of 'mirrors' - lose the local
disk, pop the remote disk into the local machine, get on with your day.]
(Queue head change of what rsync is. Not so much complaining, I get
rsync is what it is and it's a good thing, just need to change how my
head is wrapped around it.)
So, sanity check please:
- I create a symlink locally.
- next time rsync is run, it duplicates the link on the remote, DATED AT
THAT TIME OF (remote) LINK CREATION.
- subsequent rsync's detects a time difference between local vs remote
symlinks, so again duplicates the link on the remote, DATED AT THAT TIME
OF (remote) LINK CREATION. (This re-copying, er, link recreation, occurs
ad infinitum.)
Am I understanding this correctly?
(It's what I'm seeing.)
How do I stop this repeat entry in my logs?
In googling, I can see, and have used, tar to work around this. However,
I also keep seeing that rsync can be used instead of things like cp -ad,
for just this sort of thing. (Except, I'm using rsync and seeing this
sort of thing.)
rsync change request, please: When it creates that remote symlink, could
rsync please change the date/time to match that of the symlink from
which it's 'copying'? (Just like it does for any other file.)
Have I described / understood the above correctly / does it hold water?
If it is not expected that links would be recreated at every run, any
observations as to in what way (parameters likely used) I might be doing
myself in?
Thanks for listening, and thanks for your thoughts.
From henri.shustak at gmail.com Sun Oct 16 22:35:24 2016
From: henri.shustak at gmail.com (Henri Shustak)
Date: Mon, 17 Oct 2016 11:35:24 +1300
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476290898.27406.119.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
Message-ID:
>> Have you tried performing a copy to a known good local device? If a
>> local copy fails, then I would start checking the file system of the
>> source and also the hardware of that system.
>
> That's a good idea. I just tried that and it copied no problem.
Do you have another system you could try this transfer with via SSH with on the local network?
Given that the local transfer works fine, I would suggest checking the hardware and file system integrity on the remote machine. In terms of hardware checking the memory and disks would be a top priority.
Also, you could try moving the partial file out of the way and also not using the partial option and transferring again?
Hope that helps
--------------------------------------------------------------------
HTRAX 2013 Revitalised : EGYPTIAN HUMP HTRAX : Direct URL download :
http://henri.shustak.org/download/htrax/egyptian-hump-htrax.mp3
"Dr Who Meets B52's" - More reviews : http://www.jessetaylor.com.au
From paul+rsync at wurtel.net Tue Oct 18 06:36:11 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Tue, 18 Oct 2016 08:36:11 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
Message-ID: <20161018063611.GA2247@msgid.wurtel.net>
Try the transfer without -z.
Paul
From hohmann at harddiskcafe.de Tue Oct 18 10:08:00 2016
From: hohmann at harddiskcafe.de (Bernd Hohmann)
Date: Tue, 18 Oct 2016 12:08:00 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
Message-ID: <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de>
On 18.10.2016 07:03, Kip Warner wrote:
> From what I can tell, there are no hardware problems. I also ran fsck
> on the drive. The machine seems to be fine.
I can confirm the problem.
Situation here: 2 identical HP Microservers (Debian 7, on site compiled
rsync 3.1.2, connected via OpenVPN).
SSH is used for transport.
Both machines have the correct date/time set via ntpd.
All files on Client/Server are rw and have the right owner and are
copy'able. oth sides.
The "directory to backup" is a Samba-share (I stopped nmbd and smbd, no
change). Client: 200GB, 42000 files total. Enough disk-space and memory
on both sides.
All rsync instances were killed (Client/Server) before starting rsync.
tcpdump shows me a NOP packet every 2 min.
I can provoke the error doing this:
1) Start the transfer (rsync scans *all* client files and starts sending
a file)
2) ^C rsync on client
3) "pkill rsync" on server until all rsync-processes are killed. Same on
client (just to be sure)
4) Start the transfer again, now rsync scans the top directories only
and hangs (see straces below).
Commandline:
./rsync-debug -v --archive --progress
--human-readable --delete-during \
--rsync-path=/home/backup-hugo/bin/rsync-debug \
/srv/backup-bernd backup-hugo at backup-hugo.vpn:/srv/
Client says (PID 5909 = rsync, 5910 = ssh)
------------------------------------------------
[...]
5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}) = 1 (in [4], left
{239, 999990})
5910 10:13:50 read(4, "2010_20120119093643.pdf\0\3740O
[...]
\242}\30:V0124160__Nr.036_vom_10.09.2010_2012011"..., 16384) = 3072
loop:
5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}
5909 10:14:51 <... select resumed> ) = 0 (Timeout)
5909 10:14:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout)
5909 10:15:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout)
5909 10:16:51 select(6, [5], [], [5], {60, 0}
5910 10:17:51 <... select resumed> ) = 0 (Timeout)
goto loop
------------------------------------------------
Server says (PID 10331 = rsync --server, 10332 = ssh)
------------------------------------------------
[...]
10331 10:13:50 lstat("backup-bernd/Schreibtisch", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
10331 10:13:50 lstat("backup-bernd/VirtualBox", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
10331 10:13:50 lstat("backup-bernd/bin", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
10331 10:13:50 lstat("backup-bernd/projekte", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
10331 10:13:50 lstat("backup-bernd/transfer", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
10331 10:13:50 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left {59,
999991})
10331 10:13:50 write(1, "\4\0\0\7\3\20\0\0", 8) = 8
10331 10:13:50 select(4, [3], [], [3], {60, 0}
10332 10:14:50 <... select resumed> ) = 0 (Timeout)
loop:
10332 10:14:50 select(1, [0], [], [0], {60, 0}
10331 10:14:50 <... select resumed> ) = 0 (Timeout)
10331 10:14:50 select(4, [3], [], [3], {60, 0}
10332 10:15:50 <... select resumed> ) = 0 (Timeout)
goto loop
------------------------------------------------
--
Bernd Hohmann
Organisationsprogrammierer
Höhenstrasse 2 * 61130 Nidderau
Telefon: 06187/900495 * Telefax: 06187/900493
Blog: http://blog.harddiskcafe.de
From devzero at web.de Wed Oct 19 19:46:24 2016
From: devzero at web.de (devzero at web.de)
Date: Wed, 19 Oct 2016 21:46:24 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
<86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de>
Message-ID: <62005202-A93C-4772-A731-597E2D00EE25@web.de>
what does lsof tell? does rsync hang on a specific file?
i would wonder if this is a rsync problem. as you told you killed all processes.
so, on the second run rsync knows nothing from before...
roland
Am 18. Oktober 2016 12:08:00 MESZ, schrieb Bernd Hohmann :
>On 18.10.2016 07:03, Kip Warner wrote:
>
>> From what I can tell, there are no hardware problems. I also ran fsck
>> on the drive. The machine seems to be fine.
>
>I can confirm the problem.
>
>Situation here: 2 identical HP Microservers (Debian 7, on site compiled
>rsync 3.1.2, connected via OpenVPN).
>
>SSH is used for transport.
>
>Both machines have the correct date/time set via ntpd.
>
>All files on Client/Server are rw and have the right owner and are
>copy'able. oth sides.
>
>The "directory to backup" is a Samba-share (I stopped nmbd and smbd, no
>change). Client: 200GB, 42000 files total. Enough disk-space and memory
>on both sides.
>
>All rsync instances were killed (Client/Server) before starting rsync.
>
>tcpdump shows me a NOP packet every 2 min.
>
>
>I can provoke the error doing this:
>
>1) Start the transfer (rsync scans *all* client files and starts
>sending
>a file)
>
>2) ^C rsync on client
>
>3) "pkill rsync" on server until all rsync-processes are killed. Same
>on
>client (just to be sure)
>
>4) Start the transfer again, now rsync scans the top directories only
>and hangs (see straces below).
>
>
>Commandline:
>
>./rsync-debug -v --archive --progress
> --human-readable --delete-during \
> --rsync-path=/home/backup-hugo/bin/rsync-debug \
> /srv/backup-bernd backup-hugo at backup-hugo.vpn:/srv/
>
>
>Client says (PID 5909 = rsync, 5910 = ssh)
>------------------------------------------------
>[...]
>5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}) = 1 (in [4], left
>{239, 999990})
>5910 10:13:50 read(4, "2010_20120119093643.pdf\0\3740O
>[...]
>\242}\30:V0124160__Nr.036_vom_10.09.2010_2012011"..., 16384) = 3072
>
>loop:
>5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}
>5909 10:14:51 <... select resumed> ) = 0 (Timeout)
>5909 10:14:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout)
>5909 10:15:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout)
>5909 10:16:51 select(6, [5], [], [5], {60, 0}
>5910 10:17:51 <... select resumed> ) = 0 (Timeout)
>goto loop
>------------------------------------------------
>
>Server says (PID 10331 = rsync --server, 10332 = ssh)
>------------------------------------------------
>[...]
>10331 10:13:50 lstat("backup-bernd/Schreibtisch",
>{st_mode=S_IFDIR|0755,
>st_size=4096, ...}) = 0
>10331 10:13:50 lstat("backup-bernd/VirtualBox", {st_mode=S_IFDIR|0755,
>st_size=4096, ...}) = 0
>10331 10:13:50 lstat("backup-bernd/bin", {st_mode=S_IFDIR|0755,
>st_size=4096, ...}) = 0
>10331 10:13:50 lstat("backup-bernd/projekte", {st_mode=S_IFDIR|0755,
>st_size=4096, ...}) = 0
>10331 10:13:50 lstat("backup-bernd/transfer", {st_mode=S_IFDIR|0755,
>st_size=4096, ...}) = 0
>10331 10:13:50 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left
>{59,
>999991})
>10331 10:13:50 write(1, "\4\0\0\7\3\20\0\0", 8) = 8
>10331 10:13:50 select(4, [3], [], [3], {60, 0}
>10332 10:14:50 <... select resumed> ) = 0 (Timeout)
>
>loop:
>10332 10:14:50 select(1, [0], [], [0], {60, 0}
>10331 10:14:50 <... select resumed> ) = 0 (Timeout)
>10331 10:14:50 select(4, [3], [], [3], {60, 0}
>10332 10:15:50 <... select resumed> ) = 0 (Timeout)
>goto loop
>------------------------------------------------
>
>--
>Bernd Hohmann
>Organisationsprogrammierer
>Höhenstrasse 2 * 61130 Nidderau
>Telefon: 06187/900495 * Telefax: 06187/900493
>Blog: http://blog.harddiskcafe.de
>
>
>
>--
>Please use reply-all for most replies to avoid omitting the mailing
>list.
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read:
>http://www.catb.org/~esr/faqs/smart-questions.html
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From hohmann at harddiskcafe.de Wed Oct 19 21:12:55 2016
From: hohmann at harddiskcafe.de (Bernd Hohmann)
Date: Wed, 19 Oct 2016 23:12:55 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <62005202-A93C-4772-A731-597E2D00EE25@web.de>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
<86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de>
<62005202-A93C-4772-A731-597E2D00EE25@web.de>
Message-ID: <35442d5e-ab3e-d608-29be-a1839f5556bf@harddiskcafe.de>
On 19.10.2016 21:46, devzero at web.de wrote:
> what does lsof tell? does rsync hang on a specific file?
Nothing. I ruled this out already.
> i would wonder if this is a rsync problem. as you told you killed all
> processes.
> so, on the second run rsync knows nothing from before...
Maybe something stupid in the SSH(D) stack. I checked all running ssh
instances for orphans - nothing.
Current status quo of testing:
1) After restarting the client, rsync starts working synchronizing
changed/missing files on the server via vpn and runs into a timeout
after a couple of hours.
2) Restarting the backup (after pkill'ing rsync on both sides, just to
be sure) runs into a timeout and no attempt was made to rsync files.
3) Hint: If I remove a directory (13 Files, 50GB total), everything is fine.
Are there some magic Unix file attributes on ext4?
Bernd
--
Bernd Hohmann
Organisationsprogrammierer
Höhenstrasse 2 * 61130 Nidderau
Telefon: 06187/900495 * Telefax: 06187/900493
Blog: http://blog.harddiskcafe.de
From kip at thevertigo.com Thu Oct 20 01:24:28 2016
From: kip at thevertigo.com (Kip Warner)
Date: Wed, 19 Oct 2016 18:24:28 -0700
Subject: rsync: connection unexpectedly closed
In-Reply-To: <20161018063611.GA2247@msgid.wurtel.net>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
<20161018063611.GA2247@msgid.wurtel.net>
Message-ID: <1476926668.1807.13.camel@thevertigo.com>
On Tue, 2016-10-18 at 08:36 +0200, Paul Slootman wrote:
> Try the transfer without -z.
>
> Paul
I ended up giving up. What I did was I just removed the 30GB file
(which I really didn't need anyways) and the transfer carried on
without a hitch.
--
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: This is a digitally signed message part
URL:
From space.ship.traveller at gmail.com Thu Oct 20 09:24:23 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Thu, 20 Oct 2016 22:24:23 +1300
Subject: -e escape rule
Message-ID:
Hello,
I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.
Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.
Here is an example command generated by some Ruby code:
rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
We can check that something like this is valid:
files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz -- What Rsync should be executing
However this gives me an error
command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.
I'm looking in the function:
static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
int *f_in_p, int *f_out_p)
This function splits based purely on whitespace:
args[argc++] = t;
while (*f != ' ' || in_quote) {
// consume token...
I feel that this function should also handle backslash escapes.
I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.
Kind regards,
Samuel
From paul+rsync at wurtel.net Thu Oct 20 11:03:32 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Thu, 20 Oct 2016 13:03:32 +0200
Subject: -e escape rule
In-Reply-To:
References:
Message-ID: <20161020110332.GA30679@msgid.wurtel.net>
On Thu 20 Oct 2016, Samuel Williams wrote:
>
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
>
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
>
> Here is an example command generated by some Ruby code:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
There's no reason to escape an "=" sign in the above command. Try:
rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
Or even:
rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
I'd probably create an entry for this host in ~/.ssh/config :
Host example.backup.server.com
User backup
IdentityFile /etc/synco/id_rsa
ConnectTimeout 60
BatchMode=yes
and then just use:
rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
If you're dynamically doing this, you can pass a config file with -F.
Paul
From space.ship.traveller at gmail.com Thu Oct 20 12:22:32 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Fri, 21 Oct 2016 01:22:32 +1300
Subject: -e escape rule
In-Reply-To: <20161020110332.GA30679@msgid.wurtel.net>
References:
<20161020110332.GA30679@msgid.wurtel.net>
Message-ID:
> There's no reason to escape an "=" sign in the above command.
Okay, so at a certain level I agree with you and that's how I've fixed
the issue.
All the stuff about config files is not feasible for a variety of
reasons - I'm aware of those options. I appreciate that you took the
time to explain them.
I think the problem is the assumptions in do_cmd about the contents of
the -e argument.
In my opinion, the way this is split needs to be deferred back to the
user's shell to work correctly. I'm not sure if there is an easy way
to do this that works for any kind of shell.
Assuming that -e is stored into rsh_cmd, and is user supplied for
executing on the shell, the only valid thing you can really do with it
is:
execl("/bin/sh", "sh", "-c", rsh_cmd, (char *) NULL);
(assuming that sh is the user's current shell, might want to use
$SHELL or something else).
It might make more sense if rsh_cmd used string substitution rather
than argument splitting, e.g. -e 'ssh $HOST' this way, no splitting
would be required. In lots of ways this is more flexible. For example,
appending the host might not be useful in some contexts.
Anyway, rsync is pretty awesome. the -e option is only a minor
annoyance, but it would be nice if this discussion could result in a
tangible improvement.
On 21 October 2016 at 00:03, Paul Slootman wrote:
> On Thu 20 Oct 2016, Samuel Williams wrote:
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> There's no reason to escape an "=" sign in the above command. Try:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> Or even:
>
> rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> I'd probably create an entry for this host in ~/.ssh/config :
>
> Host example.backup.server.com
> User backup
> IdentityFile /etc/synco/id_rsa
> ConnectTimeout 60
> BatchMode=yes
>
> and then just use:
>
> rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
>
> If you're dynamically doing this, you can pass a config file with -F.
>
>
> Paul
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
From dhoworth at mrc-lmb.cam.ac.uk Thu Oct 20 12:46:50 2016
From: dhoworth at mrc-lmb.cam.ac.uk (Dave Howorth)
Date: Thu, 20 Oct 2016 13:46:50 +0100
Subject: -e escape rule
In-Reply-To:
References:
Message-ID: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
On 2016-10-20 10:24, Samuel Williams wrote:
> Hello,
>
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
>
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
The man page for rsync
explicitly forbids the use of backslashes:
"Command-line arguments are permitted in COMMAND provided that COMMAND
is presented to rsync as a single argument. You must use spaces (not
tabs or other whitespace) to separate the command and args from each
other, and you can use single- and/or double-quotes to preserve spaces
in an argument (but not backslashes)."
So I think the rest of the question is moot ...
Cheers, Dave
> Here is an example command generated by some Ruby code:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> We can check that something like this is valid:
>
> files% echo foo\ bar\\\=baz
> foo bar\=baz -- what Rsync should be receiving
> files% echo foo bar\=baz
> foo bar=baz -- What Rsync should be executing
>
> However this gives me an error
>
> command-line: line 0: Bad configuration option: connecttimeout\\
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>
> I think the problem here is the "ConnectTimeout\\\=60", in particular
> how the equals symbol is escaped.
>
> I'm looking in the function:
>
> static pid_t do_cmd(char *cmd, char *machine, char *user, char
> **remote_argv, int remote_argc,
> int *f_in_p, int *f_out_p)
>
> This function splits based purely on whitespace:
>
> args[argc++] = t;
> while (*f != ' ' || in_quote) {
> // consume token...
>
> I feel that this function should also handle backslash escapes.
>
> I also checked using strace and it appears that this is the issue, but
> I'm open to suggestions/ideas.
>
> Kind regards,
> Samuel
>
From kmk at sanitarium.net Thu Oct 20 17:03:52 2016
From: kmk at sanitarium.net (Kevin Korb)
Date: Thu, 20 Oct 2016 13:03:52 -0400
Subject: -e escape rule
In-Reply-To:
References:
Message-ID:
The \ escapes are for the shell. Rsync never sees them and therefore
can't honor them.
On 10/20/2016 05:24 AM, Samuel Williams wrote:
> Hello,
>
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
>
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
>
> Here is an example command generated by some Ruby code:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> We can check that something like this is valid:
>
> files% echo foo\ bar\\\=baz
> foo bar\=baz -- what Rsync should be receiving
> files% echo foo bar\=baz
> foo bar=baz -- What Rsync should be executing
>
> However this gives me an error
>
> command-line: line 0: Bad configuration option: connecttimeout\\
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>
> I think the problem here is the "ConnectTimeout\\\=60", in particular
> how the equals symbol is escaped.
>
> I'm looking in the function:
>
> static pid_t do_cmd(char *cmd, char *machine, char *user, char
> **remote_argv, int remote_argc,
> int *f_in_p, int *f_out_p)
>
> This function splits based purely on whitespace:
>
> args[argc++] = t;
> while (*f != ' ' || in_quote) {
> // consume token...
>
> I feel that this function should also handle backslash escapes.
>
> I also checked using strace and it appears that this is the issue, but
> I'm open to suggestions/ideas.
>
> Kind regards,
> Samuel
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL:
From hohmann at harddiskcafe.de Thu Oct 20 20:45:33 2016
From: hohmann at harddiskcafe.de (Bernd Hohmann)
Date: Thu, 20 Oct 2016 22:45:33 +0200
Subject: rsync: connection unexpectedly closed
In-Reply-To: <1476926668.1807.13.camel@thevertigo.com>
References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop>
<1476290898.27406.119.camel@thevertigo.com>
<20161018050302.wfihlh5rxhj3f7qd@kip-desktop>
<20161018063611.GA2247@msgid.wurtel.net>
<1476926668.1807.13.camel@thevertigo.com>
Message-ID:
On 20.10.2016 03:24, Kip Warner wrote:
> I ended up giving up.
Me too.
I'm copying all files via 'scp' now - takes 3 days but no aborts or errors.
So I am very sure the problem is somewhere in rsync.
Bernd
--
Bernd Hohmann
Organisationsprogrammierer
Höhenstrasse 2 * 61130 Nidderau
Telefon: 06187/900495 * Telefax: 06187/900493
Blog: http://blog.harddiskcafe.de
From space.ship.traveller at gmail.com Fri Oct 21 00:20:41 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Fri, 21 Oct 2016 13:20:41 +1300
Subject: -e escape rule
In-Reply-To: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID:
Hi Dave, thanks for point that out. I didn't realise there was a
detailed explanation of that field in the man page, I only saw the
summary. Yes, that clearly explains how it's supposed to work.
On 21 October 2016 at 01:46, Dave Howorth wrote:
> On 2016-10-20 10:24, Samuel Williams wrote:
>>
>> Hello,
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>
>
> The man page for rsync
> explicitly forbids the use of backslashes:
>
> "Command-line arguments are permitted in COMMAND provided that COMMAND is
> presented to rsync as a single argument. You must use spaces (not tabs or
> other whitespace) to separate the command and args from each other, and you
> can use single- and/or double-quotes to preserve spaces in an argument (but
> not backslashes)."
>
> So I think the rest of the question is moot ...
>
> Cheers, Dave
>
>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>>
>> We can check that something like this is valid:
>>
>> files% echo foo\ bar\\\=baz
>> foo bar\=baz -- what Rsync should be receiving
>> files% echo foo bar\=baz
>> foo bar=baz -- What Rsync should be executing
>>
>> However this gives me an error
>>
>> command-line: line 0: Bad configuration option: connecttimeout\\
>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>>
>> I think the problem here is the "ConnectTimeout\\\=60", in particular
>> how the equals symbol is escaped.
>>
>> I'm looking in the function:
>>
>> static pid_t do_cmd(char *cmd, char *machine, char *user, char
>> **remote_argv, int remote_argc,
>> int *f_in_p, int *f_out_p)
>>
>> This function splits based purely on whitespace:
>>
>> args[argc++] = t;
>> while (*f != ' ' || in_quote) {
>> // consume token...
>>
>> I feel that this function should also handle backslash escapes.
>>
>> I also checked using strace and it appears that this is the issue, but
>> I'm open to suggestions/ideas.
>>
>> Kind regards,
>> Samuel
>>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
From space.ship.traveller at gmail.com Fri Oct 21 01:35:05 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Fri, 21 Oct 2016 14:35:05 +1300
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID:
Hmm, so after reviewing this in a bit more detail.. it would be very
nice if at least backslash escapes would be supported. Quoted strings
are pretty ugly, especially when nested several levels. Anyway, just
my 2 cents.
From samba-bugs at samba.org Fri Oct 21 06:09:05 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 21 Oct 2016 06:09:05 +0000
Subject: [Bug 12386] New: Copy Loop
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12386
Bug ID: 12386
Summary: Copy Loop
Product: rsync
Version: 3.1.3
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: crysanthus at gmail.com
QA Contact: rsync-qa at samba.org
If you use .* as a rsync source it will be looped until end of disk
ie.
rsync -rvhIW crp/.* rohanamobile/public/
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Fri Oct 21 06:12:10 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 21 Oct 2016 06:12:10 +0000
Subject: [Bug 12386] Copy Loop
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12386
--- Comment #1 from Kevin Korb ---
That isn't a bug you are using it wrong. drop the .* and it will do what you
expect. There should almost never be a * in the source parameter.
Also, don't rsync without --times unless you have a really good reason.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From wayned at samba.org Fri Oct 21 21:22:48 2016
From: wayned at samba.org (Wayne Davison)
Date: Fri, 21 Oct 2016 14:22:48 -0700
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID:
On Thu, Oct 20, 2016 at 6:35 PM, Samuel Williams <
space.ship.traveller at gmail.com> wrote:
> it would be very nice if at least backslash escapes would be supported
>
They are supported just fine. In your first message you passed several
option\\\=args to ssh, and it was *ssh* that gave you an error (it didn't
like that it received a backslash as part of the option name).
..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From samba-bugs at samba.org Fri Oct 21 21:36:34 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Fri, 21 Oct 2016 21:36:34 +0000
Subject: [Bug 12386] Copy Loop
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12386
Wayne Davison changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |INVALID
--- Comment #2 from Wayne Davison ---
You'd probably be better served by asking about something like this on the
mailing list, as there is no issue with .* concerning rsync -- this is entirely
a shell issue.
In this instance you're probably using bash, which is so stupid that it expands
".*" to include "..", which is obviously not what you want. I personally use
zsh, which doesn't have this issue.
When using bash, you could instead try a "path/.??*" (if all your dot files are
at least 2 chars after the dot) or something like ".[a-z]*" (though you might
also need A-Z and 0-9 etc added).
--
You are receiving this mail because:
You are the QA Contact for the bug.
From devzero at web.de Tue Oct 25 09:20:23 2016
From: devzero at web.de (devzero at web.de)
Date: Tue, 25 Oct 2016 11:20:23 +0200
Subject: error code 255 - not mentioned in manpage?
Message-ID:
hi,
is there a reason why error code 255 is not mentioned in the manpage
and wouldn`t it make sense to add "255 Unexplained Error" there
for completeness ?
I`m writing a script which checks exit values and while testing it, i
got value 255, looked into the manpage and scratched my head what`s the
meaning of it.
ok - it`s obvious if you take a look at rsync error output, but i think
all exit values should be documented.
regards
roland
# cat rsync.err
ssh: connect to host ...... port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]
From paul+rsync at wurtel.net Tue Oct 25 09:37:40 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Tue, 25 Oct 2016 11:37:40 +0200
Subject: error code 255 - not mentioned in manpage?
In-Reply-To:
References:
Message-ID: <20161025093740.GA26536@msgid.wurtel.net>
On Tue 25 Oct 2016, devzero at web.de wrote:
>
> is there a reason why error code 255 is not mentioned in the manpage
> and wouldn`t it make sense to add "255 Unexplained Error" there
> for completeness ?
It wouldn't be unexplained then anymore, would it? :-)
Paul
From devzero at web.de Wed Oct 26 09:34:09 2016
From: devzero at web.de (devzero at web.de)
Date: Wed, 26 Oct 2016 11:34:09 +0200
Subject: O_NOATIME ?
Message-ID:
Hello,
since we are using rsync for backing up millions of files in a virtual environment, and most of the virtual machines run on SSD cached storage, i`d be curious how that negatively impacts lifetime of the SSD`s when we do rsync run every night for backup
my question:
does rsync normal file comparison run to determine if anything has changed change atime of any files ?
for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not sure under which conditions atime is changed.
grepping the source for O_NOATIME in rsync3.txt i found :
- Propagate atimes and do not modify them. This is very ugly on
Unix. It might be better to try to add O_NOATIME to kernels, and
call that.
furhermore, apparently there _IS_ O_NOATIME in linux kernels for a while:
http://man7.org/linux/man-pages/man2/open.2.html
O_NOATIME (since Linux 2.6.8)
Do not update the file last access time (st_atime in the
inode) when the file is read(2).
This flag can be employed only if one of the following
conditions is true:
* The effective UID of the process matches the owner UID of
the file.
* The calling process has the CAP_FOWNER capability in its
user namespace and the owner UID of the file has a mapping
in the namespace.
This flag is intended for use by indexing or backup programs,
where its use can significantly reduce the amount of disk
activity. This flag may not be effective on all filesystems.
One example is NFS, where the server maintains the access
time.
so, maybe someone likes to comment on NOATIME !?
maybe it could be useful to make rsync honour O_NOATIME ?
regards
roland
From paul+rsync at wurtel.net Wed Oct 26 09:46:50 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Wed, 26 Oct 2016 11:46:50 +0200
Subject: O_NOATIME ?
In-Reply-To:
References:
Message-ID: <20161026094650.GA27031@msgid.wurtel.net>
On Wed 26 Oct 2016, devzero at web.de wrote:
>
> since we are using rsync for backing up millions of files in a virtual environment, and most of the virtual machines run on SSD cached storage, i`d be curious how that negatively impacts lifetime of the SSD`s when we do rsync run every night for backup
>
> my question:
> does rsync normal file comparison run to determine if anything has changed change atime of any files ?
>
> for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not sure under which conditions atime is changed.
Most filesystems on modern linux systems should be mounted with the
relatime option.
The atime will then only be updated if either the mtime or ctime is
newer than the atime, or if the atime is older than a defined interval.
Note that simply using stat will not update the atime, as the *file*
itself has not been accessed, only its metadata.
There's also the nodiratime option that can be useful, as directories
are read to find what files exist in those directories; and it's
seldomly useful to maintain the atime of a directory.
You can also use noatime as a mount option, but then be sure that no
application uses the atimes of files; e.g. something like mutt use the
atime and mtime to determine whether there is a mail file with unread
mail.
Paul
From tytso at mit.edu Thu Oct 27 04:29:06 2016
From: tytso at mit.edu (Theodore Ts'o)
Date: Thu, 27 Oct 2016 00:29:06 -0400
Subject: O_NOATIME ?
In-Reply-To: <20161026094650.GA27031@msgid.wurtel.net>
References:
<20161026094650.GA27031@msgid.wurtel.net>
Message-ID: <20161027042906.pdeqhegscxxveaep@thunk.org>
On Wed, Oct 26, 2016 at 11:46:50AM +0200, Paul Slootman wrote:
> Most filesystems on modern linux systems should be mounted with the
> relatime option.
This is the default...
> You can also use noatime as a mount option, but then be sure that no
> application uses the atimes of files; e.g. something like mutt use the
> atime and mtime to determine whether there is a mail file with unread
> mail.
With the latest Linux kernels (4.0+), there is also the "lazytime"
mount option. This causes the kernel to avoid writing back inodes
that only have "dirty timestamps", until either (a) some other change
is made to the inode, (b) fsync(), syncfs(), or sync() is called, (c)
an undeleted inode is evicted from memory or the file system is
unounted, (d) 24 hours have gone by, or (e) in the case of ext4, if
some other change is made to an inode in the same inode table block as
an inode with a dirty timestamp (so we need to do the disk I/O
anyway).
With lazytime, the timestamp is updated in memory, so stat(2) will
always return the correct timestamp, and in normal practice, the
timestamps will be (eventually) updated on disk. However if a
timestsamp gets updated multiple times --- for example, a database
file getting updated with O_DIRECT writes, instead of the mtime field
being written out every 30 seconds, we can defer the timestamp updates
and collapse them into a much smaller set of inode table block writes.
Cheers,
- Ted
From space.ship.traveller at gmail.com Fri Oct 28 12:39:00 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Sat, 29 Oct 2016 01:39:00 +1300
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID:
Wayne, I guess if you apply standard shell escaping logic you end up
with a string which isn't processed correctly, yes, ssh complains, but
only because Rsync passed the backslashes through without dealing with
them. My opinion is that if Rsync is splitting a string based on
whitespace it also needs to take into account backslashes.
From rsync at jelmail.com Fri Oct 28 14:32:03 2016
From: rsync at jelmail.com (John Lane)
Date: Fri, 28 Oct 2016 15:32:03 +0100
Subject: rsync send "non-rsync" options to the server side
(--remote-option) and (--protect-args)
In-Reply-To:
References:
Message-ID: <1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com>
Hello, I asked the below back at the beginning of August but have
received no replies. If anyone can help me with this problem it would be
much appreciated. If I have not provided some necessary information then
please let me know so I may do so. Thanks in advance.
> I have been working on a backup server where I have a server-side script
> that wraps the server-side rsync invocation. I have used the client-side
> rsync -M (--remote-option) to send options to the server script,
> removing them from the command-line prior to invoking the server-side
> rsync. This was discussed on Stack Exchange
> (http://unix.stackexchange.com/questions/297143). It works well.
>
> However, one observation is that the rsync client's -s (--protect-args)
> option passes all arguments directly into the rsync server and only
> sends a minimal rsync command-line to launch the server-side rsync.
>
> This means my script does not see -M options when -s is used. It also
> means that the server-side rsync receives options not intended for it
> and these cause it to terminate.
>
> I note from the man-page that the -s option "will eventually become a
> new default setting at some as-yet-undetermined point in the future".
> When that happens, it will break what I have above.
>
> So, a couple of questions:
>
> * is using -M the right way to send non-rsync options to the server side ?
>
> * could it be considered for -s to not affect non-rsync options that
> have been specified with -M ?
>
> Thanks for your attention.
>
From wayned at samba.org Sat Oct 29 02:10:40 2016
From: wayned at samba.org (Wayne Davison)
Date: Fri, 28 Oct 2016 19:10:40 -0700
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID:
On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams <
space.ship.traveller at gmail.com> wrote:
> Rsync passed the backslashes through without dealing with them.
>
Yeah, it only does space-splitting and that's all it will ever do. It still
looks to me like there is a bug in the original escaping, since any command
receiving that string is receiving a backslash that is not supposed to be
there. It should only be escaping the string enough to get it to rsync, not
trying to guess what rsync is going to do with it after it gets it.
To work around that bug, you could consider using an ssh-calling shell
script instead of manually specifying the ssh command and args to rsync.
..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From wayned at samba.org Sat Oct 29 02:14:43 2016
From: wayned at samba.org (Wayne Davison)
Date: Fri, 28 Oct 2016 19:14:43 -0700
Subject: rsync send "non-rsync" options to the server side
(--remote-option) and (--protect-args)
In-Reply-To: <1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com>
References:
<1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com>
Message-ID:
If you want to pass non-rsync args (etc) you should be using --rsync-path.
The -M option is only for sending rsync-related options.
..wayne..
On Fri, Oct 28, 2016 at 7:32 AM, John Lane wrote:
>
> Hello, I asked the below back at the beginning of August but have
> received no replies. If anyone can help me with this problem it would be
> much appreciated. If I have not provided some necessary information then
> please let me know so I may do so. Thanks in advance.
>
> > I have been working on a backup server where I have a server-side script
> > that wraps the server-side rsync invocation. I have used the client-side
> > rsync -M (--remote-option) to send options to the server script,
> > removing them from the command-line prior to invoking the server-side
> > rsync. This was discussed on Stack Exchange
> > (http://unix.stackexchange.com/questions/297143). It works well.
> >
> > However, one observation is that the rsync client's -s (--protect-args)
> > option passes all arguments directly into the rsync server and only
> > sends a minimal rsync command-line to launch the server-side rsync.
> >
> > This means my script does not see -M options when -s is used. It also
> > means that the server-side rsync receives options not intended for it
> > and these cause it to terminate.
> >
> > I note from the man-page that the -s option "will eventually become a
> > new default setting at some as-yet-undetermined point in the future".
> > When that happens, it will break what I have above.
> >
> > So, a couple of questions:
> >
> > * is using -M the right way to send non-rsync options to the server side
> ?
> >
> > * could it be considered for -s to not affect non-rsync options that
> > have been specified with -M ?
> >
> > Thanks for your attention.
> >
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/
> mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From kmk at sanitarium.net Sat Oct 29 02:41:56 2016
From: kmk at sanitarium.net (Kevin Korb)
Date: Fri, 28 Oct 2016 22:41:56 -0400
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
Message-ID: <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
A \ is a shell escape. Every level of shell consumes one level of
escape. For every session of a shell every example "\ " becomes " " and
every example of "\\" becomes "\".
The raync command line is a mix of local shell commands + commands run
via a remote shell via ssh.
The simple fact is that if only \ escapes are used, the local shell will
utilize those escapes and rsync will never see them.
For every level of shell an escape \ does its job.
If you want rsync to interpret a space char as other than whitespace
then a \ is one way to do so.
But one we are talking about the other side of an rsync we are talking
about a local command line that consumes a \. Then we are talking about
a remote command line that consumes another \. So, within the -e
parameter, multiple stacked \ characters are required.
Simply put, it is absolutely absurd to \ escape an rsync command line
while interpreting the subshell -e parameter and the remote shell
parameters the same way.
At best you are insisting that ssh do something entirely nonsensical
while ensuring that the first s in "ssh" is interpreted as an "s".
On 10/28/2016 10:10 PM, Wayne Davison wrote:
>
> On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams
> >
> wrote:
>
> Rsync passed the backslashes through without dealing with them.
>
>
> Yeah, it only does space-splitting and that's all it will ever do. It
> still looks to me like there is a bug in the original escaping, since
> any command receiving that string is receiving a backslash that is not
> supposed to be there. It should only be escaping the string enough to
> get it to rsync, not trying to guess what rsync is going to do with it
> after it gets it.
>
> To work around that bug, you could consider using an ssh-calling shell
> script instead of manually specifying the ssh command and args to rsync.
>
> ..wayne..
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL:
From space.ship.traveller at gmail.com Sat Oct 29 03:25:11 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Sat, 29 Oct 2016 16:25:11 +1300
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
Message-ID:
Kevin, I agree with what you are saying on some level, but I don't
think the code does what you think it does. rsync -e "foo\\ bar" will
be executed by the shell and yield a cmd string in the do_cmd function
of "foo\ bar". This will be split incorrectly into an argv of ["foo\",
"bar"]. I'm not sure in what reality this makes any sense. All I'm
proposing is that the do_cmd split function interprets the -e as a
string that would normally be executed on the command line, and if it
needs to be split, follow the normal rules of the command line for
handling whitespace.
On 29 October 2016 at 16:24, Samuel Williams
wrote:
> Kevin, I agree with what you are saying on some level, but I don't
> think the code does what you think it does. rsync -e "foo\\ bar" will
> be executed by the shell and yield a cmd string in the do_cmd function
> of "foo\ bar". This will be split incorrectly into an argv of ["foo\",
> "bar"]. I'm not sure in what reality this makes any sense. All I'm
> proposing is that the do_cmd split function interprets the -e as a
> string that would normally be executed on the command line, and if it
> needs to be split, follow the normal rules of the command line for
> handling whitespace.
>
> On 29 October 2016 at 15:41, Kevin Korb wrote:
>> A \ is a shell escape. Every level of shell consumes one level of
>> escape. For every session of a shell every example "\ " becomes " " and
>> every example of "\\" becomes "\".
>>
>> The raync command line is a mix of local shell commands + commands run
>> via a remote shell via ssh.
>>
>> The simple fact is that if only \ escapes are used, the local shell will
>> utilize those escapes and rsync will never see them.
>>
>> For every level of shell an escape \ does its job.
>>
>> If you want rsync to interpret a space char as other than whitespace
>> then a \ is one way to do so.
>>
>> But one we are talking about the other side of an rsync we are talking
>> about a local command line that consumes a \. Then we are talking about
>> a remote command line that consumes another \. So, within the -e
>> parameter, multiple stacked \ characters are required.
>>
>> Simply put, it is absolutely absurd to \ escape an rsync command line
>> while interpreting the subshell -e parameter and the remote shell
>> parameters the same way.
>>
>> At best you are insisting that ssh do something entirely nonsensical
>> while ensuring that the first s in "ssh" is interpreted as an "s".
>>
>>
>> On 10/28/2016 10:10 PM, Wayne Davison wrote:
>>>
>>> On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams
>>> >
>>> wrote:
>>>
>>> Rsync passed the backslashes through without dealing with them.
>>>
>>>
>>> Yeah, it only does space-splitting and that's all it will ever do. It
>>> still looks to me like there is a bug in the original escaping, since
>>> any command receiving that string is receiving a backslash that is not
>>> supposed to be there. It should only be escaping the string enough to
>>> get it to rsync, not trying to guess what rsync is going to do with it
>>> after it gets it.
>>>
>>> To work around that bug, you could consider using an ssh-calling shell
>>> script instead of manually specifying the ssh command and args to rsync.
>>>
>>> ..wayne..
>>>
>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. Kevin at FutureQuest.net (work)
>> Orlando, Florida kmk at sanitarium.net (personal)
>> Web page: http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
From space.ship.traveller at gmail.com Sat Oct 29 04:21:47 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Sat, 29 Oct 2016 17:21:47 +1300
Subject: -e escape rule
In-Reply-To:
References:
<637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
Message-ID:
> Yeah, it only does space-splitting and that's all it will ever do. It still looks to me like there is a bug in the original escaping, since any command receiving that string is receiving a backslash that is not supposed to be there. It should only be escaping the string enough to get it to rsync, not trying to guess what rsync is going to do with it after it gets it.
I'm not proposing some additional characters to split on, but quite
the opposite, to handle the backslash escaped spaces correctly and NOT
split. Rest assured, there is no bug with the original escaping. For
your edification:
$ echo \I\'\m\ \a\ \s\t\r\i\n\g
I'm a string
From paul+rsync at wurtel.net Sat Oct 29 12:13:42 2016
From: paul+rsync at wurtel.net (Paul Slootman)
Date: Sat, 29 Oct 2016 14:13:42 +0200
Subject: -e escape rule
In-Reply-To:
References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
Message-ID: <20161029121342.GB7038@msgid.wurtel.net>
On Sat 29 Oct 2016, Samuel Williams wrote:
> I'm not proposing some additional characters to split on, but quite
> the opposite, to handle the backslash escaped spaces correctly and NOT
> split. Rest assured, there is no bug with the original escaping. For
> your edification:
>
> $ echo \I\'\m\ \a\ \s\t\r\i\n\g
> I'm a string
The point is that the original escaping DOUBLE escapes an equals sign:
foo\\\=bar
It shouldn't, there's no reason to.
Paul
From space.ship.traveller at gmail.com Sat Oct 29 12:36:17 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Sun, 30 Oct 2016 01:36:17 +1300
Subject: -e escape rule
In-Reply-To: <20161029121342.GB7038@msgid.wurtel.net>
References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
<20161029121342.GB7038@msgid.wurtel.net>
Message-ID:
> The point is that the original escaping DOUBLE escapes an equals sign:
> foo\\\=bar
> It shouldn't, there's no reason to.
If you paste into your command line:
rsync -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes
The list of arguments would be (i.e. the values in ARGV):
['rsync', '-e', 'ssh -l backup -i /etc/synco/id_rsa -o
ConnectTimeout\=60 -o BatchMode\=yes']
The command
ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes
Is a correct and valid shell command.
When RSync parses this in do_cmd, it should convert the '\=` sequence
into '=' but it doesn't.. This intuition is derived from the fact that
if you instead passed the string to `system('ssh -l backup -i
/etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes')` that the
ARGV generated would be ['ssh', '-l', 'backup', '-i',
'/etc/synco/id_rsa', '-o', 'ConnectTimeout=60', '-o',
'BatchMode=yes'].
Basically, even if there WAS no reason to do so, doesn't mean it's
invalid or undesirable. In theory, even passing in -e \\s\\s\\h should
be valid.
On 30 October 2016 at 01:13, Paul Slootman wrote:
> On Sat 29 Oct 2016, Samuel Williams wrote:
>
>> I'm not proposing some additional characters to split on, but quite
>> the opposite, to handle the backslash escaped spaces correctly and NOT
>> split. Rest assured, there is no bug with the original escaping. For
>> your edification:
>>
>> $ echo \I\'\m\ \a\ \s\t\r\i\n\g
>> I'm a string
>
> The point is that the original escaping DOUBLE escapes an equals sign:
> foo\\\=bar
> It shouldn't, there's no reason to.
>
>
> Paul
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
From wayned at samba.org Sat Oct 29 15:49:37 2016
From: wayned at samba.org (Wayne Davison)
Date: Sat, 29 Oct 2016 08:49:37 -0700
Subject: -e escape rule
In-Reply-To:
References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
<20161029121342.GB7038@msgid.wurtel.net>
Message-ID:
On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams <
space.ship.traveller at gmail.com> wrote:
> The command
>
> ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes
>
> Is a correct and valid shell command.
>
It is, but there is no shell involved, and assuming that a shell is being
used is invalid. Adding backslash escaping now could potentially screw up
anyone currently passing a backslash in their existing rsync scripts (a
small group of people, but probably non-zero). It's tempting to go ahead
and do this, but I think I'll just leave it as it is.
If you want to work around that buggy escaping library, you could change
"ssh" to "ssh-nobs" and put the following into that file:
#!/usr/bin/perl
exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV;
That adds backslash removal on the way to the ssh. It does not allow you
to backslash escape spaces, though, but someone could extend the script to
support that.
..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From wayned at samba.org Sat Oct 29 16:37:53 2016
From: wayned at samba.org (Wayne Davison)
Date: Sat, 29 Oct 2016 09:37:53 -0700
Subject: Links repeatedly being copied???
In-Reply-To: <5803E04A.40201@gmail.com>
References:
<5803E04A.40201@gmail.com>
Message-ID:
On Sun, Oct 16, 2016 at 1:17 PM, B. S. wrote:
> I have for the longest time been seeing '.L..t......'s in my rsync log,
> despite no changes to such in quite some time, and in trying to track it
> down, been baffled.
>
This means that rsync is trying to set the timestamp on the symlink to
match the modify-time on the source symlink. The initial dot tells you
that no "transfer" is happening (i.e. no change in symlink value). Some
filesystems don't allow a symlink's mtime to be set, so rsync can keep
trying to set the time and never be able to succeed. You may want to use
this option: --omit-link-times (-J).
..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From samba-bugs at samba.org Sat Oct 29 21:39:46 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 29 Oct 2016 21:39:46 +0000
Subject: [Bug 12367] temporary lines in --progress output are not cleared
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12367
Wayne Davison changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #2 from Wayne Davison ---
This is apparently only happening with no -r combined with --progress. In this
case there can be a series of file-count outputs (with CR) prior to the start
of the transfer, but no LF before it moves on to other output.
Git commit e02b89d0d fixes this.
--
You are receiving this mail because:
You are the QA Contact for the bug.
From samba-bugs at samba.org Sat Oct 29 22:33:52 2016
From: samba-bugs at samba.org (samba-bugs at samba.org)
Date: Sat, 29 Oct 2016 22:33:52 +0000
Subject: [Bug 12367] temporary lines in --progress output are not cleared
In-Reply-To:
References:
Message-ID:
https://bugzilla.samba.org/show_bug.cgi?id=12367
--- Comment #3 from Wayne Davison ---
Actually, I diagnosed this further, and noticed that the file counts shouldn't
be output at all -- the check for progress output was being fooled by a
temporary setting of the xfer_dirs var when -R is used without -d or -r. I've
changed the show_filelist_p function into a var that is set during the init
call so that it is always evaluated the same way.
This git commit also contains a fix for the newline that could get output with
--quiet: ff66fd4bb
--
You are receiving this mail because:
You are the QA Contact for the bug.
From space.ship.traveller at gmail.com Sun Oct 30 07:06:00 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Sun, 30 Oct 2016 20:06:00 +1300
Subject: -e escape rule
In-Reply-To:
References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk>
<6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net>
<20161029121342.GB7038@msgid.wurtel.net>
Message-ID:
> assuming that a shell is being used is invalid
I never made this assumption. I looked directly at the source code and
I stated that "I feel that this function should also handle backslash
escapes."
I think the assumption that splitting the command works the same way
as (all?) major shells, is not inappropriate given the circumstances,
and it seems like you agree.
> but probably non-zero
Well, I understand where you are coming from. You don't want to break
existing code.
The work-around is to use quotes to escape the white-space sequence. It's okay..
But it's also a surprise that backslash escape sequences don't work
according to intuition of how commands are normally executed. If you
supplied the string in -e to system, it would work as expected..
Unfortunately, this is the default when using Shellwords.join in Ruby.
So, I had to write a custom RSync "join" function to produce an
appropriate command for -e argument.
On 30 October 2016 at 04:49, Wayne Davison wrote:
> On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams
> wrote:
>>
>> The command
>>
>> ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes
>>
>> Is a correct and valid shell command.
>
>
> It is, but there is no shell involved, and assuming that a shell is being
> used is invalid. Adding backslash escaping now could potentially screw up
> anyone currently passing a backslash in their existing rsync scripts (a
> small group of people, but probably non-zero). It's tempting to go ahead and
> do this, but I think I'll just leave it as it is.
>
> If you want to work around that buggy escaping library, you could change
> "ssh" to "ssh-nobs" and put the following into that file:
>
> #!/usr/bin/perl
> exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV;
>
> That adds backslash removal on the way to the ssh. It does not allow you to
> backslash escape spaces, though, but someone could extend the script to
> support that.
>
> ..wayne..
>
From devzero at web.de Mon Oct 31 23:36:22 2016
From: devzero at web.de (devzero at web.de)
Date: Tue, 01 Nov 2016 00:36:22 +0100
Subject: rsync show files changed during transfer - how?
Message-ID: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de>
i'm using rsync for backup and, as rsync can detect if files have vanished during transfer, i wonder how rsync can tell which files got modified during transfer (i.e. which are not consistent on the destination side after transfer)
apparently, rsync can't show that information?
wouldn't that be an extremely useful feature if rsync could do another additional mtime or even checksum comparison after each file transfer?
for example emc networker notifies about "changed during save". i'm curious wg rsync doesn't
regards
roland
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From kmk at sanitarium.net Mon Oct 31 23:56:55 2016
From: kmk at sanitarium.net (Kevin Korb)
Date: Mon, 31 Oct 2016 19:56:55 -0400
Subject: rsync show files changed during transfer - how?
In-Reply-To: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de>
References: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de>
Message-ID: <130c554a-6a20-a023-f82c-6422f07ae7e9@sanitarium.net>
When rsync tells you that a file vanished it is reporting a real error
from the file system. It saw a file that needed to be transferred but
when it attempted to read the file it got a file not found error from
the FS/OS.
There is no equivalent error for "hey, the file you were reading changed
while you were reading it". Rsync would essentially have to read the
file twice and compare.
This is what LVM and similar forms of snapshots are for.
On 10/31/2016 07:36 PM, devzero at web.de wrote:
> i'm using rsync for backup and, as rsync can detect if files have
> vanished during transfer, i wonder how rsync can tell which files got
> modified during transfer (i.e. which are not consistent on the
> destination side after transfer)
>
> apparently, rsync can't show that information?
>
> wouldn't that be an extremely useful feature if rsync could do another
> additional mtime or even checksum comparison after each file transfer?
>
> for example emc networker notifies about "changed during save". i'm
> curious wg rsync doesn't
>
> regards
> roland
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
> gesendet.
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: