From samba-bugs at samba.org Sat Oct 1 13:58:26 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 01 Oct 2016 13:58:26 +0000 Subject: [Bug 12305] New: --fallocate and --sparse works wrong Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 Bug ID: 12305 Summary: --fallocate and --sparse works wrong Product: rsync Version: 3.1.1 Hardware: All OS: Linux Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter: socketpair at gmail.com QA Contact: rsync-qa at samba.org 1.If I specify --fallocate without --sparse: Sparse areas of SRC become fallocated on DST. This is not what I expect. 2. If I specify --sparse without --fallocate: Fallocated areas of SRC become sparse areas on DST, meaning that very important "attribute" is copied. 3. If I specify both --sparse and --fallocate: Fallocate will win. see 1. 4. If I specify none of these flags: sparse/preallocate information will be lost on DST. This all is very sad. So, during synchronisation, rsync should send type of the each area, which should be on of: 1) containig user-data 2) sparse (contains zeroes) 3) preallocated (contain zeroes) And receiver should apply that information on target file. Note, that plain file may have preallocated space AFTER END OF FILE -- it is not a bug, and should be transferred. Also, receiver side may not handle all that features, so it should be controllable what rsync receiver will do if making sparse or preallocated area fails with EOPNOTSUPP or so. Detecting holes and preallocated areas may give wrong information since some FSes fake answer. So it should be controllable see: man 2 fallocate for possible modern operations. man 2 lseek: detecting that areas (SEEK_HOLE/SEEK_DATA) -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 1 14:00:02 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 01 Oct 2016 14:00:02 +0000 Subject: [Bug 12305] --fallocate and --sparse work wrong In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 --- Comment #1 from Коренберг Марк --- typo: very important "attribute" is NOT copied -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 1 19:20:30 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 01 Oct 2016 19:20:30 +0000 Subject: [Bug 12305] --fallocate and --sparse work wrong In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 Andrey Gursky changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrey.gursky at e-mail.ua --- Comment #2 from Andrey Gursky --- Hi Марк, I stumbled over this almost a year ago. See https://bugzilla.samba.org/show_bug.cgi?id=11588 Regards, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 1 22:07:29 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 01 Oct 2016 22:07:29 +0000 Subject: [Bug 12305] --fallocate and --sparse work wrong In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 Wayne Davison changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |DUPLICATE --- Comment #3 from Wayne Davison --- *** This bug has been marked as a duplicate of bug 11588 *** -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 1 22:07:29 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 01 Oct 2016 22:07:29 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 Wayne Davison changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |socketpair at gmail.com --- Comment #1 from Wayne Davison --- *** Bug 12305 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 2 07:33:16 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 02 Oct 2016 07:33:16 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #2 from Коренберг Марк --- Bug 12305 has much more detailed description -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 2 09:45:58 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 02 Oct 2016 09:45:58 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #3 from Andrey Gursky --- (In reply to Коренберг Марк from comment #2) Марк, nevertheless it is still the same issue. You're welcome to add the details here. Please, be sure, to fix the typo: instead of --fallocate it should be --preallocate. Thanks, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Mon Oct 3 19:27:52 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Mon, 03 Oct 2016 19:27:52 +0000 Subject: [Bug 12336] New: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE Tech Support number Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12336 Bug ID: 12336 Summary: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE Tech Support number Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter: erajasthan778 at gmail.com QA Contact: rsync-qa at samba.org ❶❽667698127 SKYPE phone number 18667698127 SKYPE customer service phone number 18667698127 SKYPE desktop support, SKYPE laptop customer service, contact SKYPE support, SKYPE pc support, SKYPE laptop customer care number 18667698127, SKYPE support for  , SKYPE customer care, SKYPE customer care phone number 18667698127, hewlett packard help, phone number 18667698127 for SKYPE, SKYPE online help, SKYPE laptop customer care, SKYPE helpline phone number 18667698127, SKYPE customer support, SKYPE technical support chat, SKYPE computer help, SKYPE support number 18667698127s, SKYPE technical support contact number 18667698127, SKYPE telephone number 18667698127, SKYPE technical support phone number 18667698127, SKYPE helpline, SKYPE support  , SKYPE support online, SKYPE contact number 18667698127, SKYPE help phone number 18667698127, SKYPE customer care 18667698127SKYPE helpline, SKYPE telephone support, SKYPE online support, SKYPE support contact,SKYPE chat support, hewlett packard phone number, SKYPE 18667698127 customer service, SKYPE tech support number,18667698127 SKYPE product support, hewlett  packard customer service phone number18667698127,SKYPE computer support number, SKYPE support  contact number,18667698127 SKYPE support  , SKYPE computer support, SKYPE tech support chat,18667698127 1888   468   0321SKYPE helpline number, 18667698127 SKYPE laptop support, hewlett packard tech support, SKYPE online chat, hewlett packard technical support, SKYPE help line, phone number for SKYPE support, hewlett packard support phone number, SKYPE technical support, hewlett packard customer service number,1888   468   0321 SKYPE service number, hewlett packard 18667698127 helpline, SKYPE customer care no, SKYPE customer service number, SKYPE help number, 18667698127 SKYPE customer service phone number, SKYPE  number, SKYPE support phone, SKYPE support line, hewlett packard contact number, SKYPE tech support phone number, SKYPE customer support phone number, SKYPEs help, call SKYPE support, ##SKYPE support## chat, hewlett packard support number, hewlett packard tech support number, SKYPE support telephone number, hewlett packard tech support phone number, call SKYPE, SKYPE contact support, hewlett packard technical support phone number, SKYPE support centre, hewlett packard customer support, SKYPE desktop support, SKYPE laptop customer service, contact SKYPE support, SKYPE pc support, SKYPE laptop customer care number, SKYPE support for  s, SKYPE customer care, SKYPE customer care phone number, hewlett packard help, phone number for SKYPE, SKYPE online help, SKYPE laptop customer care, SKYPE helpline phone number, SKYPE customer support, SKYPE technical support chat, SKYPE computer help, SKYPE support numbers, SKYPE technical support contact number, SKYPE telephone number, SKYPE technical support phone number, SKYPE helpline, SKYPE support  s, SKYPE support online, SKYPE contact number, SKYPE help phone number, SKYPE customer care number, contact hewlett packard by phone, SKYPE phone support, hewlett packard  s support, SKYPE tech support phone, SKYPE technical help, SKYPE laptop tech support number, contact SKYPE by phone, SKYPE support call, SKYPE computers support, hewlett packard customer service telephone number, phone number for hewlett packard, SKYPE online support chat, SKYPE laptop customer service number, SKYPE online chat support, SKYPEs customer service, hewlett packard customer service phone, SKYPE laptop tech support, SKYPE service phone number, hewlett packard   help, phone number for SKYPE  s, SKYPE troubleshooting phone number, SKYPE  number,http://bit.ly/2bwGOcj hewlett packard technical support number, contact SKYPE support phone, phone number for SKYPE support, SKYPE customer support chat, SKYPE help and support number, contact hewlett packard, SKYPE laptop support phone number, SKYPEs customer service phone number, SKYPE laptop customer service phone number, SKYPE computer support phone number, SKYPE pavilion support, SKYPE computer customer service, SKYPE customer services, hewlett packard telephone number, SKYPE helpline no, SKYPE help desk number, contact SKYPE support phone number, hewlett packard contact, SKYPE phone numbers, SKYPEs customer care number, SKYPE help and support, contact SKYPE technical support, SKYPE contact numbers, contact SKYPE support chat, call SKYPE tech support, SKYPE customer service phone, SKYPE help support, SKYPE computer tech support, SKYPE assistance phone number, SKYPE customer service telephone number, hewlett packard   support phone number, SKYPE contact support number, SKYPE support center phone number, SKYPE support phone numbers, tech support for SKYPE, SKYPE it support, SKYPE laptop helpline, SKYPE technical, SKYPE laptop technical support number, SKYPEs tech support phone number, SKYPEs support phone number, hewlett packard help desk phone number, SKYPE computer tech support phone number, SKYPE customer service number for laptop, SKYPE -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Fri Oct 7 12:20:17 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 07 Oct 2016 12:20:17 +0000 Subject: [Bug 12367] New: temporary lines in --progress output are not cleared Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12367 Bug ID: 12367 Summary: temporary lines in --progress output are not cleared Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: minor Priority: P5 Component: core Assignee: wayned at samba.org Reporter: paul at debian.org QA Contact: rsync-qa at samba.org A bug report received by Debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749165). I can verify this. emit_filelist_progress() in flist.c needs something like a "output_needs_newline_maybe" variable... I can't quite see how to implement that cleanly. >>--- Some temporary lines output by --progress are not cleared, so that the output looks strange. Here's a testcase. ------------------------------------------------------------ #!/bin/sh set -e export LC_ALL=C t=$HOME/rsync-test mkdir "$t" cd "$t" if [ ! -d a1 ]; then mkdir a1 touch a1/foo fi if [ ! -d a2 ]; then mkdir a2 for i in `seq 1 250` do mkdir -p a2/dir$i for j in `seq 1 20` do mktemp --tmpdir=a2/dir$i > /dev/null done done fi mkdir b for i in 1 2 do mkdir -p c/a$i ln -ns ../c/a$i b done rs() { rsync -KRt "$@" a2/*/* a1/foo "localhost:$t/b/" } rs rs --progress rm -r "$t" ------------------------------------------------------------ I get: a1/00 files... a2/ (sometimes without the "a2/" line). If I redirect the output to some file, I can see with "less": 0 files...^M 400 files...^M 2500 files...^M 4600 files...^Ma1/ a2/ The problem is that when "a1/" is output, the previous " 4600 files..." line was not cleared and one can still see the end of it. <<--- I can verify this. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 00:19:05 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 00:19:05 +0000 Subject: [Bug 12367] temporary lines in --progress output are not cleared In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12367 --- Comment #1 from Andrey Gursky --- Hi Paul, you reported something similar on the list (https://lists.samba.org/archive/rsync/2015-December/030478.html). I've instantly recalled that, because this issue also annoys me. If I redirect rsync output via pipe and tee the resulting file is polluted with percentage progress info. Regards, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 16:12:14 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 16:12:14 +0000 Subject: [Bug 12305] --fallocate and --sparse work wrong In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 --- Comment #4 from Carson Gaspar --- Please define what you mean by 'fallocated' in (1) and (2). Please also specify how you're determining that something has been 'fallocated'. I agree that (3) is a bug, and as the only real one that I can see, and is correctly marked a s a dup. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 16:44:44 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 16:44:44 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #4 from Carson Gaspar --- rsync currently just has the receiver turn "long" sequences of zeroes into sparse regions when --sparse is specified. If --preallocate is also specified, what would you like rsync to do? No wire protocol change required (please pick which behaviour you prefer even if you'd rather a protocol bump, as we may negotiate an older wire protocol): 1) Emit an error (as with --in-place and --sparse) 2) Disable one of the options (which one?) 3) Pre-allocate the file, but when zero regions are detected then ftruncate() it and create the sparse region. Reallocate the rest of the file space after creating the sparse region or not? (IFF receiver is on Linux and on a supported filesystem, fallocate(,FALLOC_FL_PUNCH_HOLE,...) could be used to create sparse regions without truncating the file) Wire protocol change required, have the sender determine the sparse regions, using SEEK_HOLE if available, otherwise scanning for all-zero regions: A) Preallocate 1st data region, create 1st sparse region, preallocate 2nd data region, ... -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 16:53:06 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 16:53:06 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #5 from Carson Gaspar --- (In reply to Carson Gaspar from comment #4) Actually, you never want the sender to scan for zero regions if SEEK_HOLE isn't supported, as performance would then be terrible. And a given filesystem may not support SEEK_HOLE, even if lseek() does. So we're really back to picking (1), (2), or (3), as I don't see unreliable sender sparse mapping as sensible (although using SEEK_HOLE to save source file read time and provide a hint to the receiver may be nice). -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 17:09:20 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 17:09:20 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #6 from Carson Gaspar --- For punching holes, Solaris and UnixWare support F_FREESP(64) in fcntl(). Windows supports both reporting and punching holes, but I don't know if cygwin (or any other rsync on windows platform) implements it. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 18:18:13 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 18:18:13 +0000 Subject: [Bug 11588] missing option: preallocate for all files except for sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #7 from Wayne Davison --- Created attachment 12556 --> https://bugzilla.samba.org/attachment.cgi?id=12556&action=edit Preliminary patch to support punching holes In my testing, using both a pre-allocate call on a file followed by a hole-punch call has no effect on the allocation of the blocks (though it does zero them). I tested --sparse and --inplace with this, and it worked fine on one system (with a new enough linux kernel). There are cases where the sparseness will be lost, though, depending on OS & filesystem. I'm thinking we just update the docs to mention that if you combine --sparse with --inplace (and/or --preallocate) that you might not get the sparseness preserved. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 19:44:32 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 19:44:32 +0000 Subject: [Bug 12305] --fallocate and --sparse work wrong In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12305 --- Comment #5 from Коренберг Марк --- Fallocated: areas of the file that has been fallocate()d, but stillnot written. Technically, on sender, even written parts that was written, but contain zeroes may be considered as fallocated areas. I mean that receiver should call fallocate() on that region instead writing zeroes. How to determine: How to determine if area is alocated: lseek() + SEEK_DATA/SEEK_HOLE. And after that, check if non-sparse area contains zeroes. Another way - is to examine fiemap ( https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt ) -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 20:10:52 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 20:10:52 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #8 from Andrey Gursky --- (In reply to Wayne Davison from comment #7) Wayne, since this bug made rsync unusable for me, I fixed that and implemented additional checks needed for ext4 a month or two after I reported this bug and saw no reaction at all. Now a couple of people got interested and you also, so I can share my work. It's not tiny. > In my testing, using both a pre-allocate call on a file followed by a > hole-punch call has no effect on the allocation of the blocks (though it does > zero them). Yes, this is tricky. Hole-punch works only for full filesystem blocks (e.g., default 4K). Issuing few partial hole-punch requests wouldn't work, even if they cover the whole block. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 21:03:22 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 21:03:22 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #9 from Wayne Davison --- > Hole-punch works only for full filesystem blocks That has nothing to do with it. If you fallocate() the full file length and then (on the same file handle) try to punch out parts of the allocated file, no blocks change away from becoming allocated. Looks like a bug in Linux to me. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 21:05:32 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 21:05:32 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #10 from Wayne Davison --- > ... I can share my work. Sounds interesting! Looking forward to seeing what you've come up with. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 21:28:49 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 21:28:49 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #11 from Theodore Ts'o --- Re: #9. I'm not able to reproduce the described behavior. If you want to follow up on what you think is a kernel bug, please send a simple repro program or script and what version of the kernel you are using to the linux-ext4 mailing list. Thanks!! Cheers, {/usr/projects/docker/dropbox} (master) 1009% fallocate -o 0 -l 128M test.file {/usr/projects/docker/dropbox} (master) 1010% filefrag -v test.file Filesystem type is: ef53 File size of test.file is 134217728 (32768 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 4095: 55990272.. 55994367: 4096: unwritten 1: 4096.. 8191: 56025088.. 56029183: 4096: 55994368: unwritten 2: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten 3: 10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten 4: 14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten 5: 16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten 6: 20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten 7: 28672.. 32767: 56352768.. 56356863: 4096: 56313856: last,unwritten,eof test.file: 8 extents found {/usr/projects/docker/dropbox} (master) 1011% xfs_io -c "fpunch 65536 65536" test.file {/usr/projects/docker/dropbox} (master) 1012% filefrag -v test.file Filesystem type is: ef53 File size of test.file is 134217728 (32768 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 15: 55990272.. 55990287: 16: unwritten 1: 32.. 4095: 55990304.. 55994367: 4064: unwritten 2: 4096.. 8191: 56025088.. 56029183: 4096: 55994368: unwritten 3: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten 4: 10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten 5: 14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten 6: 16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten 7: 20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten 8: 28672.. 32767: 56352768.. 56356863: 4096: 56313856: last,unwritten,eof test.file: 8 extents found {/usr/projects/docker/dropbox} (master) 1013% uname -a Linux callcc 4.8.0-00041-gecd2f69 #3 SMP Mon Oct 3 02:56:05 EDT 2016 x86_64 GNU/Linux -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 22:32:09 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 22:32:09 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #12 from Andrey Gursky --- (In reply to Wayne Davison from comment #9) >> Hole-punch works only for full filesystem blocks > That has nothing to do with it. Wayne, OK, might be. I haven't tested it the exactly way you're doing it now, since I did it other way. But this was important to take care in order to fix the bugs related to sparse with inplace [1], [2]. But due to missing response I've hold back the fix (not yet throughly tested) for me. [1] About data/token send/receive protocol part and more https://lists.samba.org/archive/rsync/2015-December/030471.html [2] Status of --inplace and --sparse in rsync or alternative? https://lists.samba.org/archive/rsync/2015-December/030472.html -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 22:36:56 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 22:36:56 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #13 from Andrey Gursky --- (In reply to Theodore Ts'o from comment #11) Theo, I believe "on the same file handle" is the unusual prerequisite to trigger the behavior described by Wayne. Or such a test is already contained in e2fsprogs (in a C test program, not a shell script)? Regards, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 23:30:23 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 23:30:23 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #14 from Theodore Ts'o --- >I believe "on the same file handle" is the unusual prerequisite to trigger the >behavior described by Wayne. I was fairly sure that was a red herring, so I was trying to save myself some time, but no, it doesn't replicate even if you use the same file descriptor.... {/usr/projects/linux/ext4} (origin) 1009% strace /tmp/test-fallocate execve("/tmp/test-fallocate", ["/tmp/test-fallocate"], [/* 64 vars */]) = 0 brk(NULL) = 0x187e000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20e7d2a000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=101609, ...}) = 0 mmap(NULL, 101609, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f20e7d11000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\3\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1685264, ...}) = 0 mmap(NULL, 3791264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f20e776d000 mprotect(0x7f20e7902000, 2093056, PROT_NONE) = 0 mmap(0x7f20e7b01000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x194000) = 0x7f20e7b01000 mmap(0x7f20e7b07000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f20e7b07000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20e7d0f000 arch_prctl(ARCH_SET_FS, 0x7f20e7d0f700) = 0 mprotect(0x7f20e7b01000, 16384, PROT_READ) = 0 mprotect(0x7f20e7d2d000, 4096, PROT_READ) = 0 munmap(0x7f20e7d11000, 101609) = 0 open("test-file", O_WRONLY|O_CREAT|O_TRUNC, 0700) = 3 fallocate(3, 0, 0, 1048576) = 0 fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 32768, 32768) = 0 close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ {/usr/projects/linux/ext4} (origin) 1010% filefrag -v test-file Filesystem type is: ef53 File size of test-file is 1048576 (256 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 7: 62130432.. 62130439: 8: unwritten 1: 16.. 255: 62130448.. 62130687: 240: last,unwritten,eof test-file: 1 extent found -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 23:31:03 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 23:31:03 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #15 from Theodore Ts'o --- Created attachment 12557 --> https://bugzilla.samba.org/attachment.cgi?id=12557&action=edit Test program to show that fallocate followed by punch hole works just fine.... -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 8 23:47:03 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 08 Oct 2016 23:47:03 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #16 from Andrey Gursky --- (In reply to Theodore Ts'o from comment #15) Theo, thanks for taking time to test it! This works for me too (Debian Testing 4.7.4-2, ext4): $ /usr/sbin/filefrag -v test-file Filesystem type is: ef53 File size of test-file is 1048576 (256 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 7: 14482176.. 14482183: 8: unwritten 1: 16.. 255: 14482192.. 14482431: 240: last,unwritten,eof test-file: 1 extent found $ -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 04:15:11 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 04:15:11 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #17 from Wayne Davison --- Take the test program and change the SYS_fallocate to use the FALLOC_FL_KEEP_SIZE flag (don't forget to "rm test-file") and it will fail. Rsync always pre-allocates with FALLOC_FL_KEEP_SIZE when the flag is available. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 04:16:40 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 04:16:40 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #18 from Wayne Davison --- FYI, I tested on Linux 4.2.0 and 3.10.0 (I don't have a newer kernel running here to try). -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 04:27:27 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 04:27:27 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #19 from Wayne Davison --- Also, to be more like rsync would do you can follow the hole-punch with a seek and a write so that the file ends up with a non-zero size. Apparently if I change the order to do the seek & the write first and THEN punch hole it works. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 13:15:38 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 13:15:38 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #20 from Andrey Gursky --- (In reply to Wayne Davison from comment #17) >From what I see, it doesn't fail, since the file is not preallocated at all with FALLOC_FL_KEEP_SIZE, but just a fully sparse file is created (consisting of only one big hole): $ ls -ls test-file 1024 -rwx------ 1 andrey andrey 0 Oct 9 14:47 test-file $ /usr/sbin/filefrag -v test-file Filesystem type is: ef53 File size of test-file is 0 (0 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 255: 14828800.. 14829055: 256: last,unwritten,eof test-file: 1 extent found -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 13:53:57 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 13:53:57 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #21 from Andrey Gursky --- (In reply to Andrey Gursky from comment #20) Sorry, it is indeed preallocated. Other still holds: hole-punch doesn't fail because the file already consists of only hole. Such file I would call a preallocated data-sparse. In opposite to the usual not preallocated space-sparse with truncate: $ truncate -s 1048576 test-sparse $ ls -ls test-sparse 0 -rw-r--r-- 1 andrey andrey 1048576 Oct 9 15:34 test-sparse $ /usr/sbin/filefrag -v test-sparse Filesystem type is: ef53 File size of test-sparse is 1048576 (256 blocks of 4096 bytes) test-sparse: 0 extents found -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 14:04:07 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 14:04:07 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #22 from Andrey Gursky --- (In reply to Andrey Gursky from comment #21) Continuing to think aloud. It's not really a hole, it's already reserved space. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sun Oct 9 14:54:48 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sun, 09 Oct 2016 14:54:48 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #23 from Wayne Davison --- > Continuing to think aloud. It's not really a hole, it's already reserved space. Exactly, and it's impossible to punch holes in that allocation. I'm changing my patch to give the file a size to deal with this anomaly. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Mon Oct 10 14:38:24 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Mon, 10 Oct 2016 14:38:24 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #24 from Theodore Ts'o --- So a simple workaround would be to use fallocate with KEEP_SIZE at first, then use punch whole, write the blocks, etc., and then use either truncate to set i_size, or seek to the desired size minus one and write a single byte. Seeking to the desired size minus one is more portable, but if you want to avoid allocating an extra 4k block, you could try using truncate, and if that doesn't set i_size (it's not guaranteed by POSIX, but I believe all Linux file systems will set i_size), seeking to size-1 and writing a single zero byte is guaranteed to work. That being said, I agree that ext4 should allow punch hole to work beyond i_size, if there are blocks allocated using fallocate(2). We'll fix that for the future, but for now, the workaround suggested above is probably the simplest way to work around the issue in a way that's compatible with both the current and future behavior. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Mon Oct 10 15:02:33 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Mon, 10 Oct 2016 15:02:33 +0000 Subject: [Bug 11588] better handling for --preallocate with --sparse In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #25 from Коренберг Марк --- What about fallcate()d area beyond file size ? Will they be synchronized ? Just curious. -- You are receiving this mail because: You are the QA Contact for the bug. From kip at thevertigo.com Mon Oct 10 16:24:18 2016 From: kip at thevertigo.com (Kip Warner) Date: Mon, 10 Oct 2016 09:24:18 -0700 Subject: rsync: connection unexpectedly closed Message-ID: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Hey list, I am having problems as of late with my rsync backup. On the client side I am using the following: OPTS="-avvvrz       --compress-level=9       --itemize-changes       --delete       --delete-excluded       --human-readable       --files-from=$FILES       --include-from=$INCLUDES       --exclude-from=$EXCLUDES       --partial       --progress        --owner       --perms       --progress       --timeout=0       --times       --stats" sudo rsync -e "ssh -i ${IDENTITY_FILE} -v -p ${REMOTE_PORT}" $OPTS / $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH Note that I have temporarily disabled timeouts and added extra verbosity. The transfer to the remote host via SSH works fine, up until it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the way through, hangs, and then times out. On the client side I see the following: ... rsync: connection unexpectedly closed (3542035 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1] [sender] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255) On the server side if I attach to the rsync process via strace, I see the following: $ strace -f -p 3095 ... 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971}) 3095 write(1, "\3\0\0\7\1\0\0", 7) = 7 3095 gettimeofday({1476036967, 673095}, NULL) = 0 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999970}) 3095 write(1, "H\0\0\trecv_files(home/kip/.Virtual"..., 76) = 76 3095 gettimeofday({1476036967, 680312}, NULL) = 0 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 1 (in [3], left {40, 364402}) 3095 read(3, "B\0\0\trecv_files(home/kip/.Virtual"..., 8184) = 160 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999973}) 3095 write(1, "B\0\0\trecv_files(home/kip/.Virtual"..., 70) = 70 3095 gettimeofday({1476037227, 506412}, NULL) = 0 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971}) 3095 write(1, "V\0\0\trecv mapped home/kip/.Virtua"..., 90) = 90 3095 gettimeofday({1476037227, 512591}, NULL) = 0 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) ... a couple hundred times or so repeats ... 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0} Note that it looks like the select() call is timing out for what I presume is a regular file descriptor (4 since stdin, stdout, and stderr are 0-3 respectively). This could have nothing to do with rsync at all and could be a file system issue, but I figured I'd ask. The server the data is being uploaded to with the strace running on it has rsync version: $ rsync --version rsync  version 3.0.9  protocol version 30 The client reported: $ rsync --version rsync  version 3.1.1  protocol version 31 Any help appreciated. Regards, -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From kip at thevertigo.com Sun Oct 9 22:07:06 2016 From: kip at thevertigo.com (Kip Warner) Date: Sun, 09 Oct 2016 15:07:06 -0700 Subject: rsync: connection unexpectedly closed Message-ID: <1476050826.1424.19.camel@thevertigo.com> Hey list, I am having problems as of late with my rsync backup. On the client side I am using the following: OPTS="-avvvrz       --compress-level=9       --itemize-changes       --delete       --delete-excluded       --human-readable       --files-from=$FILES       --include-from=$INCLUDES       --exclude-from=$EXCLUDES       --partial       --progress        --owner       --perms       --progress       --timeout=0       --times       --stats" sudo rsync -e "ssh -i ${IDENTITY_FILE} -v -p ${REMOTE_PORT}" $OPTS / $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH Note that I have temporarily disabled timeouts and added extra verbosity. The transfer to the remote host via SSH works fine, up until it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the way through, hangs, and then times out. On the client side I see the following: ... rsync: connection unexpectedly closed (3542035 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1] [sender] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255) On the server side if I attach to the rsync process via strace, I see the following: $ strace -f -p 3095 ... 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971}) 3095 write(1, "\3\0\0\7\1\0\0", 7) = 7 3095 gettimeofday({1476036967, 673095}, NULL) = 0 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999970}) 3095 write(1, "H\0\0\trecv_files(home/kip/.Virtual"..., 76) = 76 3095 gettimeofday({1476036967, 680312}, NULL) = 0 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0}) = 1 (in [3], left {40, 364402}) 3095 read(3, "B\0\0\trecv_files(home/kip/.Virtual"..., 8184) = 160 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999973}) 3095 write(1, "B\0\0\trecv_files(home/kip/.Virtual"..., 70) = 70 3095 gettimeofday({1476037227, 506412}, NULL) = 0 3095 select(4, [3], [1], [1], {60, 0}) = 1 (out [1], left {59, 999971}) 3095 write(1, "V\0\0\trecv mapped home/kip/.Virtua"..., 90) = 90 3095 gettimeofday({1476037227, 512591}, NULL) = 0 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) ... a couple hundred times or so repeats ... 3095 select(4, [3], [], NULL, {60, 0}) = 0 (Timeout) 3095 select(4, [3], [], NULL, {60, 0} Note that it looks like the select() call is timing out for what I presume is a regular file descriptor (4 since stdin, stdout, and stderr are 0-3 respectively). This could have nothing to do with rsync at all and could be a file system issue, but I figured I'd ask. The server the data is being uploaded to with the strace running on it has rsync version: $ rsync --version rsync  version 3.0.9  protocol version 30 The client reported: $ rsync --version rsync  version 3.1.1  protocol version 31 Any help appreciated. Regards, -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From henri.shustak at gmail.com Wed Oct 12 00:30:03 2016 From: henri.shustak at gmail.com (Henri Shustak) Date: Wed, 12 Oct 2016 13:30:03 +1300 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Message-ID: > I am having problems as of late with my rsync backup. Have you tried performing a copy to a known good local device? If a local copy fails, then I would start checking the file system of the source and also the hardware of that system. ---------------------------- HTRAX : http://www.htrax.xyz From paul+rsync at wurtel.net Wed Oct 12 06:36:54 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Wed, 12 Oct 2016 08:36:54 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Message-ID: <20161012063654.GA11197@msgid.wurtel.net> On Mon 10 Oct 2016, Kip Warner wrote: > > The server the data is being uploaded to with the strace running on it > has rsync version: > > $ rsync --version > rsync  version 3.0.9  protocol version 30 > > The client reported: > > $ rsync --version > rsync  version 3.1.1  protocol version 31 As always it's best to first upgrade to the current version (3.1.3) if at all possible, as there's always the chance that the cause of your problems has already been fixed. Paul From kip at thevertigo.com Wed Oct 12 16:48:18 2016 From: kip at thevertigo.com (Kip Warner) Date: Wed, 12 Oct 2016 09:48:18 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Message-ID: <1476290898.27406.119.camel@thevertigo.com> On Wed, 2016-10-12 at 13:30 +1300, Henri Shustak wrote: > Have you tried performing a copy to a known good local device?  If a > local copy fails, then I would start checking the file system of the > source and also the hardware of that system. That's a good idea. I just tried that and it copied no problem. -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From kip at thevertigo.com Thu Oct 13 03:30:56 2016 From: kip at thevertigo.com (Kip Warner) Date: Wed, 12 Oct 2016 20:30:56 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161012063654.GA11197@msgid.wurtel.net> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> Message-ID: <1476329456.27406.168.camel@thevertigo.com> On Wed, 2016-10-12 at 08:36 +0200, Paul Slootman wrote: > As always it's best to first upgrade to the current version (3.1.3) > if at all possible, as there's always the chance that the cause of > your problems has already been fixed. Good call, but I believe I may have ruled this out. I didn't upgrade to 3.1.3, but both sides are running 3.1.1 protocol version 31 now. Same problem.  I think the key insight was in the strace log which showed the select() call was timed out. If I knew what type of file descriptor it was being fed, I might have a clue. It might have been a socket or something on disk. I don't know. -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From paul+rsync at wurtel.net Thu Oct 13 05:51:32 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Thu, 13 Oct 2016 07:51:32 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476329456.27406.168.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> Message-ID: <20161013055132.GA18767@msgid.wurtel.net> On Wed 12 Oct 2016, Kip Warner wrote: > > I think the key insight was in the strace log which showed the select() > call was timed out. If I knew what type of file descriptor it was being > fed, I might have a clue. It might have been a socket or something on > disk. I don't know. You can use lsof -p $pid to show what files that process has opened. On linux you can also use 'ls -l /proc/$pid/fd'. Paul From dhoworth at mrc-lmb.cam.ac.uk Thu Oct 13 09:09:04 2016 From: dhoworth at mrc-lmb.cam.ac.uk (Dave Howorth) Date: Thu, 13 Oct 2016 10:09:04 +0100 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Message-ID: On 2016-10-10 17:24, Kip Warner wrote: > Note that I have temporarily disabled timeouts and added extra > verbosity. The transfer to the remote host via SSH works fine, up until > it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the > way through, hangs, and then times out. I have a similar but different problem. I make a regular download from a site that always errors out on a particular large file. However, my rsync error symptoms are different. Unfortunately, the server admins seem to be the strong, silent types who have repeatedly changed their minds about what they think is wrong and who may or may not be attempting to solve the problem in isolation - I've failed to get any meaningful communication going with them. Fortunately, excluding the problem file is a reasonable workaround for me. Have you tried excluding the problem file from the transfer? One possibility is that the problem is not caused directly by rsync but because of some underlying filesystem glitch. What OS & filesystems are you using? Cheers, Dave From samba-bugs at samba.org Thu Oct 13 15:45:14 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Thu, 13 Oct 2016 15:45:14 +0000 Subject: [Bug 12378] New: why i cannot exclude dir/files if using option "--files-from" Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 Bug ID: 12378 Summary: why i cannot exclude dir/files if using option "--files-from" Product: rsync Version: 3.1.2 Hardware: x64 OS: Linux Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter: isaja at sissa.it QA Contact: rsync-qa at samba.org I don't understand if this is intended or not, but when I use "--files-from" and "--exclude" options together all files are synced anyhow. eg.: rsync -av -c --delete --relative --delete-excluded --exclude=".cache" --exclude="mydirToExclude" /home . it correctly doesn't copy the .cache folder. but: rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt --exclude=".cache" --exclude="mydirToExclude" /home . does copy everything. instead using a pattern such as --exclude="**myDirToExclude**" doesn't copy all lines with that pattern myFileList.txt is something like: userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55 userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15 userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe userName/.cache/winetricks/dotnet30/dotnetfx3.exe userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe userName/.cache/winetricks/msls31/InstMsiW.exe userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html Thanks, alessio -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Thu Oct 13 16:24:43 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Thu, 13 Oct 2016 16:24:43 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #1 from Kevin Korb --- It did not copy the directory you excluded it copied the files within that directory that you explicitly told it to copy and created the appropriate directories to allow that to happen. IOW, .cache is not relative to userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15 -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Thu Oct 13 22:04:15 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Thu, 13 Oct 2016 22:04:15 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #2 from Alessio --- (In reply to Kevin Korb from comment #1) Excuse me, but i don't get it, what's the difference between excluding (matching) the pattern from a --files-from and a remote host? debugging the rsync transfer from the remote host I get: [sender] hiding directory home/userName/.cache because of pattern .cache in the other case nothing is being hidden instead but .cache is inside my path: userName/.cache/winetricks/msls31/InstMsiW.exe -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Thu Oct 13 22:13:46 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Thu, 13 Oct 2016 22:13:46 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #3 from Kevin Korb --- I didn't say anything specific about a remote host. You excluded .cache then you told it to copy specific files some of which are in .cache. If there was a file named .cache inside of a directory that you told it to copy then it would be excluded. If you excluded userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded. Exclude is not patern matched against parts of things you explicitly tell rsync to copy... kmk at dementia[1%]> cd /tmp kmk at dementia[2%]> mkdir test kmk at dementia[3%]> cd !$ cd test kmk at dementia[4%]> mkdir src dst kmk at dementia[5%]> cd src kmk at dementia[6%]> mkdir -p a b c d e cache/c kmk at dementia[7%]> cd .. kmk at dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" > list kmk at dementia[9%]> cat list a b c d e cache/c kmk at dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/ building file list ... done cd+++++++++ a/ cd+++++++++ b/ cd+++++++++ c/ cd+++++++++ cache/ cd+++++++++ cache/c/ cd+++++++++ d/ cd+++++++++ e/ sent 157 bytes received 37 bytes 388.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) kmk at dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/ building file list ... done cd+++++++++ a/ cd+++++++++ b/ cd+++++++++ d/ cd+++++++++ e/ sent 106 bytes received 28 bytes 268.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) kmk at dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/ dst/ building file list ... done cd+++++++++ a/ cd+++++++++ b/ cd+++++++++ c/ cd+++++++++ d/ cd+++++++++ e/ sent 121 bytes received 31 bytes 304.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) Cache isn't really being copied it is just being created to make a place for cache/c to be stored when cache/c or c is not excluded. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Thu Oct 13 23:51:07 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Thu, 13 Oct 2016 23:51:07 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 Wayne Davison changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |WORKSFORME Status|NEW |RESOLVED --- Comment #4 from Wayne Davison --- Command args (which includes names inside a files-from file) are never excluded by an exclude directive. You told rsync to copy it, so it copies it. Excludes only affect matching of files that rsync finds inside directories that you told it to copy. It is expected that you already trimmed any unwanted files/dirs from your args before you called rsync, since those names are under your control. -- You are receiving this mail because: You are the QA Contact for the bug. From wayned at samba.org Thu Oct 13 23:58:49 2016 From: wayned at samba.org (Wayne Davison) Date: Thu, 13 Oct 2016 16:58:49 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476329456.27406.168.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> Message-ID: On Wed, Oct 12, 2016 at 8:30 PM, Kip Warner wrote: > I think the key insight was in the strace log which showed the > select() call was timed out. No, that's totally expected. While select() is waiting for I/O to arrive, it returns to rsync every 60 seconds to allow it to decide if it wants to continue waiting or do something else. You have to find the process that is exiting first to discover why the connection is closing. This assumes that it's not a network issue, where all of the programs get a closed-connection error. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmk at sanitarium.net Fri Oct 14 00:08:43 2016 From: kmk at sanitarium.net (Kevin Korb) Date: Thu, 13 Oct 2016 20:08:43 -0400 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476403504.18135.31.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> <1476403504.18135.31.camel@thevertigo.com> Message-ID: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> I don't remember whether or not you said you were running rsync over ssh but if you are you can also debug the ssh layer. You can even do it at both ends.... On the server run a debugging sshd on an alternate port with: /usr/sbin/sshd -dDp222 (note that this will only accept 1 connection, debug to the terminal, then exit) then on the client use rsync -e "ssh -vp222" to go verbose and use the debugging port. On 10/13/2016 08:05 PM, Kip Warner wrote: > On Thu, 2016-10-13 at 16:58 -0700, Wayne Davison wrote: >> No, that's totally expected. While select() is waiting for I/O to >> arrive, it returns to rsync every 60 seconds to allow it to decide if >> it wants to continue waiting or do something else. You have to find >> the process that is exiting first to discover why the connection is >> closing. This assumes that it's not a network issue, where all of the >> programs get a closed-connection >> error. > > Hey Wayne, > > I took Paul's advice and tried monitoring with lsof. This is what I > saw: > > $ lsof -p 3104 > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > rsync 3104 kip cwd DIR 0,19 4096 214695958 /home/kip/Disk_Backups/kip-desktop/yakkety > rsync 3104 kip rtd DIR 179,2 4096 2 / > rsync 3104 kip txt REG 179,2 460584 7817 /usr/bin/rsync > rsync 3104 kip mem REG 179,2 42692 3974 /lib/arm-linux-gnueabihf/libnss_files-2.13.so > rsync 3104 kip mem REG 179,2 71624 3971 /lib/arm-linux-gnueabihf/libnsl-2.13.so > rsync 3104 kip mem REG 179,2 38608 3976 /lib/arm-linux-gnueabihf/libnss_nis-2.13.so > rsync 3104 kip mem REG 179,2 26484 3972 /lib/arm-linux-gnueabihf/libnss_compat-2.13.so > rsync 3104 kip mem REG 179,2 2953776 18967 /usr/lib/locale/locale-archive > rsync 3104 kip mem REG 179,2 1216624 3965 /lib/arm-linux-gnueabihf/libc-2.13.so > rsync 3104 kip mem REG 179,2 130448 41964 /lib/arm-linux-gnueabihf/libgcc_s.so.1 > rsync 3104 kip mem REG 179,2 43024 10745 /lib/arm-linux-gnueabihf/libpopt.so.0.0.0 > rsync 3104 kip mem REG 179,2 26240 1840 /lib/arm-linux-gnueabihf/libacl.so.1.1.0 > rsync 3104 kip mem REG 179,2 17904 1854 /lib/arm-linux-gnueabihf/libattr.so.1.1.0 > rsync 3104 kip mem REG 179,2 10170 28719 /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so > rsync 3104 kip mem REG 179,2 126236 3961 /lib/arm-linux-gnueabihf/ld-2.13.so > rsync 3104 kip 1w FIFO 0,10 0t0 8455 pipe > rsync 3104 kip 2w FIFO 0,10 0t0 8456 pipe > rsync 3104 kip 3u unix 0xdbda86c0 0t0 8466 socket > > And on the strace... > > ... > 3104 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left {59, 999978}) > 3104 write(1, "h\340\0\7\0y@\312\227\241\236\255\367\312\3637\323X\206\314\250\372\362P1\202\374\"i\265c\16"..., 57452) = 57452 > 3104 select(4, [3], [], [3], {60, 0}) = 0 (Timeout) > (repeats above line until client (sender) reports connection timed out) > > Sender reports the following until it times out... > > ... > match at 32949141504 last_match=32949141504 j=251382 len=131072 n=0 > 32.95G 96% 83.55MB/s 0:00:12 > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From kip at thevertigo.com Fri Oct 14 01:27:42 2016 From: kip at thevertigo.com (Kip Warner) Date: Thu, 13 Oct 2016 18:27:42 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> Message-ID: <1476408462.18135.35.camel@thevertigo.com> On Thu, 2016-10-13 at 10:09 +0100, Dave Howorth wrote: > Have you tried excluding the problem file from the transfer? Hey Dave. All the other files appear to sync, up until it gets to that one large file. Then it stalls, and finally times out. I could tell it to exclude that important file, but that would defeat the purpose of my backup. > One possibility is that the problem is not caused directly by rsync > but  because of some underlying filesystem glitch. What OS & > filesystems are you using? That could well be, but how to know? Client side: $ lsb_release -a LSB Version: security-9.20160110ubuntu5-amd64:security-9.20160110ubuntu5-noarch Distributor ID: Ubuntu Description: Ubuntu 16.10 Release: 16.10 Codename: yakkety Server side: $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 7.11 (wheezy) Release: 7.11 Codename: wheezy -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From samba-bugs at samba.org Fri Oct 14 09:20:04 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 14 Oct 2016 09:20:04 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #5 from Alessio --- (In reply to Wayne Davison from comment #4) Well if it's by design I will not go further. But for me it's a missing "feature" because one has to prepare the list beforehand while rsync could do the job "while it's there". It's like the impossibility to sync files newer than specific date, which I really miss, but it's another story. Thanks for your help! (In reply to Kevin Korb from comment #3) Thanks for the detailed example and explanation it's exactly what happens to me. And if you remove the "--files-from" you see that "cache" folder is really excluded from the copy. I was wondering why the behavior is different applying the "--files-from" or not and unfortunately WayneD said it's by design. Thanks for your help -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Fri Oct 14 16:57:33 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 14 Oct 2016 16:57:33 +0000 Subject: [Bug 12378] why i cannot exclude dir/files if using option "--files-from" In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #6 from Kevin Korb --- If you want to sync files newer than say 3 days ago that is what --files-from is for... cd /source find . -mtime -3 -print | rsync -vai --files-from=- . /target The primary purpose of --files-from is to give rsync the power of find. -- You are receiving this mail because: You are the QA Contact for the bug. From kip at thevertigo.com Sat Oct 15 07:02:16 2016 From: kip at thevertigo.com (Kip Warner) Date: Sat, 15 Oct 2016 00:02:16 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> <1476403504.18135.31.camel@thevertigo.com> <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> Message-ID: <1476514936.31856.8.camel@thevertigo.com> On Thu, 2016-10-13 at 20:08 -0400, Kevin Korb wrote: > I don't remember whether or not you said you were running rsync over > ssh but if you are you can also debug the ssh layer.  You can even do > it at both ends.... Yes, I'm doing this over an SSH tunnel. > On the server run a debugging sshd on an alternate port with: > /usr/sbin/sshd -dDp222 > (note that this will only accept 1 connection, debug to the terminal, > then exit) then on the client use rsync -e "ssh -vp222" to go verbose > and use the debugging port. This is a great idea except that I only have one port forwarded to the server (beyond my control) and that port is for the current SSH session I would need to spawn another one on some other inaccessible port. -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From kip at thevertigo.com Sun Oct 16 02:54:41 2016 From: kip at thevertigo.com (Kip Warner) Date: Sat, 15 Oct 2016 19:54:41 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> <1476403504.18135.31.camel@thevertigo.com> <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> Message-ID: <1476586481.31856.31.camel@thevertigo.com> On Thu, 2016-10-13 at 20:08 -0400, Kevin Korb wrote: > I don't remember whether or not you said you were running rsync over > ssh but if you are you can also debug the ssh layer.  You can even do > it at both ends.... > > On the server run a debugging sshd on an alternate port with: > /usr/sbin/sshd -dDp222 > (note that this will only accept 1 connection, debug to the terminal, > then exit) then on the client use rsync -e "ssh -vp222" to go verbose > and use the debugging port. Hey Kevin, I managed to get a chance to try your suggestion. I noticed something very interesting. On the server side, there's actually two processes of rsync that appear to be spawned by the client's connection.  Running the ssh daemon on the server side, I notice that long after the server dies like so... ... debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req exec Read error from remote host : Connection reset by peer debug1: do_cleanup debug1: do_cleanup debug1: PAM: cleanup debug1: PAM: closing session Sessions still open, not unmounting debug1: PAM: deleting credentials ...and the client rsync dies... match at 32949010432 last_match=32949010432 j=251381 len=131072 n=0 match at 32949141504 last_match=32949141504 j=251382 len=131072 n=0 32.95G 96% 74.13MB/s 0:00:14 packet_write_wait: Connection to port 22223: Broken pipe rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.1] [sender] _exit_cleanup(code=10, file=io.c, line=820): about to call exit(255) ...one of the server side rsync processes according to strace is still alive... write(1, "h\340\0\7\310\376\\\233\227\241\236\255VC\31\351\323X\206\314F$\337\0321\202\374\"\320(d\271"..., 57452) = 57452 select(4, [3], [], [3], {60, 0}) = 0 (Timeout) (last line repeats a couple hundred times) ...and apparently the second rsync server side process is actually still performing reads and writes more than an hour after the connection died... ... read(1, "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\252d'h\351\371\0ny"..., 262144) = 262144 write(3, "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\252d'h\351\371\0ny"..., 262144) = 262144 ... This makes me wonder if this has something to do with memory buffers for file i/o being exhausted on the server and this server side activity long after the connection died is an effort to flush a file buffer to disk. The file this always breaks on is 30+GB, but the server hardware is a small embedded system. -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From kip at thevertigo.com Sun Oct 16 04:55:58 2016 From: kip at thevertigo.com (Kip Warner) Date: Sat, 15 Oct 2016 21:55:58 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476586481.31856.31.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <20161012063654.GA11197@msgid.wurtel.net> <1476329456.27406.168.camel@thevertigo.com> <1476403504.18135.31.camel@thevertigo.com> <86c00cab-d410-6f43-86da-24052036d9ee@sanitarium.net> <1476586481.31856.31.camel@thevertigo.com> Message-ID: <1476593758.31856.33.camel@thevertigo.com> On Sat, 2016-10-15 at 19:54 -0700, Kip Warner wrote: >     ...and apparently the second rsync server side process is > actually >     still performing reads and writes more than an hour after the >     connection died... > >     ... >     read(1, > "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\2 > 52d'h\351\371\0ny"..., 262144) = 262144 >     write(3, > "\337\363\356\1^\26L\316\17\31izD\254\27\346\267\266H\343\223\v\357\2 > 52d'h\351\371\0ny"..., 262144) = 262144 >     ... And it finally just terminated: ... select(5, [0], [4], [0], {60, 0}) = 2 (in [0], out [4], left {59, 999979}) read(0, "\373\375\372\343\276M\337\374'\201\234o\306\34\313\255\274b\16\314\206\262\10\304\7{z-\316N\347\242"..., 16384) = 16384 write(4, "B\0\0\trecv_files(home/kip/.Virtual"..., 160) = 160 select(1, [0], [], [0], {60, 0}) = 1 (in [0], left {59, 999978}) --- SIGUSR1 (User defined signal 1) @ 0 (0) --- rt_sigaction(SIGUSR1, {SIG_IGN, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0 rt_sigaction(SIGUSR2, {SIG_IGN, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0 close(1) = 0 write(3, "\236\244\355{\356$\237\377\223\237\34\366\275\244\350\t\253\206<\266\3634\371\376\214\377\f\212\211|\310\263"..., 164033) = 164033 close(3) = 0 lstat64("home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", {st_mode=S_IFREG|0600, st_size=32949305537, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 utimensat(AT_FDCWD, "home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", {UTIME_NOW, {0, 854158541}}, AT_SYMLINK_NOFOLLOW) = 0 rename("home/kip/.VirtualBox/Machines/Woe64 8.1/.Woe64 8.1.vdi.1J1zb5", "home/kip/.VirtualBox/Machines/Woe64 8.1/Woe64 8.1.vdi") = 0 gettimeofday({1476587334, 151625}, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) gettimeofday({1476587334, 255616}, NULL) = 0 exit_group(19) = ? Process 30619 detached -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From bs27975 at gmail.com Sun Oct 16 20:17:14 2016 From: bs27975 at gmail.com (B. S.) Date: Sun, 16 Oct 2016 16:17:14 -0400 Subject: Links repeatedly being copied??? In-Reply-To: References: Message-ID: <5803E04A.40201@gmail.com> (Kubuntu 12.04.5 LTS rsync version 3.0.9 protocol version 30) I rsync the data ('data master') partition of my main machine to the global ('local global repository') partition, and this global partition to two other remote ('remote global repository') partitions, nightly. local -> global, global -> 2 x remotes. This is done via cifs - i.e. no daemons involved. (mount //remote/dest /mnt/remote/dest, rsync /src /mnt/remote/dest) I have for the longest time been seeing '.L..t......'s in my rsync log, despite no changes to such in quite some time, and in trying to track it down, been baffled. Only yesterday did I notice (man rsync): (1) -l, --links copy symlinks as symlinks (2) -l, --links When symlinks are encountered, recreate the symlink on the destination. The aha moment was tracking the word 'recreate'! Sanity check please: rsync DOES NOT COPY links just as it would any other file, it recreates them (at each creation / change detection)? If so, could 'man rsync', (1) above, please s/copy/recreate/? I might have picked up on things rather earlier than I did. [i.e. rsync mirror ... doesn't. At least for how I think of 'mirrors' - lose the local disk, pop the remote disk into the local machine, get on with your day.] (Queue head change of what rsync is. Not so much complaining, I get rsync is what it is and it's a good thing, just need to change how my head is wrapped around it.) So, sanity check please: - I create a symlink locally. - next time rsync is run, it duplicates the link on the remote, DATED AT THAT TIME OF (remote) LINK CREATION. - subsequent rsync's detects a time difference between local vs remote symlinks, so again duplicates the link on the remote, DATED AT THAT TIME OF (remote) LINK CREATION. (This re-copying, er, link recreation, occurs ad infinitum.) Am I understanding this correctly? (It's what I'm seeing.) How do I stop this repeat entry in my logs? In googling, I can see, and have used, tar to work around this. However, I also keep seeing that rsync can be used instead of things like cp -ad, for just this sort of thing. (Except, I'm using rsync and seeing this sort of thing.) rsync change request, please: When it creates that remote symlink, could rsync please change the date/time to match that of the symlink from which it's 'copying'? (Just like it does for any other file.) Have I described / understood the above correctly / does it hold water? If it is not expected that links would be recreated at every run, any observations as to in what way (parameters likely used) I might be doing myself in? Thanks for listening, and thanks for your thoughts. From henri.shustak at gmail.com Sun Oct 16 22:35:24 2016 From: henri.shustak at gmail.com (Henri Shustak) Date: Mon, 17 Oct 2016 11:35:24 +1300 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476290898.27406.119.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> Message-ID: >> Have you tried performing a copy to a known good local device? If a >> local copy fails, then I would start checking the file system of the >> source and also the hardware of that system. > > That's a good idea. I just tried that and it copied no problem. Do you have another system you could try this transfer with via SSH with on the local network? Given that the local transfer works fine, I would suggest checking the hardware and file system integrity on the remote machine. In terms of hardware checking the memory and disks would be a top priority. Also, you could try moving the partial file out of the way and also not using the partial option and transferring again? Hope that helps -------------------------------------------------------------------- HTRAX 2013 Revitalised : EGYPTIAN HUMP HTRAX : Direct URL download : http://henri.shustak.org/download/htrax/egyptian-hump-htrax.mp3 "Dr Who Meets B52's" - More reviews : http://www.jessetaylor.com.au From paul+rsync at wurtel.net Tue Oct 18 06:36:11 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Tue, 18 Oct 2016 08:36:11 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> Message-ID: <20161018063611.GA2247@msgid.wurtel.net> Try the transfer without -z. Paul From hohmann at harddiskcafe.de Tue Oct 18 10:08:00 2016 From: hohmann at harddiskcafe.de (Bernd Hohmann) Date: Tue, 18 Oct 2016 12:08:00 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> Message-ID: <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de> On 18.10.2016 07:03, Kip Warner wrote: > From what I can tell, there are no hardware problems. I also ran fsck > on the drive. The machine seems to be fine. I can confirm the problem. Situation here: 2 identical HP Microservers (Debian 7, on site compiled rsync 3.1.2, connected via OpenVPN). SSH is used for transport. Both machines have the correct date/time set via ntpd. All files on Client/Server are rw and have the right owner and are copy'able. oth sides. The "directory to backup" is a Samba-share (I stopped nmbd and smbd, no change). Client: 200GB, 42000 files total. Enough disk-space and memory on both sides. All rsync instances were killed (Client/Server) before starting rsync. tcpdump shows me a NOP packet every 2 min. I can provoke the error doing this: 1) Start the transfer (rsync scans *all* client files and starts sending a file) 2) ^C rsync on client 3) "pkill rsync" on server until all rsync-processes are killed. Same on client (just to be sure) 4) Start the transfer again, now rsync scans the top directories only and hangs (see straces below). Commandline: ./rsync-debug -v --archive --progress --human-readable --delete-during \ --rsync-path=/home/backup-hugo/bin/rsync-debug \ /srv/backup-bernd backup-hugo at backup-hugo.vpn:/srv/ Client says (PID 5909 = rsync, 5910 = ssh) ------------------------------------------------ [...] 5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}) = 1 (in [4], left {239, 999990}) 5910 10:13:50 read(4, "2010_20120119093643.pdf\0\3740O [...] \242}\30:V0124160__Nr.036_vom_10.09.2010_2012011"..., 16384) = 3072 loop: 5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0} 5909 10:14:51 <... select resumed> ) = 0 (Timeout) 5909 10:14:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout) 5909 10:15:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout) 5909 10:16:51 select(6, [5], [], [5], {60, 0} 5910 10:17:51 <... select resumed> ) = 0 (Timeout) goto loop ------------------------------------------------ Server says (PID 10331 = rsync --server, 10332 = ssh) ------------------------------------------------ [...] 10331 10:13:50 lstat("backup-bernd/Schreibtisch", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 10331 10:13:50 lstat("backup-bernd/VirtualBox", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 10331 10:13:50 lstat("backup-bernd/bin", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 10331 10:13:50 lstat("backup-bernd/projekte", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 10331 10:13:50 lstat("backup-bernd/transfer", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 10331 10:13:50 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left {59, 999991}) 10331 10:13:50 write(1, "\4\0\0\7\3\20\0\0", 8) = 8 10331 10:13:50 select(4, [3], [], [3], {60, 0} 10332 10:14:50 <... select resumed> ) = 0 (Timeout) loop: 10332 10:14:50 select(1, [0], [], [0], {60, 0} 10331 10:14:50 <... select resumed> ) = 0 (Timeout) 10331 10:14:50 select(4, [3], [], [3], {60, 0} 10332 10:15:50 <... select resumed> ) = 0 (Timeout) goto loop ------------------------------------------------ -- Bernd Hohmann Organisationsprogrammierer Höhenstrasse 2 * 61130 Nidderau Telefon: 06187/900495 * Telefax: 06187/900493 Blog: http://blog.harddiskcafe.de From devzero at web.de Wed Oct 19 19:46:24 2016 From: devzero at web.de (devzero at web.de) Date: Wed, 19 Oct 2016 21:46:24 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de> Message-ID: <62005202-A93C-4772-A731-597E2D00EE25@web.de> what does lsof tell? does rsync hang on a specific file? i would wonder if this is a rsync problem. as you told you killed all processes. so, on the second run rsync knows nothing from before... roland Am 18. Oktober 2016 12:08:00 MESZ, schrieb Bernd Hohmann : >On 18.10.2016 07:03, Kip Warner wrote: > >> From what I can tell, there are no hardware problems. I also ran fsck >> on the drive. The machine seems to be fine. > >I can confirm the problem. > >Situation here: 2 identical HP Microservers (Debian 7, on site compiled >rsync 3.1.2, connected via OpenVPN). > >SSH is used for transport. > >Both machines have the correct date/time set via ntpd. > >All files on Client/Server are rw and have the right owner and are >copy'able. oth sides. > >The "directory to backup" is a Samba-share (I stopped nmbd and smbd, no >change). Client: 200GB, 42000 files total. Enough disk-space and memory >on both sides. > >All rsync instances were killed (Client/Server) before starting rsync. > >tcpdump shows me a NOP packet every 2 min. > > >I can provoke the error doing this: > >1) Start the transfer (rsync scans *all* client files and starts >sending >a file) > >2) ^C rsync on client > >3) "pkill rsync" on server until all rsync-processes are killed. Same >on >client (just to be sure) > >4) Start the transfer again, now rsync scans the top directories only >and hangs (see straces below). > > >Commandline: > >./rsync-debug -v --archive --progress > --human-readable --delete-during \ > --rsync-path=/home/backup-hugo/bin/rsync-debug \ > /srv/backup-bernd backup-hugo at backup-hugo.vpn:/srv/ > > >Client says (PID 5909 = rsync, 5910 = ssh) >------------------------------------------------ >[...] >5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0}) = 1 (in [4], left >{239, 999990}) >5910 10:13:50 read(4, "2010_20120119093643.pdf\0\3740O >[...] >\242}\30:V0124160__Nr.036_vom_10.09.2010_2012011"..., 16384) = 3072 > >loop: >5910 10:13:50 select(7, [3 4], [3], NULL, {240, 0} >5909 10:14:51 <... select resumed> ) = 0 (Timeout) >5909 10:14:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout) >5909 10:15:51 select(6, [5], [], [5], {60, 0}) = 0 (Timeout) >5909 10:16:51 select(6, [5], [], [5], {60, 0} >5910 10:17:51 <... select resumed> ) = 0 (Timeout) >goto loop >------------------------------------------------ > >Server says (PID 10331 = rsync --server, 10332 = ssh) >------------------------------------------------ >[...] >10331 10:13:50 lstat("backup-bernd/Schreibtisch", >{st_mode=S_IFDIR|0755, >st_size=4096, ...}) = 0 >10331 10:13:50 lstat("backup-bernd/VirtualBox", {st_mode=S_IFDIR|0755, >st_size=4096, ...}) = 0 >10331 10:13:50 lstat("backup-bernd/bin", {st_mode=S_IFDIR|0755, >st_size=4096, ...}) = 0 >10331 10:13:50 lstat("backup-bernd/projekte", {st_mode=S_IFDIR|0755, >st_size=4096, ...}) = 0 >10331 10:13:50 lstat("backup-bernd/transfer", {st_mode=S_IFDIR|0755, >st_size=4096, ...}) = 0 >10331 10:13:50 select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left >{59, >999991}) >10331 10:13:50 write(1, "\4\0\0\7\3\20\0\0", 8) = 8 >10331 10:13:50 select(4, [3], [], [3], {60, 0} >10332 10:14:50 <... select resumed> ) = 0 (Timeout) > >loop: >10332 10:14:50 select(1, [0], [], [0], {60, 0} >10331 10:14:50 <... select resumed> ) = 0 (Timeout) >10331 10:14:50 select(4, [3], [], [3], {60, 0} >10332 10:15:50 <... select resumed> ) = 0 (Timeout) >goto loop >------------------------------------------------ > >-- >Bernd Hohmann >Organisationsprogrammierer >Höhenstrasse 2 * 61130 Nidderau >Telefon: 06187/900495 * Telefax: 06187/900493 >Blog: http://blog.harddiskcafe.de > > > >-- >Please use reply-all for most replies to avoid omitting the mailing >list. >To unsubscribe or change options: >https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: >http://www.catb.org/~esr/faqs/smart-questions.html -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hohmann at harddiskcafe.de Wed Oct 19 21:12:55 2016 From: hohmann at harddiskcafe.de (Bernd Hohmann) Date: Wed, 19 Oct 2016 23:12:55 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <62005202-A93C-4772-A731-597E2D00EE25@web.de> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> <86ab68b0-04e7-71ad-c475-0d6498e3b077@harddiskcafe.de> <62005202-A93C-4772-A731-597E2D00EE25@web.de> Message-ID: <35442d5e-ab3e-d608-29be-a1839f5556bf@harddiskcafe.de> On 19.10.2016 21:46, devzero at web.de wrote: > what does lsof tell? does rsync hang on a specific file? Nothing. I ruled this out already. > i would wonder if this is a rsync problem. as you told you killed all > processes. > so, on the second run rsync knows nothing from before... Maybe something stupid in the SSH(D) stack. I checked all running ssh instances for orphans - nothing. Current status quo of testing: 1) After restarting the client, rsync starts working synchronizing changed/missing files on the server via vpn and runs into a timeout after a couple of hours. 2) Restarting the backup (after pkill'ing rsync on both sides, just to be sure) runs into a timeout and no attempt was made to rsync files. 3) Hint: If I remove a directory (13 Files, 50GB total), everything is fine. Are there some magic Unix file attributes on ext4? Bernd -- Bernd Hohmann Organisationsprogrammierer Höhenstrasse 2 * 61130 Nidderau Telefon: 06187/900495 * Telefax: 06187/900493 Blog: http://blog.harddiskcafe.de From kip at thevertigo.com Thu Oct 20 01:24:28 2016 From: kip at thevertigo.com (Kip Warner) Date: Wed, 19 Oct 2016 18:24:28 -0700 Subject: rsync: connection unexpectedly closed In-Reply-To: <20161018063611.GA2247@msgid.wurtel.net> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> <20161018063611.GA2247@msgid.wurtel.net> Message-ID: <1476926668.1807.13.camel@thevertigo.com> On Tue, 2016-10-18 at 08:36 +0200, Paul Slootman wrote: > Try the transfer without -z. > > Paul I ended up giving up. What I did was I just removed the 30GB file (which I really didn't need anyways) and the transfer carried on without a hitch. -- Kip Warner -- Senior Software Engineer OpenPGP encrypted/signed mail preferred http://www.thevertigo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: This is a digitally signed message part URL: From space.ship.traveller at gmail.com Thu Oct 20 09:24:23 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Thu, 20 Oct 2016 22:24:23 +1300 Subject: -e escape rule Message-ID: Hello, I'm using Ruby's Shellwords module, which generates a string from an array, suitable for shell evaluation. Ruby's implementation prefers escaping whitespace with a backslash rather than quotes. However, this appears to cause some kind of issue in Rsync when it computes argv from -e option. Here is an example command generated by some Ruby code: rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ We can check that something like this is valid: files% echo foo\ bar\\\=baz foo bar\=baz -- what Rsync should be receiving files% echo foo bar\=baz foo bar=baz -- What Rsync should be executing However this gives me an error command-line: line 0: Bad configuration option: connecttimeout\\ rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] I think the problem here is the "ConnectTimeout\\\=60", in particular how the equals symbol is escaped. I'm looking in the function: static pid_t do_cmd(char *cmd, char *machine, char *user, char **remote_argv, int remote_argc, int *f_in_p, int *f_out_p) This function splits based purely on whitespace: args[argc++] = t; while (*f != ' ' || in_quote) { // consume token... I feel that this function should also handle backslash escapes. I also checked using strace and it appears that this is the issue, but I'm open to suggestions/ideas. Kind regards, Samuel From paul+rsync at wurtel.net Thu Oct 20 11:03:32 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Thu, 20 Oct 2016 13:03:32 +0200 Subject: -e escape rule In-Reply-To: References: Message-ID: <20161020110332.GA30679@msgid.wurtel.net> On Thu 20 Oct 2016, Samuel Williams wrote: > > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option. > > Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ There's no reason to escape an "=" sign in the above command. Try: rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ Or even: rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ I'd probably create an entry for this host in ~/.ssh/config : Host example.backup.server.com User backup IdentityFile /etc/synco/id_rsa ConnectTimeout 60 BatchMode=yes and then just use: rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ If you're dynamically doing this, you can pass a config file with -F. Paul From space.ship.traveller at gmail.com Thu Oct 20 12:22:32 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Fri, 21 Oct 2016 01:22:32 +1300 Subject: -e escape rule In-Reply-To: <20161020110332.GA30679@msgid.wurtel.net> References: <20161020110332.GA30679@msgid.wurtel.net> Message-ID: > There's no reason to escape an "=" sign in the above command. Okay, so at a certain level I agree with you and that's how I've fixed the issue. All the stuff about config files is not feasible for a variety of reasons - I'm aware of those options. I appreciate that you took the time to explain them. I think the problem is the assumptions in do_cmd about the contents of the -e argument. In my opinion, the way this is split needs to be deferred back to the user's shell to work correctly. I'm not sure if there is an easy way to do this that works for any kind of shell. Assuming that -e is stored into rsh_cmd, and is user supplied for executing on the shell, the only valid thing you can really do with it is: execl("/bin/sh", "sh", "-c", rsh_cmd, (char *) NULL); (assuming that sh is the user's current shell, might want to use $SHELL or something else). It might make more sense if rsh_cmd used string substitution rather than argument splitting, e.g. -e 'ssh $HOST' this way, no splitting would be required. In lots of ways this is more flexible. For example, appending the host might not be useful in some contexts. Anyway, rsync is pretty awesome. the -e option is only a minor annoyance, but it would be nice if this discussion could result in a tangible improvement. On 21 October 2016 at 00:03, Paul Slootman wrote: > On Thu 20 Oct 2016, Samuel Williams wrote: >> >> I'm using Ruby's Shellwords module, which generates a string from an >> array, suitable for shell evaluation. >> >> Ruby's implementation prefers escaping whitespace with a backslash >> rather than quotes. However, this appears to cause some kind of issue >> in Rsync when it computes argv from -e option. >> >> Here is an example command generated by some Ruby code: >> >> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ >> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest >> ../../latest/etc/ /etc/ >> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > There's no reason to escape an "=" sign in the above command. Try: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > Or even: > > rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > I'd probably create an entry for this host in ~/.ssh/config : > > Host example.backup.server.com > User backup > IdentityFile /etc/synco/id_rsa > ConnectTimeout 60 > BatchMode=yes > > and then just use: > > rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > > If you're dynamically doing this, you can pass a config file with -F. > > > Paul > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html From dhoworth at mrc-lmb.cam.ac.uk Thu Oct 20 12:46:50 2016 From: dhoworth at mrc-lmb.cam.ac.uk (Dave Howorth) Date: Thu, 20 Oct 2016 13:46:50 +0100 Subject: -e escape rule In-Reply-To: References: Message-ID: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> On 2016-10-20 10:24, Samuel Williams wrote: > Hello, > > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option. The man page for rsync explicitly forbids the use of backslashes: "Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes)." So I think the rest of the question is moot ... Cheers, Dave > Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > We can check that something like this is valid: > > files% echo foo\ bar\\\=baz > foo bar\=baz -- what Rsync should be receiving > files% echo foo bar\=baz > foo bar=baz -- What Rsync should be executing > > However this gives me an error > > command-line: line 0: Bad configuration option: connecttimeout\\ > rsync: connection unexpectedly closed (0 bytes received so far) [sender] > rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] > > I think the problem here is the "ConnectTimeout\\\=60", in particular > how the equals symbol is escaped. > > I'm looking in the function: > > static pid_t do_cmd(char *cmd, char *machine, char *user, char > **remote_argv, int remote_argc, > int *f_in_p, int *f_out_p) > > This function splits based purely on whitespace: > > args[argc++] = t; > while (*f != ' ' || in_quote) { > // consume token... > > I feel that this function should also handle backslash escapes. > > I also checked using strace and it appears that this is the issue, but > I'm open to suggestions/ideas. > > Kind regards, > Samuel > From kmk at sanitarium.net Thu Oct 20 17:03:52 2016 From: kmk at sanitarium.net (Kevin Korb) Date: Thu, 20 Oct 2016 13:03:52 -0400 Subject: -e escape rule In-Reply-To: References: Message-ID: The \ escapes are for the shell. Rsync never sees them and therefore can't honor them. On 10/20/2016 05:24 AM, Samuel Williams wrote: > Hello, > > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option. > > Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > We can check that something like this is valid: > > files% echo foo\ bar\\\=baz > foo bar\=baz -- what Rsync should be receiving > files% echo foo bar\=baz > foo bar=baz -- What Rsync should be executing > > However this gives me an error > > command-line: line 0: Bad configuration option: connecttimeout\\ > rsync: connection unexpectedly closed (0 bytes received so far) [sender] > rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] > > I think the problem here is the "ConnectTimeout\\\=60", in particular > how the equals symbol is escaped. > > I'm looking in the function: > > static pid_t do_cmd(char *cmd, char *machine, char *user, char > **remote_argv, int remote_argc, > int *f_in_p, int *f_out_p) > > This function splits based purely on whitespace: > > args[argc++] = t; > while (*f != ' ' || in_quote) { > // consume token... > > I feel that this function should also handle backslash escapes. > > I also checked using strace and it appears that this is the issue, but > I'm open to suggestions/ideas. > > Kind regards, > Samuel > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From hohmann at harddiskcafe.de Thu Oct 20 20:45:33 2016 From: hohmann at harddiskcafe.de (Bernd Hohmann) Date: Thu, 20 Oct 2016 22:45:33 +0200 Subject: rsync: connection unexpectedly closed In-Reply-To: <1476926668.1807.13.camel@thevertigo.com> References: <20161010162417.hd3hag6nzj7zegl6@kip-desktop> <1476290898.27406.119.camel@thevertigo.com> <20161018050302.wfihlh5rxhj3f7qd@kip-desktop> <20161018063611.GA2247@msgid.wurtel.net> <1476926668.1807.13.camel@thevertigo.com> Message-ID: On 20.10.2016 03:24, Kip Warner wrote: > I ended up giving up. Me too. I'm copying all files via 'scp' now - takes 3 days but no aborts or errors. So I am very sure the problem is somewhere in rsync. Bernd -- Bernd Hohmann Organisationsprogrammierer Höhenstrasse 2 * 61130 Nidderau Telefon: 06187/900495 * Telefax: 06187/900493 Blog: http://blog.harddiskcafe.de From space.ship.traveller at gmail.com Fri Oct 21 00:20:41 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Fri, 21 Oct 2016 13:20:41 +1300 Subject: -e escape rule In-Reply-To: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: Hi Dave, thanks for point that out. I didn't realise there was a detailed explanation of that field in the man page, I only saw the summary. Yes, that clearly explains how it's supposed to work. On 21 October 2016 at 01:46, Dave Howorth wrote: > On 2016-10-20 10:24, Samuel Williams wrote: >> >> Hello, >> >> I'm using Ruby's Shellwords module, which generates a string from an >> array, suitable for shell evaluation. >> >> Ruby's implementation prefers escaping whitespace with a backslash >> rather than quotes. However, this appears to cause some kind of issue >> in Rsync when it computes argv from -e option. > > > The man page for rsync > explicitly forbids the use of backslashes: > > "Command-line arguments are permitted in COMMAND provided that COMMAND is > presented to rsync as a single argument. You must use spaces (not tabs or > other whitespace) to separate the command and args from each other, and you > can use single- and/or double-quotes to preserve spaces in an argument (but > not backslashes)." > > So I think the rest of the question is moot ... > > Cheers, Dave > > >> Here is an example command generated by some Ruby code: >> >> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ >> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest >> ../../latest/etc/ /etc/ >> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ >> >> We can check that something like this is valid: >> >> files% echo foo\ bar\\\=baz >> foo bar\=baz -- what Rsync should be receiving >> files% echo foo bar\=baz >> foo bar=baz -- What Rsync should be executing >> >> However this gives me an error >> >> command-line: line 0: Bad configuration option: connecttimeout\\ >> rsync: connection unexpectedly closed (0 bytes received so far) [sender] >> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] >> >> I think the problem here is the "ConnectTimeout\\\=60", in particular >> how the equals symbol is escaped. >> >> I'm looking in the function: >> >> static pid_t do_cmd(char *cmd, char *machine, char *user, char >> **remote_argv, int remote_argc, >> int *f_in_p, int *f_out_p) >> >> This function splits based purely on whitespace: >> >> args[argc++] = t; >> while (*f != ' ' || in_quote) { >> // consume token... >> >> I feel that this function should also handle backslash escapes. >> >> I also checked using strace and it appears that this is the issue, but >> I'm open to suggestions/ideas. >> >> Kind regards, >> Samuel >> > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html From space.ship.traveller at gmail.com Fri Oct 21 01:35:05 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Fri, 21 Oct 2016 14:35:05 +1300 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: Hmm, so after reviewing this in a bit more detail.. it would be very nice if at least backslash escapes would be supported. Quoted strings are pretty ugly, especially when nested several levels. Anyway, just my 2 cents. From samba-bugs at samba.org Fri Oct 21 06:09:05 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 21 Oct 2016 06:09:05 +0000 Subject: [Bug 12386] New: Copy Loop Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12386 Bug ID: 12386 Summary: Copy Loop Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter: crysanthus at gmail.com QA Contact: rsync-qa at samba.org If you use .* as a rsync source it will be looped until end of disk ie. rsync -rvhIW crp/.* rohanamobile/public/ -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Fri Oct 21 06:12:10 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 21 Oct 2016 06:12:10 +0000 Subject: [Bug 12386] Copy Loop In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12386 --- Comment #1 from Kevin Korb --- That isn't a bug you are using it wrong. drop the .* and it will do what you expect. There should almost never be a * in the source parameter. Also, don't rsync without --times unless you have a really good reason. -- You are receiving this mail because: You are the QA Contact for the bug. From wayned at samba.org Fri Oct 21 21:22:48 2016 From: wayned at samba.org (Wayne Davison) Date: Fri, 21 Oct 2016 14:22:48 -0700 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: On Thu, Oct 20, 2016 at 6:35 PM, Samuel Williams < space.ship.traveller at gmail.com> wrote: > it would be very nice if at least backslash escapes would be supported > They are supported just fine. In your first message you passed several option\\\=args to ssh, and it was *ssh* that gave you an error (it didn't like that it received a backslash as part of the option name). ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From samba-bugs at samba.org Fri Oct 21 21:36:34 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Fri, 21 Oct 2016 21:36:34 +0000 Subject: [Bug 12386] Copy Loop In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12386 Wayne Davison changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #2 from Wayne Davison --- You'd probably be better served by asking about something like this on the mailing list, as there is no issue with .* concerning rsync -- this is entirely a shell issue. In this instance you're probably using bash, which is so stupid that it expands ".*" to include "..", which is obviously not what you want. I personally use zsh, which doesn't have this issue. When using bash, you could instead try a "path/.??*" (if all your dot files are at least 2 chars after the dot) or something like ".[a-z]*" (though you might also need A-Z and 0-9 etc added). -- You are receiving this mail because: You are the QA Contact for the bug. From devzero at web.de Tue Oct 25 09:20:23 2016 From: devzero at web.de (devzero at web.de) Date: Tue, 25 Oct 2016 11:20:23 +0200 Subject: error code 255 - not mentioned in manpage? Message-ID: hi, is there a reason why error code 255 is not mentioned in the manpage and wouldn`t it make sense to add "255 Unexplained Error" there for completeness ? I`m writing a script which checks exit values and while testing it, i got value 255, looked into the manpage and scratched my head what`s the meaning of it. ok - it`s obvious if you take a look at rsync error output, but i think all exit values should be documented. regards roland # cat rsync.err ssh: connect to host ...... port 22: Connection refused rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9] From paul+rsync at wurtel.net Tue Oct 25 09:37:40 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Tue, 25 Oct 2016 11:37:40 +0200 Subject: error code 255 - not mentioned in manpage? In-Reply-To: References: Message-ID: <20161025093740.GA26536@msgid.wurtel.net> On Tue 25 Oct 2016, devzero at web.de wrote: > > is there a reason why error code 255 is not mentioned in the manpage > and wouldn`t it make sense to add "255 Unexplained Error" there > for completeness ? It wouldn't be unexplained then anymore, would it? :-) Paul From devzero at web.de Wed Oct 26 09:34:09 2016 From: devzero at web.de (devzero at web.de) Date: Wed, 26 Oct 2016 11:34:09 +0200 Subject: O_NOATIME ? Message-ID: Hello, since we are using rsync for backing up millions of files in a virtual environment, and most of the virtual machines run on SSD cached storage, i`d be curious how that negatively impacts lifetime of the SSD`s when we do rsync run every night for backup my question: does rsync normal file comparison run to determine if anything has changed change atime of any files ? for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not sure under which conditions atime is changed. grepping the source for O_NOATIME in rsync3.txt i found : - Propagate atimes and do not modify them. This is very ugly on Unix. It might be better to try to add O_NOATIME to kernels, and call that. furhermore, apparently there _IS_ O_NOATIME in linux kernels for a while: http://man7.org/linux/man-pages/man2/open.2.html O_NOATIME (since Linux 2.6.8) Do not update the file last access time (st_atime in the inode) when the file is read(2). This flag can be employed only if one of the following conditions is true: * The effective UID of the process matches the owner UID of the file. * The calling process has the CAP_FOWNER capability in its user namespace and the owner UID of the file has a mapping in the namespace. This flag is intended for use by indexing or backup programs, where its use can significantly reduce the amount of disk activity. This flag may not be effective on all filesystems. One example is NFS, where the server maintains the access time. so, maybe someone likes to comment on NOATIME !? maybe it could be useful to make rsync honour O_NOATIME ? regards roland From paul+rsync at wurtel.net Wed Oct 26 09:46:50 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Wed, 26 Oct 2016 11:46:50 +0200 Subject: O_NOATIME ? In-Reply-To: References: Message-ID: <20161026094650.GA27031@msgid.wurtel.net> On Wed 26 Oct 2016, devzero at web.de wrote: > > since we are using rsync for backing up millions of files in a virtual environment, and most of the virtual machines run on SSD cached storage, i`d be curious how that negatively impacts lifetime of the SSD`s when we do rsync run every night for backup > > my question: > does rsync normal file comparison run to determine if anything has changed change atime of any files ? > > for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not sure under which conditions atime is changed. Most filesystems on modern linux systems should be mounted with the relatime option. The atime will then only be updated if either the mtime or ctime is newer than the atime, or if the atime is older than a defined interval. Note that simply using stat will not update the atime, as the *file* itself has not been accessed, only its metadata. There's also the nodiratime option that can be useful, as directories are read to find what files exist in those directories; and it's seldomly useful to maintain the atime of a directory. You can also use noatime as a mount option, but then be sure that no application uses the atimes of files; e.g. something like mutt use the atime and mtime to determine whether there is a mail file with unread mail. Paul From tytso at mit.edu Thu Oct 27 04:29:06 2016 From: tytso at mit.edu (Theodore Ts'o) Date: Thu, 27 Oct 2016 00:29:06 -0400 Subject: O_NOATIME ? In-Reply-To: <20161026094650.GA27031@msgid.wurtel.net> References: <20161026094650.GA27031@msgid.wurtel.net> Message-ID: <20161027042906.pdeqhegscxxveaep@thunk.org> On Wed, Oct 26, 2016 at 11:46:50AM +0200, Paul Slootman wrote: > Most filesystems on modern linux systems should be mounted with the > relatime option. This is the default... > You can also use noatime as a mount option, but then be sure that no > application uses the atimes of files; e.g. something like mutt use the > atime and mtime to determine whether there is a mail file with unread > mail. With the latest Linux kernels (4.0+), there is also the "lazytime" mount option. This causes the kernel to avoid writing back inodes that only have "dirty timestamps", until either (a) some other change is made to the inode, (b) fsync(), syncfs(), or sync() is called, (c) an undeleted inode is evicted from memory or the file system is unounted, (d) 24 hours have gone by, or (e) in the case of ext4, if some other change is made to an inode in the same inode table block as an inode with a dirty timestamp (so we need to do the disk I/O anyway). With lazytime, the timestamp is updated in memory, so stat(2) will always return the correct timestamp, and in normal practice, the timestamps will be (eventually) updated on disk. However if a timestsamp gets updated multiple times --- for example, a database file getting updated with O_DIRECT writes, instead of the mtime field being written out every 30 seconds, we can defer the timestamp updates and collapse them into a much smaller set of inode table block writes. Cheers, - Ted From space.ship.traveller at gmail.com Fri Oct 28 12:39:00 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Sat, 29 Oct 2016 01:39:00 +1300 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: Wayne, I guess if you apply standard shell escaping logic you end up with a string which isn't processed correctly, yes, ssh complains, but only because Rsync passed the backslashes through without dealing with them. My opinion is that if Rsync is splitting a string based on whitespace it also needs to take into account backslashes. From rsync at jelmail.com Fri Oct 28 14:32:03 2016 From: rsync at jelmail.com (John Lane) Date: Fri, 28 Oct 2016 15:32:03 +0100 Subject: rsync send "non-rsync" options to the server side (--remote-option) and (--protect-args) In-Reply-To: References: Message-ID: <1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com> Hello, I asked the below back at the beginning of August but have received no replies. If anyone can help me with this problem it would be much appreciated. If I have not provided some necessary information then please let me know so I may do so. Thanks in advance. > I have been working on a backup server where I have a server-side script > that wraps the server-side rsync invocation. I have used the client-side > rsync -M (--remote-option) to send options to the server script, > removing them from the command-line prior to invoking the server-side > rsync. This was discussed on Stack Exchange > (http://unix.stackexchange.com/questions/297143). It works well. > > However, one observation is that the rsync client's -s (--protect-args) > option passes all arguments directly into the rsync server and only > sends a minimal rsync command-line to launch the server-side rsync. > > This means my script does not see -M options when -s is used. It also > means that the server-side rsync receives options not intended for it > and these cause it to terminate. > > I note from the man-page that the -s option "will eventually become a > new default setting at some as-yet-undetermined point in the future". > When that happens, it will break what I have above. > > So, a couple of questions: > > * is using -M the right way to send non-rsync options to the server side ? > > * could it be considered for -s to not affect non-rsync options that > have been specified with -M ? > > Thanks for your attention. > From wayned at samba.org Sat Oct 29 02:10:40 2016 From: wayned at samba.org (Wayne Davison) Date: Fri, 28 Oct 2016 19:10:40 -0700 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams < space.ship.traveller at gmail.com> wrote: > Rsync passed the backslashes through without dealing with them. > Yeah, it only does space-splitting and that's all it will ever do. It still looks to me like there is a bug in the original escaping, since any command receiving that string is receiving a backslash that is not supposed to be there. It should only be escaping the string enough to get it to rsync, not trying to guess what rsync is going to do with it after it gets it. To work around that bug, you could consider using an ssh-calling shell script instead of manually specifying the ssh command and args to rsync. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wayned at samba.org Sat Oct 29 02:14:43 2016 From: wayned at samba.org (Wayne Davison) Date: Fri, 28 Oct 2016 19:14:43 -0700 Subject: rsync send "non-rsync" options to the server side (--remote-option) and (--protect-args) In-Reply-To: <1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com> References: <1d0c16eb-15e8-4dda-659c-4b01d3b1a3a9@jelmail.com> Message-ID: If you want to pass non-rsync args (etc) you should be using --rsync-path. The -M option is only for sending rsync-related options. ..wayne.. On Fri, Oct 28, 2016 at 7:32 AM, John Lane wrote: > > Hello, I asked the below back at the beginning of August but have > received no replies. If anyone can help me with this problem it would be > much appreciated. If I have not provided some necessary information then > please let me know so I may do so. Thanks in advance. > > > I have been working on a backup server where I have a server-side script > > that wraps the server-side rsync invocation. I have used the client-side > > rsync -M (--remote-option) to send options to the server script, > > removing them from the command-line prior to invoking the server-side > > rsync. This was discussed on Stack Exchange > > (http://unix.stackexchange.com/questions/297143). It works well. > > > > However, one observation is that the rsync client's -s (--protect-args) > > option passes all arguments directly into the rsync server and only > > sends a minimal rsync command-line to launch the server-side rsync. > > > > This means my script does not see -M options when -s is used. It also > > means that the server-side rsync receives options not intended for it > > and these cause it to terminate. > > > > I note from the man-page that the -s option "will eventually become a > > new default setting at some as-yet-undetermined point in the future". > > When that happens, it will break what I have above. > > > > So, a couple of questions: > > > > * is using -M the right way to send non-rsync options to the server side > ? > > > > * could it be considered for -s to not affect non-rsync options that > > have been specified with -M ? > > > > Thanks for your attention. > > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/ > mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmk at sanitarium.net Sat Oct 29 02:41:56 2016 From: kmk at sanitarium.net (Kevin Korb) Date: Fri, 28 Oct 2016 22:41:56 -0400 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> Message-ID: <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> A \ is a shell escape. Every level of shell consumes one level of escape. For every session of a shell every example "\ " becomes " " and every example of "\\" becomes "\". The raync command line is a mix of local shell commands + commands run via a remote shell via ssh. The simple fact is that if only \ escapes are used, the local shell will utilize those escapes and rsync will never see them. For every level of shell an escape \ does its job. If you want rsync to interpret a space char as other than whitespace then a \ is one way to do so. But one we are talking about the other side of an rsync we are talking about a local command line that consumes a \. Then we are talking about a remote command line that consumes another \. So, within the -e parameter, multiple stacked \ characters are required. Simply put, it is absolutely absurd to \ escape an rsync command line while interpreting the subshell -e parameter and the remote shell parameters the same way. At best you are insisting that ssh do something entirely nonsensical while ensuring that the first s in "ssh" is interpreted as an "s". On 10/28/2016 10:10 PM, Wayne Davison wrote: > > On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams > > > wrote: > > Rsync passed the backslashes through without dealing with them. > > > Yeah, it only does space-splitting and that's all it will ever do. It > still looks to me like there is a bug in the original escaping, since > any command receiving that string is receiving a backslash that is not > supposed to be there. It should only be escaping the string enough to > get it to rsync, not trying to guess what rsync is going to do with it > after it gets it. > > To work around that bug, you could consider using an ssh-calling shell > script instead of manually specifying the ssh command and args to rsync. > > ..wayne.. > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From space.ship.traveller at gmail.com Sat Oct 29 03:25:11 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Sat, 29 Oct 2016 16:25:11 +1300 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> Message-ID: Kevin, I agree with what you are saying on some level, but I don't think the code does what you think it does. rsync -e "foo\\ bar" will be executed by the shell and yield a cmd string in the do_cmd function of "foo\ bar". This will be split incorrectly into an argv of ["foo\", "bar"]. I'm not sure in what reality this makes any sense. All I'm proposing is that the do_cmd split function interprets the -e as a string that would normally be executed on the command line, and if it needs to be split, follow the normal rules of the command line for handling whitespace. On 29 October 2016 at 16:24, Samuel Williams wrote: > Kevin, I agree with what you are saying on some level, but I don't > think the code does what you think it does. rsync -e "foo\\ bar" will > be executed by the shell and yield a cmd string in the do_cmd function > of "foo\ bar". This will be split incorrectly into an argv of ["foo\", > "bar"]. I'm not sure in what reality this makes any sense. All I'm > proposing is that the do_cmd split function interprets the -e as a > string that would normally be executed on the command line, and if it > needs to be split, follow the normal rules of the command line for > handling whitespace. > > On 29 October 2016 at 15:41, Kevin Korb wrote: >> A \ is a shell escape. Every level of shell consumes one level of >> escape. For every session of a shell every example "\ " becomes " " and >> every example of "\\" becomes "\". >> >> The raync command line is a mix of local shell commands + commands run >> via a remote shell via ssh. >> >> The simple fact is that if only \ escapes are used, the local shell will >> utilize those escapes and rsync will never see them. >> >> For every level of shell an escape \ does its job. >> >> If you want rsync to interpret a space char as other than whitespace >> then a \ is one way to do so. >> >> But one we are talking about the other side of an rsync we are talking >> about a local command line that consumes a \. Then we are talking about >> a remote command line that consumes another \. So, within the -e >> parameter, multiple stacked \ characters are required. >> >> Simply put, it is absolutely absurd to \ escape an rsync command line >> while interpreting the subshell -e parameter and the remote shell >> parameters the same way. >> >> At best you are insisting that ssh do something entirely nonsensical >> while ensuring that the first s in "ssh" is interpreted as an "s". >> >> >> On 10/28/2016 10:10 PM, Wayne Davison wrote: >>> >>> On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams >>> > >>> wrote: >>> >>> Rsync passed the backslashes through without dealing with them. >>> >>> >>> Yeah, it only does space-splitting and that's all it will ever do. It >>> still looks to me like there is a bug in the original escaping, since >>> any command receiving that string is receiving a backslash that is not >>> supposed to be there. It should only be escaping the string enough to >>> get it to rsync, not trying to guess what rsync is going to do with it >>> after it gets it. >>> >>> To work around that bug, you could consider using an ssh-calling shell >>> script instead of manually specifying the ssh command and args to rsync. >>> >>> ..wayne.. >>> >>> >> >> -- >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >> Kevin Korb Phone: (407) 252-6853 >> Systems Administrator Internet: >> FutureQuest, Inc. Kevin at FutureQuest.net (work) >> Orlando, Florida kmk at sanitarium.net (personal) >> Web page: http://www.sanitarium.net/ >> PGP public key available on web site. >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >> >> >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html From space.ship.traveller at gmail.com Sat Oct 29 04:21:47 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Sat, 29 Oct 2016 17:21:47 +1300 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> Message-ID: > Yeah, it only does space-splitting and that's all it will ever do. It still looks to me like there is a bug in the original escaping, since any command receiving that string is receiving a backslash that is not supposed to be there. It should only be escaping the string enough to get it to rsync, not trying to guess what rsync is going to do with it after it gets it. I'm not proposing some additional characters to split on, but quite the opposite, to handle the backslash escaped spaces correctly and NOT split. Rest assured, there is no bug with the original escaping. For your edification: $ echo \I\'\m\ \a\ \s\t\r\i\n\g I'm a string From paul+rsync at wurtel.net Sat Oct 29 12:13:42 2016 From: paul+rsync at wurtel.net (Paul Slootman) Date: Sat, 29 Oct 2016 14:13:42 +0200 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> Message-ID: <20161029121342.GB7038@msgid.wurtel.net> On Sat 29 Oct 2016, Samuel Williams wrote: > I'm not proposing some additional characters to split on, but quite > the opposite, to handle the backslash escaped spaces correctly and NOT > split. Rest assured, there is no bug with the original escaping. For > your edification: > > $ echo \I\'\m\ \a\ \s\t\r\i\n\g > I'm a string The point is that the original escaping DOUBLE escapes an equals sign: foo\\\=bar It shouldn't, there's no reason to. Paul From space.ship.traveller at gmail.com Sat Oct 29 12:36:17 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Sun, 30 Oct 2016 01:36:17 +1300 Subject: -e escape rule In-Reply-To: <20161029121342.GB7038@msgid.wurtel.net> References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> <20161029121342.GB7038@msgid.wurtel.net> Message-ID: > The point is that the original escaping DOUBLE escapes an equals sign: > foo\\\=bar > It shouldn't, there's no reason to. If you paste into your command line: rsync -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes The list of arguments would be (i.e. the values in ARGV): ['rsync', '-e', 'ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes'] The command ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes Is a correct and valid shell command. When RSync parses this in do_cmd, it should convert the '\=` sequence into '=' but it doesn't.. This intuition is derived from the fact that if you instead passed the string to `system('ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes')` that the ARGV generated would be ['ssh', '-l', 'backup', '-i', '/etc/synco/id_rsa', '-o', 'ConnectTimeout=60', '-o', 'BatchMode=yes']. Basically, even if there WAS no reason to do so, doesn't mean it's invalid or undesirable. In theory, even passing in -e \\s\\s\\h should be valid. On 30 October 2016 at 01:13, Paul Slootman wrote: > On Sat 29 Oct 2016, Samuel Williams wrote: > >> I'm not proposing some additional characters to split on, but quite >> the opposite, to handle the backslash escaped spaces correctly and NOT >> split. Rest assured, there is no bug with the original escaping. For >> your edification: >> >> $ echo \I\'\m\ \a\ \s\t\r\i\n\g >> I'm a string > > The point is that the original escaping DOUBLE escapes an equals sign: > foo\\\=bar > It shouldn't, there's no reason to. > > > Paul > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html From wayned at samba.org Sat Oct 29 15:49:37 2016 From: wayned at samba.org (Wayne Davison) Date: Sat, 29 Oct 2016 08:49:37 -0700 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> <20161029121342.GB7038@msgid.wurtel.net> Message-ID: On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams < space.ship.traveller at gmail.com> wrote: > The command > > ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes > > Is a correct and valid shell command. > It is, but there is no shell involved, and assuming that a shell is being used is invalid. Adding backslash escaping now could potentially screw up anyone currently passing a backslash in their existing rsync scripts (a small group of people, but probably non-zero). It's tempting to go ahead and do this, but I think I'll just leave it as it is. If you want to work around that buggy escaping library, you could change "ssh" to "ssh-nobs" and put the following into that file: #!/usr/bin/perl exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV; That adds backslash removal on the way to the ssh. It does not allow you to backslash escape spaces, though, but someone could extend the script to support that. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wayned at samba.org Sat Oct 29 16:37:53 2016 From: wayned at samba.org (Wayne Davison) Date: Sat, 29 Oct 2016 09:37:53 -0700 Subject: Links repeatedly being copied??? In-Reply-To: <5803E04A.40201@gmail.com> References: <5803E04A.40201@gmail.com> Message-ID: On Sun, Oct 16, 2016 at 1:17 PM, B. S. wrote: > I have for the longest time been seeing '.L..t......'s in my rsync log, > despite no changes to such in quite some time, and in trying to track it > down, been baffled. > This means that rsync is trying to set the timestamp on the symlink to match the modify-time on the source symlink. The initial dot tells you that no "transfer" is happening (i.e. no change in symlink value). Some filesystems don't allow a symlink's mtime to be set, so rsync can keep trying to set the time and never be able to succeed. You may want to use this option: --omit-link-times (-J). ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From samba-bugs at samba.org Sat Oct 29 21:39:46 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 29 Oct 2016 21:39:46 +0000 Subject: [Bug 12367] temporary lines in --progress output are not cleared In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12367 Wayne Davison changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #2 from Wayne Davison --- This is apparently only happening with no -r combined with --progress. In this case there can be a series of file-count outputs (with CR) prior to the start of the transfer, but no LF before it moves on to other output. Git commit e02b89d0d fixes this. -- You are receiving this mail because: You are the QA Contact for the bug. From samba-bugs at samba.org Sat Oct 29 22:33:52 2016 From: samba-bugs at samba.org (samba-bugs at samba.org) Date: Sat, 29 Oct 2016 22:33:52 +0000 Subject: [Bug 12367] temporary lines in --progress output are not cleared In-Reply-To: References: Message-ID: https://bugzilla.samba.org/show_bug.cgi?id=12367 --- Comment #3 from Wayne Davison --- Actually, I diagnosed this further, and noticed that the file counts shouldn't be output at all -- the check for progress output was being fooled by a temporary setting of the xfer_dirs var when -R is used without -d or -r. I've changed the show_filelist_p function into a var that is set during the init call so that it is always evaluated the same way. This git commit also contains a fix for the newline that could get output with --quiet: ff66fd4bb -- You are receiving this mail because: You are the QA Contact for the bug. From space.ship.traveller at gmail.com Sun Oct 30 07:06:00 2016 From: space.ship.traveller at gmail.com (Samuel Williams) Date: Sun, 30 Oct 2016 20:06:00 +1300 Subject: -e escape rule In-Reply-To: References: <637676b4-b134-6d6c-f776-01b16400a6c3@mrc-lmb.cam.ac.uk> <6b562492-9cc8-2e7d-c6a5-62ea0121c76b@sanitarium.net> <20161029121342.GB7038@msgid.wurtel.net> Message-ID: > assuming that a shell is being used is invalid I never made this assumption. I looked directly at the source code and I stated that "I feel that this function should also handle backslash escapes." I think the assumption that splitting the command works the same way as (all?) major shells, is not inappropriate given the circumstances, and it seems like you agree. > but probably non-zero Well, I understand where you are coming from. You don't want to break existing code. The work-around is to use quotes to escape the white-space sequence. It's okay.. But it's also a surprise that backslash escape sequences don't work according to intuition of how commands are normally executed. If you supplied the string in -e to system, it would work as expected.. Unfortunately, this is the default when using Shellwords.join in Ruby. So, I had to write a custom RSync "join" function to produce an appropriate command for -e argument. On 30 October 2016 at 04:49, Wayne Davison wrote: > On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams > wrote: >> >> The command >> >> ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes >> >> Is a correct and valid shell command. > > > It is, but there is no shell involved, and assuming that a shell is being > used is invalid. Adding backslash escaping now could potentially screw up > anyone currently passing a backslash in their existing rsync scripts (a > small group of people, but probably non-zero). It's tempting to go ahead and > do this, but I think I'll just leave it as it is. > > If you want to work around that buggy escaping library, you could change > "ssh" to "ssh-nobs" and put the following into that file: > > #!/usr/bin/perl > exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV; > > That adds backslash removal on the way to the ssh. It does not allow you to > backslash escape spaces, though, but someone could extend the script to > support that. > > ..wayne.. > From devzero at web.de Mon Oct 31 23:36:22 2016 From: devzero at web.de (devzero at web.de) Date: Tue, 01 Nov 2016 00:36:22 +0100 Subject: rsync show files changed during transfer - how? Message-ID: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de> i'm using rsync for backup and, as rsync can detect if files have vanished during transfer, i wonder how rsync can tell which files got modified during transfer (i.e. which are not consistent on the destination side after transfer) apparently, rsync can't show that information? wouldn't that be an extremely useful feature if rsync could do another additional mtime or even checksum comparison after each file transfer? for example emc networker notifies about "changed during save". i'm curious wg rsync doesn't regards roland -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmk at sanitarium.net Mon Oct 31 23:56:55 2016 From: kmk at sanitarium.net (Kevin Korb) Date: Mon, 31 Oct 2016 19:56:55 -0400 Subject: rsync show files changed during transfer - how? In-Reply-To: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de> References: <9E297454-B061-4B2F-9434-D762F396C4EB@web.de> Message-ID: <130c554a-6a20-a023-f82c-6422f07ae7e9@sanitarium.net> When rsync tells you that a file vanished it is reporting a real error from the file system. It saw a file that needed to be transferred but when it attempted to read the file it got a file not found error from the FS/OS. There is no equivalent error for "hey, the file you were reading changed while you were reading it". Rsync would essentially have to read the file twice and compare. This is what LVM and similar forms of snapshots are for. On 10/31/2016 07:36 PM, devzero at web.de wrote: > i'm using rsync for backup and, as rsync can detect if files have > vanished during transfer, i wonder how rsync can tell which files got > modified during transfer (i.e. which are not consistent on the > destination side after transfer) > > apparently, rsync can't show that information? > > wouldn't that be an extremely useful feature if rsync could do another > additional mtime or even checksum comparison after each file transfer? > > for example emc networker notifies about "changed during save". i'm > curious wg rsync doesn't > > regards > roland > -- > Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail > gesendet. > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: