<div dir="ltr">Yes, I removed "--no-inc-recursive", without success.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature">--<br>Best regards / Met vriendelijke groet,<br><br>Aron Rotteveel</div></div>
<br><div class="gmail_quote">2015-03-27 12:24 GMT+01:00 Kevin Korb <span dir="ltr"><<a href="mailto:kmk@sanitarium.net" target="_blank">kmk@sanitarium.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
</span>Have you tried removing --no-inc-recursive yet?<br>
<span class=""><br>
On 03/27/2015 07:19 AM, Aron Rotteveel wrote:<br>
> Hi Roland,<br>
><br>
> Thanks for the reply. Memory usage on both machines seem fine. The<br>
> server has 4GB's of RAM, of which about 3GB is used during the file<br>
> list build and about 1.5GB is used during the actual transfer. The<br>
> client has 16GB of RAM with a peak usage of 8.5GB.<br>
><br>
> I just tried three transfers in a row and it consistently breaks at<br>
> a certain point, after which I get the "ERROR: out of memory in<br>
> flist_expand [sender]" error. There is not much special to mention<br>
> regarding the file on which it breaks: it's a 22KB JPEG file with<br>
> no special attributes.<br>
><br>
> The backup server is running Debian 7.8, the client runs on CentOS<br>
> 5.11.<br>
><br>
> A `find . | wc -l` in the backup directory results in 7434013<br>
> files.<br>
><br>
> -- Best regards / Met vriendelijke groet,<br>
><br>
> Aron Rotteveel<br>
><br>
> 2015-03-19 20:10 GMT+01:00 <<a href="mailto:devzero@web.de">devzero@web.de</a><br>
</span>> <mailto:<a href="mailto:devzero@web.de">devzero@web.de</a>>>:<br>
<span class="">><br>
> Hi Aron,<br>
><br>
> i hope it`s ok for you if i bring this back on-list. Your issue<br>
> and the way or possible fix to resolve it may be interesting for<br>
> others too (that may include future searches etc)<br>
><br>
> so with 3.1.1 we are a step further....<br>
><br>
> i don`t really have a clue what`s happening here but my next step<br>
> would be taking a closer look on how the memory usage of rsync on<br>
> the client and server grows.<br>
><br>
> you could log it like this: while true;do ps -eo<br>
> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile<br>
><br>
> does it grow continuously? does the oom situation reproducibly<br>
> happen at a certain size ? what`s the client and server platform?<br>
> how many files? (-> <a href="https://rsync.samba.org/FAQ.html#5" target="_blank">https://rsync.samba.org/FAQ.html#5</a> ! )<br>
><br>
> regards roland<br>
><br>
</span><span class="">> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* "Aron<br>
> Rotteveel" <<a href="mailto:rotteveel.aron@gmail.com">rotteveel.aron@gmail.com</a><br>
</span>> <mailto:<a href="mailto:rotteveel.aron@gmail.com">rotteveel.aron@gmail.com</a>>> *An:* <a href="mailto:devzero@web.de">devzero@web.de</a><br>
> <mailto:<a href="mailto:devzero@web.de">devzero@web.de</a>> *Betreff:* Re: rsync 3.0.9 segmentation<br>
<span class="">> fault In addition to my last message:<br>
><br>
</span>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is used<br>
> during peak. * I tried using --no-inc-recursive, but it does not<br>
<div><div class="h5">> solve the issue.<br>
><br>
> What currrently is puzzling me is the question of why I am<br>
> receiving these errors when my server seems to have plenty of<br>
> memory to spare.<br>
><br>
> -- Best regards / Met vriendelijke groet,<br>
><br>
> Aron Rotteveel<br>
><br>
> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel<br>
> <<a href="mailto:rotteveel.aron@gmail.com">rotteveel.aron@gmail.com</a>>:<br>
><br>
> Hi Roland,<br>
><br>
> I just upgrade both the client and host to 3.1.1 and seem to memory<br>
> related issues now:<br>
><br>
> ERROR: out of memory in make_file [sender] rsync error: error<br>
> allocating core memory buffers (code 22) at util2.c(102)<br>
> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,<br>
> line=102): about to call exit(22) [Receiver] _exit_cleanup(code=22,<br>
> file=io.c, line=1633): about to call exit(22)<br>
> ----------------------------------------------------------------------------<br>
><br>
><br>
rsnapshot encountered an error! The program was invoked with<br>
> these options: /usr/bin/rsnapshot -c \<br>
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync<br>
> ----------------------------------------------------------------------------<br>
><br>
> -- Best regards / Met vriendelijke groet,<br>
><br>
> Aron Rotteveel<br>
><br>
> 2015-03-18 23:43 GMT+01:00 <<a href="mailto:devzero@web.de">devzero@web.de</a>>:<br>
><br>
> Hi,<br>
><br>
> rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs<br>
> have been fixed since then.<br>
><br>
> Is there a chance to update to the latest rsync version and retry<br>
> with that ?<br>
><br>
> regards Roland<br>
><br>
</div></div><span class="">> *Gesendet:* Dienstag, 17. März 2015 um 11:51 Uhr *Von:* "Aron<br>
> Rotteveel" <<a href="mailto:rotteveel.aron@gmail.com">rotteveel.aron@gmail.com</a>> *An:* <a href="mailto:rsync@lists.samba.org">rsync@lists.samba.org</a><br>
</span>> *Betreff:* rsync 3.0.9 segmentation fault Hi,<br>
<div><div class="h5">><br>
> I am experiencing segfaults when transferring files via rsync<br>
> though sudo. Setup:<br>
><br>
> - Backupserver initiates the rsync command with --delete -vvv<br>
> --no-inc-recursive --numeric-ids --delete-excluded --relative<br>
> --rsync-path=/home/backupuser/rsync-wrapper.sh - rsync-wrapper.sh<br>
> (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@"; - user<br>
> "backupuser" has sudo access to the rsync command - Both host and<br>
> client are running 3.0.9<br>
><br>
> The transfer starts and some files are actually transferred. Once a<br>
> certain file is reached (plain PHP file, no special characters or<br>
> any other peculiarities) it segfaults.<br>
><br>
> rsync host output:<br>
><br>
> [sender]<br>
> make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)<br>
> rsync: connection unexpectedly closed (51261222 bytes received so<br>
> far) [Receiver] rsync error: unexplained error (code 139) at<br>
> io.c(605) [Receiver=3.0.9] [Receiver] _exit_cleanup(code=12,<br>
> file=io.c, line=605): about to call exit(139)<br>
> ----------------------------------------------------------------------------<br>
><br>
><br>
rsnapshot encountered an error! The program was invoked with<br>
> these options: /usr/bin/rsnapshot -c \<br>
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync<br>
> ----------------------------------------------------------------------------<br>
><br>
><br>
ERROR: /usr/bin/rsync returned 139 while processing<br>
> backupuser@<redacted>:/backup/<br>
><br>
> Client output when using gdb to debug the coredump:<br>
><br>
> warning: no loadable sections found in added symbol-file<br>
> system-supplied DSO at 0x7fff015fd000 Core was generated by<br>
> `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids .<br>
> /backup'. Program terminated with signal 11, Segmentation fault. #0<br>
> 0x00000035cda7b441 in memcpy () from /lib64/libc.so.6<br>
><br>
> Any help would be greatly appreciated. Please let me know if<br>
> additional info is required to properly debug this issue. -- Best<br>
> regards / Met vriendelijke groet,<br>
><br>
> Aron Rotteveel -- Please use reply-all for most replies to avoid<br>
> omitting the mailing list. To unsubscribe or change options:<br>
> <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a> Before posting,<br>
> read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
><br>
><br>
><br>
><br>
<br>
</div></div><span class="">- --<br>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
Kevin Korb Phone: (407) 252-6853<br>
Systems Administrator Internet:<br>
FutureQuest, Inc. Kevin@FutureQuest.net (work)<br>
Orlando, Florida <a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a> (personal)<br>
Web page: <a href="http://www.sanitarium.net/" target="_blank">http://www.sanitarium.net/</a><br>
PGP public key available on web site.<br>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2<br>
<br>
</span>iEYEARECAAYFAlUVPe8ACgkQVKC1jlbQAQdJfQCdFpMSsACi2df9P8eJZQAi4qeB<br>
xA8Amwbpj9PGpzeQfrd4VnAytU9dN1uc<br>
=1Ayf<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br></div>