rsync memory usage with block-size option

[Jarrid Graham]

   I have been using rsync to sync a SQL database dump(127GB in size) on a
server (cygwin to linux) for many years now w/o a problem, until they put a
maintence plan to reindex the database once a week. This is all done over a
2Mbit link normally took 3-4 hrs now I noticed it was running the next day
and took days so I looked into it more. After I discovered the
--only-write-batch it makes a 52 GB file, even hitting it with 7z it only
gets down to 5G plus it makes it harder to script the process. Then I got
to fooling around with the -B option and found on a smaller test database
that when I get to -B1024 it make the smallest file so I tried that. I get
an Out of memory error in receive_sums so I back it off and when I get to
about -B5120 it is good but only makes the file 36GB. I was wondering is
there a hard limit, the machine has tons of free ram, the program used
1.1GB of RAM at -B5120,the version is 3.0.9-1 in cygwin on the sender via
an ssh connection.
   One other thing I thought about but I don't see that you can is if you
could the --only-write-batch and read batches to/from stdout so I could 7z
them in line I know you can gzip with the -z option and even setting the
level to 9 I still get about %25 better compression with 7z but everything
adds another level of complexity.

Thanks,
Jarrid Graham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130228/18fcfc5e/attachment.html>