rsync memory usage with block-size option

[Jarrid Graham]

OK it is the text dump, uncompressed, looks like the new maint. plan does a
rebuild index then a reorganize index. Of course I do not know a lot about
SQL, but when I watch the rsync run most of the data is generated in the
last 25% of the run, I know the data is the same in there but maybe now it
is not in the same order or something. I tried the --fuzzy option that that
helped a small amount about 2% in size. I mostly wanted to try the
block-size but if there is a better option I am all for it, the bad thing
is it takes 4 hrs per run.
Thanks


On Thu, Feb 28, 2013 at 3:32 PM, Kevin Korb <kmk at sanitarium.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> When you say an SQL database dump do you mean you are rsyncing the raw
> text file or are you compressing it?  If you are compressing it you
> might want to try gzip with the --rsyncable patch as it can
> significantly improve rsync's ability to delta-xfer compressed files.
>
> Also, what do you mean by reindex?  That doesn't sound like something
> that should affect the dumps unless you are dumping something that
> would be easier to regenerate after a restore.
>
> On 02/28/13 14:37, Jarrid Graham wrote:
> > I have been using rsync to sync a SQL database dump(127GB in size)
> > on a server (cygwin to linux) for many years now w/o a problem,
> > until they put a maintence plan to reindex the database once a
> > week. This is all done over a 2Mbit link normally took 3-4 hrs now
> > I noticed it was running the next day and took days so I looked
> > into it more. After I discovered the --only-write-batch it makes a
> > 52 GB file, even hitting it with 7z it only gets down to 5G plus it
> > makes it harder to script the process. Then I got to fooling around
> > with the -B option and found on a smaller test database that when I
> > get to -B1024 it make the smallest file so I tried that. I get an
> > Out of memory error in receive_sums so I back it off and when I get
> > to about -B5120 it is good but only makes the file 36GB. I was
> > wondering is there a hard limit, the machine has tons of free ram,
> > the program used 1.1GB of RAM at -B5120,the version is 3.0.9-1 in
> > cygwin on the sender via an ssh connection. One other thing I
> > thought about but I don't see that you can is if you could the
> > --only-write-batch and read batches to/from stdout so I could 7z
> > them in line I know you can gzip with the -z option and even
> > setting the level to 9 I still get about %25 better compression
> > with 7z but everything adds another level of complexity.
> >
> > Thanks, Jarrid Graham
> >
> >
> >
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>         Kevin Korb                      Phone:    (407) 252-6853
>         Systems Administrator           Internet:
>         FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>         Orlando, Florida                kmk at sanitarium.net (personal)
>         Web page:                       http://www.sanitarium.net/
>         PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iEYEARECAAYFAlEvvtMACgkQVKC1jlbQAQeDMgCfXJnfrzijHI+X+kL9b8kbCgcj
> IVMAoMMHytQ9tXOeXNX9J7UchEw2AF+F
> =1SPZ
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130228/78dc11be/attachment.html>