<div dir="ltr">Bruce there is also bacula which seems to be available for all the os's you are running.<div><br></div><div><a href="http://www.bacula.org/en/?page=documentation">http://www.bacula.org/en/?page=documentation</a><br>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 10, 2014 at 11:24 PM, <span dir="ltr"><<a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Well the local test was just to a test to see if I could understand why the remote sync of the exchange database was so slow. I've heard that rsync is less efficient for local copies but this isn't like 80% the performance, or half the performance.. It's a massive difference - which I wasn't expecting to see.<br>
<br>
I have looked at unison but I also backup several linux and bsd systems here and rsync just seemed like a good choice since it's still be developed and it works on everything.<br>
<br>
In the end, I'm still looking to understand why a sync of a large exchange database file is taking 30-40 hours to finish and if there's anything I can do to help reduce that window. I can't add a cache drive to the FreeBSD server very easily at the moment so I was trying to narrow down if the issue is on the BSD side or Windows side or maybe a mix of both.<br>
<br>
I wish a native windows client of rsync existed :)<br>
<br>
------ Original Message ------<br>
From: "Kevin Korb" <<a href="mailto:kmk@sanitarium.net" target="_blank">kmk@sanitarium.net</a>><br>
To: <a href="mailto:rsync@lists.samba.org" target="_blank">rsync@lists.samba.org</a><br>
Sent: 2/10/2014 4:09:15 PM<br>
Subject: Re: Rsync performance with large exchange database files<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Rsync is known to be pretty inefficient on local copies (-W is forced<br>
there btw) and cygwin doesn't really help with that either.<br>
Essentially, when not networking rsync isn't much smarter than cp but<br>
it has a ton of extra overhead.<br>
<br>
Also, maybe you want unison since there is a native Windows version of it?<br>
<br>
On 02/10/2014 05:05 PM, <a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Clean copy. I even used the -W flag to see if it made a difference<br>
but, nope.<br>
<br>
I'm testing this same test on some of my other servers too. See if<br>
there's any common-ground I can find.<br>
<br>
On another servers (MS SQL Server) with faster disks I tried a<br>
similar test just now. There's only the C drive on this server but<br>
I used my same test file from the other server and used rsync<br>
(3.1.0) to copy the folder from one folder to another folder and it<br>
kicked off and got up to about 25MB/sec. Thing is if I just use<br>
windows to copy the same file from one folder to the other it does<br>
the whole file (3.7GB) in about 5.5 seconds (timed with my phone)<br>
so that's also a pretty massive difference.<br>
<br>
Maybe this is normal and I've just not noticed it on these other<br>
servers since they have a much smaller amount of data to backup?<br>
Still seems like some thing is wrong. I wouldn't expect the speed<br>
difference to be that huge.<br>
<br>
<br>
<br>
------ Original Message ------ From: "Cary Lewis"<br>
<<a href="mailto:cary.lewis@gmail.com" target="_blank">cary.lewis@gmail.com</a>> To: <a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a> Sent: 2/10/2014 3:56:35<br>
PM Subject: Re: Rsync performance with large exchange database<br>
files<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
when you were doing rsync from /cygdrive/c to /cygdrive/d was<br>
the exchange file already there? Or was it clean copy?<br>
<br>
On Mon, Feb 10, 2014 at 4:03 PM, <<a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Okay, so I've done some testing..<br>
<br>
I created a roughly 4gb file from one of the smaller exchange<br>
database files.<br>
<br>
If I copy that to remotely to my desktop, I get about<br>
45-50MB/sec read speed off the D (exchange database) drive. If<br>
I copy that back to the C drive (just the OS) for the Windows<br>
server it writes to the C drive at almost 100MB/sec over the<br>
network.<br>
<br>
If I copy directly from the server D drive to it's C drive<br>
using windows it's around 45MB/sec<br>
<br>
Inside cygwin using just the copy command I get about<br>
35-45MB/sec transfer speed so there's a little hit just from<br>
cygwin.<br>
<br>
Using rsync to "sync" the file from the D drive to the C drive<br>
with the --progress option. I'm getting about 2-2.5MB/sec<br>
transfer speed<br>
<br>
The server is being used... So I've run the tests a few times<br>
thoughout the last hour or so and these are about my average<br>
numbers.<br>
<br>
Why would rsync be so much slower? Is there something I can<br>
test to help figure this out? I'm using rsync on a couple dozen<br>
Windows servers and it's been working great so I'm not sure why<br>
this one is acting weird.<br>
<br>
<br>
<br>
------ Original Message ------ From: "Kevin Korb"<br>
<<a href="mailto:kmk@sanitarium.net" target="_blank">kmk@sanitarium.net</a>> To: <a href="mailto:rsync@lists.samba.org" target="_blank">rsync@lists.samba.org</a> Sent: 2/10/2014<br>
10:57:08 AM Subject: Re: Rsync performance with large exchange<br>
database files<br>
<br>
</blockquote></blockquote>
3.1.0 will probably help some.<br>
<br>
What are the specs of the FreeBSD system? I have found that ZFS on<br>
FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is<br>
the minimum if dedup is disabled and 16BG of RAM for when dedup is<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
enabled.<br>
</blockquote></blockquote></blockquote></blockquote>
<br>
Also, a cache disk helps a lot.<br>
<br>
On 02/10/2014 10:22 AM, <a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
------ Original Message ------ From: <a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a><br>
<mailto:<a href="mailto:bruce@sqls.net" target="_blank">bruce@sqls.net</a>> To: <a href="mailto:rsync@lists.samba.org" target="_blank">rsync@lists.samba.org</a><br>
<mailto:<a href="mailto:rsync@lists.samba.org" target="_blank">rsync@lists.samba.org</a>> Sent: 2/10/2014 8:38:06 AM<br>
Subject: Rsync performance with large exchange database<br>
files<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm using a mixture of FreeBSD w/ ZFS+snapshots and<br>
rsync to backup all the servers at my day job. This<br>
works pretty good overall but on one server it's not<br>
working so well :)<br>
<br>
We have an Exchange 2003 server with 4 separate mail<br>
store databases. One of them is roughly 900GB the<br>
others are ~200GB, ~160GB, and ~50GB. Rsync seems to<br>
spend a lot of time trying to find the differences in<br>
the files. On the Windows server where rsync is kicked<br>
off there's very little CPU or RAM usage for the rsync<br>
client. On the server rsync (rsyncd, no ssh) is using<br>
around 70-85% of a cpu (well, half a cpu due to hyper<br>
threading). I'm using VSS on the windows server to take<br>
a snapshot and expose it then running rsync from that<br>
to avoid locking issues.<br>
<br>
Is there anything I should check to help narrow down<br>
"problems?" or any settings I should try that could<br>
help speed things up any?<br>
<br>
Below is the final output of the last two rsync runs to<br>
give you an idea. It's taking 30-40+ hours to finish<br>
even though it's only transferring 80-160GB of change.<br>
Right now I'm testing this against a local rsync server<br>
so it should get pretty fast network performance.<br>
Eventually it will be moved to our off-site backup but<br>
that connection is still pretty fast (20 MBbit) and<br>
the backup is only hitting 800-1000 Kbytes/sec.<br>
<br>
<br>
Number of files: 19 Number of files transferred: 6<br>
Total file size: 1265.74G bytes Total transferred file<br>
size: 1057.06G bytes Literal data: 160.67G bytes<br>
Matched data: 896.39G bytes File list size: 482 File<br>
list generation time: 0.001 seconds File list transfer<br>
time: 0.000 seconds Total bytes sent: 160.71G Total<br>
bytes received: 73.74M<br>
<br>
sent 160.71G bytes received 73.74M bytes 991.84K<br>
bytes/sec total size is 1265.74G speedup is 7.87<br>
[sender] _exit_cleanup(code=0,<br>
file=/home/lapo/package/rsync-<u></u>3.0.9-1/src/rsync-3.0.9/main.<u></u>c,<br>
<br>
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
line=1052): about to call exit(0)<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
real 2833m1.324s user 2225m55.906s sys 45m10.015s<br>
<br>
Number of files: 11 Number of files transferred: 6<br>
Total file size: 1268.78G bytes Total transferred file<br>
size: 1251.04G bytes Literal data: 83.43G bytes Matched<br>
data: 1167.61G bytes File list size: 216 File list<br>
generation time: 1.360 seconds File list transfer time:<br>
0.000 seconds Total bytes sent: 83.48G Total bytes<br>
received: 87.25M<br>
<br>
sent 83.48G bytes received 87.25M bytes 836.85K<br>
bytes/sec total size is 1268.78G speedup is 15.18<br>
<br>
real 1745m5.647s user 1129m14.000s sys 39m58.875s<br>
<br>
Thanks (in advance) for the help :)<br>
</blockquote>
<br>
<br>
rsync options I'm using on the client are : -rltihv<br>
--progress --stats --inplace --modify-window=1<br>
<br>
On the windows client I'm using cygwin + rsync 3.0.9 but<br>
I'm going to test 3.1.0 there and see if there's a<br>
difference.<br>
<br>
On the server it's rsync 3.1.0 running rsyncd.<br>
<br>
Perhaps useful bit of information :).<br>
<br>
<br>
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-- Please use reply-all for most replies to avoid omitting<br>
the mailing list. To unsubscribe or change options:<br>
<a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/<u></u>mailman/listinfo/rsync</a> Before<br>
posting, read:<br>
<a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/<u></u>smart-questions.html</a><br>
</blockquote>
<br>
<br>
-- Please use reply-all for most replies to avoid omitting the<br>
mailing list. To unsubscribe or change options:<br>
<a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/<u></u>mailman/listinfo/rsync</a> Before posting,<br>
read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/<u></u>smart-questions.html</a><br>
</blockquote></blockquote>
<br>
</blockquote>
<br>
- --<br>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^<u></u>`'~*-,._.,-*~'`^`'~*-,._.,-*~'<u></u>`^`'~*-,._.,-*~<br>
Kevin Korb Phone: (407) 252-6853<br>
Systems Administrator Internet:<br>
FutureQuest, Inc. Kevin@FutureQuest.net (work)<br>
Orlando, Florida <a href="mailto:kmk@sanitarium.net" target="_blank">kmk@sanitarium.net</a> (personal)<br>
Web page: <a href="http://www.sanitarium.net/" target="_blank">http://www.sanitarium.net/</a><br>
PGP public key available on web site.<br>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^<u></u>`'~*-,._.,-*~'`^`'~*-,._.,-*~'<u></u>`^`'~*-,._.,-*~<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.22 (GNU/Linux)<br>
Comment: Using GnuPG with Thunderbird - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
iEYEARECAAYFAlL5TgsACgkQVKC1jl<u></u>bQAQfNTwCfUV1iYYTujpQLHQxdBD1K<u></u>FLm8<br>
N9kAoMlhdECrmIT/<u></u>LwC0tUHgXcYMjDfI<br>
=42pY<br>
-----END PGP SIGNATURE-----<span class="HOEnZb"><font color="#888888"><br>
--<br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/<u></u>mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/<u></u>smart-questions.html</a><br>
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/<u></u>mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/<u></u>smart-questions.html</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Jonathan Aquilina
</div>