Rsync performance with large exchange database files
bruce at sqls.net
bruce at sqls.net
Mon Feb 10 15:24:41 MST 2014
Well the local test was just to a test to see if I could understand why
the remote sync of the exchange database was so slow. I've heard that
rsync is less efficient for local copies but this isn't like 80% the
performance, or half the performance.. It's a massive difference - which
I wasn't expecting to see.
I have looked at unison but I also backup several linux and bsd systems
here and rsync just seemed like a good choice since it's still be
developed and it works on everything.
In the end, I'm still looking to understand why a sync of a large
exchange database file is taking 30-40 hours to finish and if there's
anything I can do to help reduce that window. I can't add a cache drive
to the FreeBSD server very easily at the moment so I was trying to
narrow down if the issue is on the BSD side or Windows side or maybe a
mix of both.
I wish a native windows client of rsync existed :)
------ Original Message ------
From: "Kevin Korb" <kmk at sanitarium.net>
To: rsync at lists.samba.org
Sent: 2/10/2014 4:09:15 PM
Subject: Re: Rsync performance with large exchange database files
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Rsync is known to be pretty inefficient on local copies (-W is forced
>there btw) and cygwin doesn't really help with that either.
>Essentially, when not networking rsync isn't much smarter than cp but
>it has a ton of extra overhead.
>
>Also, maybe you want unison since there is a native Windows version of
>it?
>
>On 02/10/2014 05:05 PM, bruce at sqls.net wrote:
>>
>> Clean copy. I even used the -W flag to see if it made a difference
>> but, nope.
>>
>> I'm testing this same test on some of my other servers too. See if
>> there's any common-ground I can find.
>>
>> On another servers (MS SQL Server) with faster disks I tried a
>> similar test just now. There's only the C drive on this server but
>> I used my same test file from the other server and used rsync
>> (3.1.0) to copy the folder from one folder to another folder and it
>> kicked off and got up to about 25MB/sec. Thing is if I just use
>> windows to copy the same file from one folder to the other it does
>> the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
>> so that's also a pretty massive difference.
>>
>> Maybe this is normal and I've just not noticed it on these other
>> servers since they have a much smaller amount of data to backup?
>> Still seems like some thing is wrong. I wouldn't expect the speed
>> difference to be that huge.
>>
>>
>>
>> ------ Original Message ------ From: "Cary Lewis"
>> <cary.lewis at gmail.com> To: bruce at sqls.net Sent: 2/10/2014 3:56:35
>> PM Subject: Re: Rsync performance with large exchange database
>> files
>>
>>> when you were doing rsync from /cygdrive/c to /cygdrive/d was
>>> the exchange file already there? Or was it clean copy?
>>>
>>> On Mon, Feb 10, 2014 at 4:03 PM, <bruce at sqls.net> wrote:
>>>> Okay, so I've done some testing..
>>>>
>>>> I created a roughly 4gb file from one of the smaller exchange
>>>> database files.
>>>>
>>>> If I copy that to remotely to my desktop, I get about
>>>> 45-50MB/sec read speed off the D (exchange database) drive. If
>>>> I copy that back to the C drive (just the OS) for the Windows
>>>> server it writes to the C drive at almost 100MB/sec over the
>>>> network.
>>>>
>>>> If I copy directly from the server D drive to it's C drive
>>>> using windows it's around 45MB/sec
>>>>
>>>> Inside cygwin using just the copy command I get about
>>>> 35-45MB/sec transfer speed so there's a little hit just from
>>>> cygwin.
>>>>
>>>> Using rsync to "sync" the file from the D drive to the C drive
>>>> with the --progress option. I'm getting about 2-2.5MB/sec
>>>> transfer speed
>>>>
>>>> The server is being used... So I've run the tests a few times
>>>> thoughout the last hour or so and these are about my average
>>>> numbers.
>>>>
>>>> Why would rsync be so much slower? Is there something I can
>>>> test to help figure this out? I'm using rsync on a couple dozen
>>>> Windows servers and it's been working great so I'm not sure why
>>>> this one is acting weird.
>>>>
>>>>
>>>>
>>>> ------ Original Message ------ From: "Kevin Korb"
>>>> <kmk at sanitarium.net> To: rsync at lists.samba.org Sent: 2/10/2014
>>>> 10:57:08 AM Subject: Re: Rsync performance with large exchange
>>>> database files
>>>>
>> 3.1.0 will probably help some.
>>
>> What are the specs of the FreeBSD system? I have found that ZFS on
>> FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
>> the minimum if dedup is disabled and 16BG of RAM for when dedup is
>>>>>> enabled.
>>
>> Also, a cache disk helps a lot.
>>
>> On 02/10/2014 10:22 AM, bruce at sqls.net wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------ Original Message ------ From: bruce at sqls.net
>>>>>>> <mailto:bruce at sqls.net> To: rsync at lists.samba.org
>>>>>>> <mailto:rsync at lists.samba.org> Sent: 2/10/2014 8:38:06 AM
>>>>>>> Subject: Rsync performance with large exchange database
>>>>>>> files
>>>>>>>
>>>>>>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and
>>>>>>>> rsync to backup all the servers at my day job. This
>>>>>>>> works pretty good overall but on one server it's not
>>>>>>>> working so well :)
>>>>>>>>
>>>>>>>> We have an Exchange 2003 server with 4 separate mail
>>>>>>>> store databases. One of them is roughly 900GB the
>>>>>>>> others are ~200GB, ~160GB, and ~50GB. Rsync seems to
>>>>>>>> spend a lot of time trying to find the differences in
>>>>>>>> the files. On the Windows server where rsync is kicked
>>>>>>>> off there's very little CPU or RAM usage for the rsync
>>>>>>>> client. On the server rsync (rsyncd, no ssh) is using
>>>>>>>> around 70-85% of a cpu (well, half a cpu due to hyper
>>>>>>>> threading). I'm using VSS on the windows server to take
>>>>>>>> a snapshot and expose it then running rsync from that
>>>>>>>> to avoid locking issues.
>>>>>>>>
>>>>>>>> Is there anything I should check to help narrow down
>>>>>>>> "problems?" or any settings I should try that could
>>>>>>>> help speed things up any?
>>>>>>>>
>>>>>>>> Below is the final output of the last two rsync runs to
>>>>>>>> give you an idea. It's taking 30-40+ hours to finish
>>>>>>>> even though it's only transferring 80-160GB of change.
>>>>>>>> Right now I'm testing this against a local rsync server
>>>>>>>> so it should get pretty fast network performance.
>>>>>>>> Eventually it will be moved to our off-site backup but
>>>>>>>> that connection is still pretty fast (20 MBbit) and
>>>>>>>> the backup is only hitting 800-1000 Kbytes/sec.
>>>>>>>>
>>>>>>>>
>>>>>>>> Number of files: 19 Number of files transferred: 6
>>>>>>>> Total file size: 1265.74G bytes Total transferred file
>>>>>>>> size: 1057.06G bytes Literal data: 160.67G bytes
>>>>>>>> Matched data: 896.39G bytes File list size: 482 File
>>>>>>>> list generation time: 0.001 seconds File list transfer
>>>>>>>> time: 0.000 seconds Total bytes sent: 160.71G Total
>>>>>>>> bytes received: 73.74M
>>>>>>>>
>>>>>>>> sent 160.71G bytes received 73.74M bytes 991.84K
>>>>>>>> bytes/sec total size is 1265.74G speedup is 7.87
>>>>>>>> [sender] _exit_cleanup(code=0,
>>>>>>>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c,
>>>>>>>>
>>>>>>>>
>line=1052): about to call exit(0)
>>>>>>>>
>>>>>>>> real 2833m1.324s user 2225m55.906s sys 45m10.015s
>>>>>>>>
>>>>>>>> Number of files: 11 Number of files transferred: 6
>>>>>>>> Total file size: 1268.78G bytes Total transferred file
>>>>>>>> size: 1251.04G bytes Literal data: 83.43G bytes Matched
>>>>>>>> data: 1167.61G bytes File list size: 216 File list
>>>>>>>> generation time: 1.360 seconds File list transfer time:
>>>>>>>> 0.000 seconds Total bytes sent: 83.48G Total bytes
>>>>>>>> received: 87.25M
>>>>>>>>
>>>>>>>> sent 83.48G bytes received 87.25M bytes 836.85K
>>>>>>>> bytes/sec total size is 1268.78G speedup is 15.18
>>>>>>>>
>>>>>>>> real 1745m5.647s user 1129m14.000s sys 39m58.875s
>>>>>>>>
>>>>>>>> Thanks (in advance) for the help :)
>>>>>>>
>>>>>>>
>>>>>>> rsync options I'm using on the client are : -rltihv
>>>>>>> --progress --stats --inplace --modify-window=1
>>>>>>>
>>>>>>> On the windows client I'm using cygwin + rsync 3.0.9 but
>>>>>>> I'm going to test 3.1.0 there and see if there's a
>>>>>>> difference.
>>>>>>>
>>>>>>> On the server it's rsync 3.1.0 running rsyncd.
>>>>>>>
>>>>>>> Perhaps useful bit of information :).
>>>>>>>
>>>>>>>
>>>>>>>
>>
>>>>> -- Please use reply-all for most replies to avoid omitting
>>>>> the mailing list. To unsubscribe or change options:
>>>>> https://lists.samba.org/mailman/listinfo/rsync Before
>>>>> posting, read:
>>>>> http://www.catb.org/~esr/faqs/smart-questions.html
>>>>
>>>>
>>>> -- Please use reply-all for most replies to avoid omitting the
>>>> mailing list. To unsubscribe or change options:
>>>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>>>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>
>- --
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2.0.22 (GNU/Linux)
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8
>N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI
>=42pY
>-----END PGP SIGNATURE-----
>--
>Please use reply-all for most replies to avoid omitting the mailing
>list.
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read:
>http://www.catb.org/~esr/faqs/smart-questions.html
More information about the rsync
mailing list