Rsync performance with large exchange database files

Mon Feb 10 14:03:35 MST 2014

  Okay, so I've done some testing..

I created a roughly 4gb file from one of the smaller exchange database 
files.

If I copy that to remotely to my desktop, I get about 45-50MB/sec read 
speed off the D (exchange database) drive.  If I copy that back to the C 
drive (just the OS) for the Windows server it writes to the C drive at 
almost 100MB/sec over the network.

If I copy directly from the server D drive to it's C drive using windows 
it's around 45MB/sec

Inside cygwin using just the copy command I get about 35-45MB/sec 
transfer speed so there's a little hit just from cygwin.

Using rsync to "sync" the file from the D drive to the C drive with the 
--progress option.  I'm getting about 2-2.5MB/sec transfer speed

The server is being used... So I've run the tests a few times thoughout 
the last hour or so and these are about my average numbers.

Why would rsync be so much slower?  Is there something I can test to 
help figure this out?  I'm using rsync on a couple dozen Windows servers 
and it's been working great so I'm not sure why this one is acting 
weird.

------ Original Message ------
From: "Kevin Korb" <kmk at sanitarium.net>
To: rsync at lists.samba.org
Sent: 2/10/2014 10:57:08 AM
Subject: Re: Rsync performance with large exchange database files

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>3.1.0 will probably help some.
>
>What are the specs of the FreeBSD system? I have found that ZFS on
>FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is the
>minimum if dedup is disabled and 16BG of RAM for when dedup is enabled.
>
>Also, a cache disk helps a lot.
>
>On 02/10/2014 10:22 AM, bruce at sqls.net wrote:
>>
>>
>>  ------ Original Message ------ From: bruce at sqls.net
>>  <mailto:bruce at sqls.net> To: rsync at lists.samba.org
>>  <mailto:rsync at lists.samba.org> Sent: 2/10/2014 8:38:06 AM Subject:
>>  Rsync performance with large exchange database files
>>
>>>  I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to
>>>  backup all the servers at my day job. This works pretty good
>>>  overall but on one server it's not working so well :)
>>>
>>>  We have an Exchange 2003 server with 4 separate mail store
>>>  databases. One of them is roughly 900GB the others are ~200GB,
>>>  ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to
>>>  find the differences in the files. On the Windows server where
>>>  rsync is kicked off there's very little CPU or RAM usage for the
>>>  rsync client. On the server rsync (rsyncd, no ssh) is using
>>>  around 70-85% of a cpu (well, half a cpu due to hyper threading).
>>>  I'm using VSS on the windows server to take a snapshot and expose
>>>  it then running rsync from that to avoid locking issues.
>>>
>>>  Is there anything I should check to help narrow down "problems?"
>>>  or any settings I should try that could help speed things up
>>>  any?
>>>
>>>  Below is the final output of the last two rsync runs to give you
>>>  an idea. It's taking 30-40+ hours to finish even though it's
>>>  only transferring 80-160GB of change. Right now I'm testing this
>>>  against a local rsync server so it should get pretty fast network
>>>  performance. Eventually it will be moved to our off-site backup
>>>  but that connection is still pretty fast (20 MBbit) and the
>>>  backup is only hitting 800-1000 Kbytes/sec.
>>>
>>>
>>>  Number of files: 19 Number of files transferred: 6 Total file
>>>  size: 1265.74G bytes Total transferred file size: 1057.06G bytes
>>>  Literal data: 160.67G bytes Matched data: 896.39G bytes File list
>>>  size: 482 File list generation time: 0.001 seconds File list
>>>  transfer time: 0.000 seconds Total bytes sent: 160.71G Total
>>>  bytes received: 73.74M
>>>
>>>  sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total
>>>  size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0,
>>>  file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c,
>>>  line=1052): about to call exit(0)
>>>
>>>  real 2833m1.324s user 2225m55.906s sys 45m10.015s
>>>
>>>  Number of files: 11 Number of files transferred: 6 Total file
>>>  size: 1268.78G bytes Total transferred file size: 1251.04G bytes
>>>  Literal data: 83.43G bytes Matched data: 1167.61G bytes File list
>>>  size: 216 File list generation time: 1.360 seconds File list
>>>  transfer time: 0.000 seconds Total bytes sent: 83.48G Total bytes
>>>  received: 87.25M
>>>
>>>  sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec total
>>>  size is 1268.78G speedup is 15.18
>>>
>>>  real 1745m5.647s user 1129m14.000s sys 39m58.875s
>>>
>>>  Thanks (in advance) for the help :)
>>
>>  rsync options I'm using on the client are : -rltihv --progress
>>  --stats --inplace --modify-window=1
>>
>>  On the windows client I'm using cygwin + rsync 3.0.9 but I'm going
>>  to test 3.1.0 there and see if there's a difference.
>>
>>  On the server it's rsync 3.1.0 running rsyncd.
>>
>>  Perhaps useful bit of information :).
>>
>>
>>
>
>- --
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>  Kevin Korb Phone: (407) 252-6853
>  Systems Administrator Internet:
>  FutureQuest, Inc. Kevin at FutureQuest.net (work)
>  Orlando, Florida kmk at sanitarium.net (personal)
>  Web page: http://www.sanitarium.net/
>  PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2.0.22 (GNU/Linux)
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iEYEARECAAYFAlL5BOQACgkQVKC1jlbQAQeEVACgl7LEtMrX52lMqNukkCn3+poR
>k/oAoOdugEyjn5AeP54zvbwazhjyIKc4
>=ll+E
>-----END PGP SIGNATURE-----
>--
>Please use reply-all for most replies to avoid omitting the mailing 
>list.
>To unsubscribe or change options: 
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: 
>http://www.catb.org/~esr/faqs/smart-questions.html