why rsync update large directory so slow?

风形 149159337 at qq.com
Tue Nov 24 13:22:58 UTC 2015


Hi:
   I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
   rsync server config:
 ------------------------------------------------------------------------------------------------------
 strict modes = false
hosts allow = 10.200.121.104
hosts allow = 10.171.209.87
hosts deny = *
log file = /cygdrive/d/log/rsyncd.log
 port=8730
uid = 0
gid = 0
 # Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
[d_hst]
path = /cygdrive/d/hst
read only = false
transfer logging = yes
auth users = admin
secrets file = etc/user.pass
list = no
ignore errors
 ------------------------------------------------------------------------------------------------------
  
 client bat script:
 ------------------------------------------------------------------------------------------------------
 @echo off
 set RSYNC_HOME="c:\Program Files (x86)\cwRsync\bin\"
cd /d %RSYNC_HOME%
 :: Source root directory
set src_dir=rsync://admin@10.160.27.48:8730/d_hst
 :: Destination root directory
set dst_dir=/cygdrive/e/filetest
 :: Make sure path without space
set pwd_file="/cygdrive/c/user.pass"
 :: Log file
set log_dir=/cygdrive/d/log
 :: Synchronize Files, with 16 connections according to 16 directories(0~f)
for %%i in (0 1 2 3 4 5 6) do (
setlocal enabledelayedexpansion
set log_file=%log_dir%/%date:~3,4%%date:~8,2%%date:~11,2%%time:~0,2%%time:~3,2%%time:~6,2%_%%i.txt
:: Remove space in log_file
set "log_file=!log_file: =!"
start cmd /c rsync -avvu --delete %src_dir%/Files/%%i %dst_dir%/Files --password-file=%pwd_file% --log-file=!log_file! --log-file-format=""
)
------------------------------------------------------------------------------------------------------   
 In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. 
 When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours. 
 It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
 Does anybody encounted cases like this? Any help will be thankful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20151124/40ebcf55/attachment.html>


More information about the rsync mailing list