why rsync update large directory so slow?
风形
149159337 at qq.com
Tue Nov 24 13:22:58 UTC 2015
Hi:
I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
rsync server config:
------------------------------------------------------------------------------------------------------
strict modes = false
hosts allow = 10.200.121.104
hosts allow = 10.171.209.87
hosts deny = *
log file = /cygdrive/d/log/rsyncd.log
port=8730
uid = 0
gid = 0
# Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
[d_hst]
path = /cygdrive/d/hst
read only = false
transfer logging = yes
auth users = admin
secrets file = etc/user.pass
list = no
ignore errors
------------------------------------------------------------------------------------------------------
client bat script:
------------------------------------------------------------------------------------------------------
@echo off
set RSYNC_HOME="c:\Program Files (x86)\cwRsync\bin\"
cd /d %RSYNC_HOME%
:: Source root directory
set src_dir=rsync://admin@10.160.27.48:8730/d_hst
:: Destination root directory
set dst_dir=/cygdrive/e/filetest
:: Make sure path without space
set pwd_file="/cygdrive/c/user.pass"
:: Log file
set log_dir=/cygdrive/d/log
:: Synchronize Files, with 16 connections according to 16 directories(0~f)
for %%i in (0 1 2 3 4 5 6) do (
setlocal enabledelayedexpansion
set log_file=%log_dir%/%date:~3,4%%date:~8,2%%date:~11,2%%time:~0,2%%time:~3,2%%time:~6,2%_%%i.txt
:: Remove space in log_file
set "log_file=!log_file: =!"
start cmd /c rsync -avvu --delete %src_dir%/Files/%%i %dst_dir%/Files --password-file=%pwd_file% --log-file=!log_file! --log-file-format=""
)
------------------------------------------------------------------------------------------------------
In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job.
When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours.
It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
Does anybody encounted cases like this? Any help will be thankful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20151124/40ebcf55/attachment.html>
More information about the rsync
mailing list