--link-dest. Time to 'building file list' incrementing

Ben RUBSON ben.rubson at gmail.com
Tue Jan 8 10:03:58 UTC 2019


Hi,

As you are on Cygwin, you should consider the notexec & noacl mount options :
https://cygwin.com/cygwin-ug-net/using.html#mount-table <https://cygwin.com/cygwin-ug-net/using.html#mount-table>

They impact stat() performance.

Ben

> On 8 Jan 2019, at 10:56, John Simpson via rsync <rsync at lists.samba.org> wrote:
> 
> Any ideas anyone?
> 
> I still need at least a weekly backup of all data.
> 
> The current workaround is just for the most active directories.
> 
> Are there any diagnostics I can do which might shed some light on this?
> 
> Thanks
> 
> John
> 
>> On 4 Jan 2019 09:53, John Simpson via rsync <rsync at lists.samba.org> wrote:
>> 
>> Kevin
>> 
>> The link-dest parameter is a single directory (the previous day's directory), the destination is today's directory.
>> 
>> I haven't tried deleting a backup,  there's no particular need in space terms,  at the current rate there's enough space for several years of daily backups.
>> 
>> I've reverted to daily backups on a small subset of the total; the full backup now takes around 30 hours.  Clearly not practical.
>> 
>> As the small subset takes only a few minutes to complete I can't yet see if this time is incrementing too.
>> 
>> John   On 3 Jan 2019 17:06, Kevin Korb via rsync <rsync at lists.samba.org> wrote:
>>> 
>>> It does normally take some time to analyze large trees of files.  It has 
>>> to call stat() on each file to get the size and timestamp. 
>>> 
>>> However, 15 hours seems a bit excessive even though I have never tried 
>>> to do this on Windows or a NAS system.  Just to be clear, is your 
>>> --link-dest parameter a single directory or are you trying to tell it to 
>>> use all of the previous backups? 
>>> 
>>> Also, have you deleted a backup yet?  In my experience that takes a lot 
>>> longer than running one so if you need 15 hours to run a backup I would 
>>> expect deleting one to take about a week. 
>>> 
>>> On 1/3/19 4:23 AM, John Simpson via rsync wrote: 
>>>> 
>>>> 
>>>> I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen. 
>>>> 
>>>> The command is as follows... 
>>>> 
>>>> rsync -rlptDhPR \ 
>>>>      --password-file=password \ 
>>>>      --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ 
>>>>      --Stats \ 
>>>>      --delete \ 
>>>>      --log-file=logfilename \ 
>>>>      --link-dest=linkdestpath \ 
>>>>      sourceDirectory \ 
>>>>      rsync at 192.168.1.2::destinationDirectory 
>>>> 
>>>> I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up. 
>>>> 
>>>> The first log file looked like this... 
>>>> 
>>>> 2018/10/01 23:00:14 [2164] building file list 
>>>> ...transfer file list here 
>>>> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1) 
>>>> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 
>>>> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 
>>>> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 
>>>> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 
>>>> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 
>>>> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 
>>>> 2018/10/02 02:11:30 [2164] File list size: 10.42M 
>>>> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 
>>>> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 
>>>> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 
>>>> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 
>>>> 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K bytes/sec 
>>>> 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79 
>>>> 
>>>> the most recent looks like this... 
>>>> 
>>>> 2018/11/24 23:00:15 [2924] building file list 
>>>> 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/ 
>>>> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1) 
>>>> 2018/11/25 13:21:16 [2924] Number of created files: 0 
>>>> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 
>>>> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 
>>>> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 
>>>> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 
>>>> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 
>>>> 2018/11/25 13:21:16 [2924] File list size: 10.35M 
>>>> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 
>>>> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 
>>>> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 
>>>> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 
>>>> 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K bytes/sec 
>>>> 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96 
>>>> 
>>>> As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include the whole log file which correctly identifies that no files have changed) 
>>>> 
>>>> The number of files is as might be expected as is everything else. It's just the time taken "building file list" is significantly larger. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20190108/2f5dcb2f/attachment.html>


More information about the rsync mailing list