--link-dest. Time to 'building file list' incrementing

[Kevin Korb]

It does normally take some time to analyze large trees of files.  It has
to call stat() on each file to get the size and timestamp.

However, 15 hours seems a bit excessive even though I have never tried
to do this on Windows or a NAS system.  Just to be clear, is your
--link-dest parameter a single directory or are you trying to tell it to
use all of the previous backups?

Also, have you deleted a backup yet?  In my experience that takes a lot
longer than running one so if you need 15 hours to run a backup I would
expect deleting one to take about a week.

On 1/3/19 4:23 AM, John Simpson via rsync wrote:
> 
> 
> I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen.
> 
> The command is as follows...
> 
> rsync -rlptDhPR \
>     --password-file=password \
>     --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
>     --Stats \
>     --delete \
>     --log-file=logfilename \
>     --link-dest=linkdestpath \
>     sourceDirectory \
>     rsync at 192.168.1.2::destinationDirectory
> 
> I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up.
> 
> The first log file looked like this...
> 
> 2018/10/01 23:00:14 [2164] building file list
> ...transfer file list here
> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1)
> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
> 2018/10/02 02:11:30 [2164] File list size: 10.42M
> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
> 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K bytes/sec
> 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79
> 
> the most recent looks like this...
> 
> 2018/11/24 23:00:15 [2924] building file list
> 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/
> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1)
> 2018/11/25 13:21:16 [2924] Number of created files: 0
> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes
> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes
> 2018/11/25 13:21:16 [2924] File list size: 10.35M
> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
> 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K bytes/sec
> 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96
> 
> As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include the whole log file which correctly identifies that no files have changed)
> 
> The number of files is as might be expected as is everything else. It's just the time taken "building file list" is significantly larger.
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20190103/80f1193d/signature.sig>