--link-dest. Time to 'building file list' incrementing

[John Simpson]

I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen.

The command is as follows...

rsync -rlptDhPR \
    --password-file=password \
    --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
    --Stats \
    --delete \
    --log-file=logfilename \
    --link-dest=linkdestpath \
    sourceDirectory \
    rsync at 192.168.1.2::destinationDirectory

I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up.

The first log file looked like this...

2018/10/01 23:00:14 [2164] building file list
...transfer file list here
2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1)
2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
2018/10/02 02:11:30 [2164] File list size: 10.42M
2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K bytes/sec
2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79

the most recent looks like this...

2018/11/24 23:00:15 [2924] building file list
2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/
2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1)
2018/11/25 13:21:16 [2924] Number of created files: 0
2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
2018/11/25 13:21:16 [2924] Literal data: 0 bytes
2018/11/25 13:21:16 [2924] Matched data: 0 bytes
2018/11/25 13:21:16 [2924] File list size: 10.35M
2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K bytes/sec
2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96

As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include the whole log file which correctly identifies that no files have changed)

The number of files is as might be expected as is everything else. It's just the time taken "building file list" is significantly larger.