rsync dest high disk write request avg waiting time (w_await) and probable low speed

[Charles]

Is it likely that rsync is the cause of the symptoms?

If so, how and can the rsync command be changed to prevent the problem?

While running this command ...

rsync --filter=protect _Changed and deleted files/ --exclude=/_Changed 
and deleted files/ --backup --backup-dir=_Changed and deleted 
files/2021/Sep/13 at 01:20 --compress --delete-during --delete-excluded 
--devices --group --links --owner --numeric-ids --perms --timeout=600 
--protect-args --recursive --specials --times --verbose 
/home/vbox/VirtualBox VMs/aems3.iciti.av/ 
vbox2_to_backup.sar:/srv/backup/vms.vbox2.iciti.av

... Zabbix reported reported high disk write request avg waiting time 
(w_await) values of ~40 ms

The VirtualBox VM was shut down during the sync so the source files were 
static.  There were seven files to be synced, six trivially small and 
one 38 GB.  The sync took ~0.5 hours and summarised
sent 1,009,799,482 bytes  received 2,428,622 bytes  636,021.43 bytes/sec
total size is 39,736,701,202  speedup is 39.26

Investigation showed the high w_await values were associated with rsync 
rather than the HDD

1. Moved the destination to a different HDD.  Same

2. Re-scheduled to a different time.  Same

3. Wrote to the HDD locally
dd if=/dev/zero of=/srv/backup/vms.vbox2.iciti.av/trash
That wrote at ~130 MB/s and gave w_await 56 ms

4. Wrote to the HDD from the source computer
scp -p '/home/vbox/VirtualBox VMs/aems3.iciti.av/aems3.iciti.av.vdi' \
     storage4.iciti.av:/srv/backup/vms.vbox2.iciti.av/trash
... 
                                                                    100% 
   37GB  76.5MB/s   08:15
...
That wrote at ~76 MB/s and gave w_await 62 ms

5. Experimented with --bwlimit
Progressively reducing from 50M to 5M made no significant difference to 
the time taken to sync (stayed around 20 to 30 m) or w_await (stayed 
around 40 ms).
Only reducing from 5M to 3M made a significant difference  to the time 
taken to sync (60 m) while w_await stayed around 40 ms.