Is there a better way to transfer data that doesn't use so much cache?

[Rob Campbell]

I've created a script that syncs (and removes) data from as many as 4
places and puts them all in one of 2 directories.  The commands are:

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
-f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *'
"$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
-f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/"
$STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'-
Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng'
-f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+
Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *'
$PHONE/ $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *'
--files-from=<(find $PHONE -type f ! -path "*Download*" ! -path
"*.trashed*" ! -iname ????????????????????????????????.mp4 ! -iname
'*.mp4\.*')/ $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+
*Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/
$STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'- *' --files-from=<(find $PHONE -iname
????????????????????????????????.mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *'
$PHONE/Downloads/ $COMPUTER/Downloads/

rsync -avt --progress --remove-source-files --info=progress2 -f'-
screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png'
-f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'-
$STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+
Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ |
tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp'
-f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *'
$HOME/Downloads $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *'
$MYVIDEOS/ $STAGINGV/ | tee -a $LOG


The problem isn't that there are many syncs because the problem happens on
the first one that runs.  Before any of them run I run:

sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h

I do not run this before each one because it sometimes takes a while to
/proc/sys/vm/drop_caches

Is there something in the logic that can be done to make this perform
better or should I use something other than rsync or is what I am getting
as good as it will get regardless of what I use?

Some of these directories can be over a gig.  Most of these are media files
and should have exif data that has the timestamp so maybe I can get rid of
-t but it is easier to keep the timestamp of the file rather than running
exiftool to also use the create date to "touch" the file but maybe using
exiftool is a faster way?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In all things, Be Intentional.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20220803/127a1262/attachment.htm>