Is there a better way to transfer data that doesn't use so much cache?

[Robin Lee Powell]

On Wed, Aug 03, 2022 at 02:04:22PM -0400, Rob Campbell via rsync wrote:
> I've created a script that syncs (and removes) data from as many as 4
> places and puts them all in one of 2 directories.  The commands are:
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
> -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *'
> "$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
> -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/"
> $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'-
> Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng'
> -f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+
> Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *'
> $PHONE/ $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *'
> --files-from=<(find $PHONE -type f ! -path "*Download*" ! -path
> "*.trashed*" ! -iname ????????????????????????????????.mp4 ! -iname
> '*.mp4\.*')/ $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+
> *Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/
> $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'- *' --files-from=<(find $PHONE -iname
> ????????????????????????????????.mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *'
> $PHONE/Downloads/ $COMPUTER/Downloads/
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'-
> screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png'
> -f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'-
> $STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+
> Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ |
> tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp'
> -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *'
> $HOME/Downloads $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *'
> $MYVIDEOS/ $STAGINGV/ | tee -a $LOG
> 
> 
> The problem isn't that there are many syncs because the problem happens on
> the first one that runs.

You didn't actually say what the problem *is*.

I can infer from the subject that you think it's bad that rsync is
using a bunch of disk/buffer cache, but that's not rsync, that's
Linux, and it's by design; Linux uses as much RAM as it possibly can
for disk cache, always.  This improves performance.  In a
well-performing Linux system, the "free" column of "free -h" is very
low, and the "available" column is very high.

> Before any of them run I run:
> 
> sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h
> 
> I do not run this before each one because it sometimes takes a while to
> /proc/sys/vm/drop_caches

That's a great way to substantially reduce performance; why are you
doing that?

> Is there something in the logic that can be done to make this perform
> better or should I use something other than rsync or is what I am getting
> as good as it will get regardless of what I use?
> 
> Some of these directories can be over a gig.  Most of these are media files
> and should have exif data that has the timestamp so maybe I can get rid of
> -t but it is easier to keep the timestamp of the file rather than running
> exiftool to also use the create date to "touch" the file but maybe using
> exiftool is a faster way?
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> In all things, Be Intentional.

> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html