keep rsync from removing unfinished source files?
Aaron Swartz
me at aaronsw.com
Sun Sep 7 14:59:33 GMT 2008
I have two machines, speed and mass. speed has a fast Internet
connection and is running a crawler which downloads a lot of files to
disk. mass has a lot of disk space. I want to move the files from
speed to mass after they're done downloading. Ideally, I'd just run:
$ rsync --remove-source-files speed:/var/crawldir .
but I worry that rsync will unlink a source file that hasn't finished
downloading yet. (I looked at the source code and I didn't see
anything protecting against this.) Any suggestions?
Ideas I had were:
- a pause between downloading the file list and downloading the files
- an exclude rule for recently modified files
- a check to not delete a file if its file size has changed since it was copied
but I don't see any way to do any of these.
More information about the rsync
mailing list