keep rsync from removing unfinished source files?

Aaron Swartz me at
Sun Sep 7 14:59:33 GMT 2008

I have two machines, speed and mass. speed has a fast Internet
connection and is running a crawler which downloads a lot of files to
disk. mass has a lot of disk space. I want to move the files from
speed to mass after they're done downloading. Ideally, I'd just run:

    $ rsync --remove-source-files speed:/var/crawldir .

but I worry that rsync will unlink a source file that hasn't finished
downloading yet. (I looked at the source code and I didn't see
anything protecting against this.) Any suggestions?

Ideas I had were:
 - a pause between downloading the file list and downloading the files
 - an exclude rule for recently modified files
 - a check to not delete a file if its file size has changed since it was copied
but I don't see any way to do any of these.

More information about the rsync mailing list