retrieve files without local dir compare
paul+rsync at wurtel.net
Wed Oct 14 04:48:09 MDT 2009
On Wed 14 Oct 2009, Marc Mertes wrote:
> Why I need that:
> We receive meteorological data from a remote server to a local directory
> (every 5 min).
> If the data is here, it is imported by a special software, after the
> import it will be deleted from that directory. The deleting can't be
> Normally I would say, ok download all again, but we get 80GB data per day.
> If rsync compares the local dir it will download all again, because it's
> So rsync has to know what is already downloaded, and only get the new
> files WITHOUT the dir compare.
I would use --remove-source-files, but that will probably need some
adjusting on the source end of the transfer.
I've built a simple script that distributes uploaded images etc. to a
number of different (load balanced) webservers. For each webserver I
link the uploaded file to a directory, and the webservers fetch the
files using --remove-source-files.
That ensures that all webservers get their files, and no bulk of data
remains on the distribution system. The principle of your "problem"
sounds similar, but as I said is probably difficult to implement without
the source system being modified.
> Til today we are doing this with ftp, but it's very unstable and not
> working good, so I would like to
> do this with rsync - if possible.
How do you know what files to FTP? With FTP you can also retrieve the
full 80GB every time... If you know what files to fetch, that's easy to
implement with rsync as well. You could also consider using
--exclude-from to exclude those files you've already seen.
More information about the rsync