patch to stop rsync from polluting the filesystem cache
tobi at oetiker.ch
Sun Apr 22 15:42:23 GMT 2007
I am using rsync for hard-link backup. I found that there is a
major problem with frequent backup filling up the file system cache
with all the data from the files being backed up. The effect is
that all the other 'sensible' data in the cache gets thrown out in
the process. This is rather unfortunate as the performance of the
system becomes very bad after running rsync.
Some research showed, that
posix_fadvise64(fd, 0, 0,POSIX_FADV_DONTNEED);
would tell the OS that it should not keep the file in cache. I
have written a patch for rsync that adds the
option which activates posix_fadvise64.
There are some caveats though:
* When calling posix_fadvise64 while writing a file, only the
part of the cache will be release which has already been
written to disk. This means we have to call fdatasync before
calling posix_fadvise64 and this will unfortunately slow down
operations considerably. On my test system I get 240 KByte/s.
The patch has been optimized, so that the impact on large files
will be considerably lowered by calling posix_fadvise64 only
after a few megabytes have been written.
* When reading a file which has been cached *Before* rsync read
it, the content of the file will be released from cache never
the less, which may not be intended. I have unfortunately not
found a method for determining if a file is in cache or not
I found that running rsync of an lvm snapshot is a good way
around this problem, since the snapshot data is cached
separately from the original. It has the additional benefit of
making the backups more consistent.
* I don't really know the rsync code, so it may be that the patch
is calling fadvise for files where this would not be necessary.
* The patch is tested only on Linux 2.6.18
If you have any input on this, please let me know.
You can get the patch from
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten
http://it.oetiker.ch tobi at oetiker.ch ++41 62 213 9902
More information about the rsync