rsync of symbolic links bug

[Tom Schmidt]

I am seeing a problem with the way the rsync of symbolic links
is done.  Here is a simple example to duplicate this issue:

prompt> # Create a test case to show rsync of symbolic link problem
prompt> mkdir foo bar
prompt> touch foo/file1 bar/file2
prompt> ln -s bar/file2 file
prompt> rsync -av . ../duplicate
building file list ... done
created directory ../duplicate
./
bar/
bar/file2
file -> bar/file2
foo/
foo/file1
wrote 236 bytes  read 52 bytes  576.00 bytes/sec
total size is 9  speedup is 0.30
prompt> # Point symbolically linked file to a new file
prompt> rm file
prompt> touch bar/file3
prompt> ln -s bar/file3 file
prompt> rsync -av . ../duplicate
building file list ... done
./
bar/
bar/file3
file -> bar/file3
wrote 220 bytes  read 36 bytes  512.00 bytes/sec
total size is 9  speedup is 0.40
prompt> # Point symbolically linked file to another new file
prompt> rm file
prompt> touch foo/file4
prompt> ln -s foo/file4 file
prompt> rsync -av . ../duplicate
building file list ... done
./
file -> foo/file4
foo/
foo/file4
wrote 236 bytes  read 36 bytes  544.00 bytes/sec
total size is 9  speedup is 0.30


ACK!  The symbolically linked file pointing to foo/file4 got
updated first, causing anyone using it to see it pointing at a
non-existant file!

This bug is essentially a race condition that can cause
the mirrored system to see the symbolic link get updated to
point to a non-existant file since the file it is pointing
at has not been mirrored yet.

We see this problem quite often when we are mirroring our
/usr/local filesystem.  For example, if we have /usr/local/lib/libz.so
symbolically linked to /usr/local/zlib/1.1.3/libz.so, but then
update it to point to /usr/local/zlib/1.1.4/libz.so, then
when the filesystem is mirrored, the /usr/local/lib/libz.so
link gets updated long before the /usr/local/zlib/1.1.4 directory
tree gets pushed.  This causes any application that is started
that using libz.so to fail while the rsync is running!  This
downtime can be quite long when pushing many updates across our
WAN to servers on the other side of the world!

It appears that the rsync pushes the updates out alphabetically,
which is why the symbolic link to bar/file3 above worked OK, but
the symbolic link to foo/file4 fails with the race condition.

The fix for this would be to have rsync internally sort its filelist
to push the updated files and directories first, then the updated
symbolic links last.  How can this be done?  Or am I missing
some rsync commandline issue to fix this problem?

My only workaround at this point is to rsync twice, the first time
without pushing any symbolic links, but this workaround can
still burn us if an update is made to the master copy between
the two rsync runs.

Thanks...Tom
-- 
Tom L. Schmidt, Manager/SysAdmin Characterization Equipment
Micron Technology, Inc.
8000 S. Federal Way  P.O. Box 6  Mail Stop 376  Boise, Idaho USA  83707-0006
mailto:tschmidt at micron.com  http://www.4schmidts.com/