rsync randomly and silently skips files

[Matt Garman]

Hi,

I'm using rsync to sync a custom production environment (binaries,
scripts, config files) out to remote sites.  The exact same command
runs twice daily from cron.  The command is basically:

    rsync -av --exclude somepath /prod_environ/ remotesite:/prod_environ/

Nothing too fancy IMO.  And until recently, it has worked exactly as
expected.

But last night, and a few days ago, at least one changed file was
silently skipped/missed.  I log the rsync job, and rsync did not
generate any errors, and exited back to the shell with code 0.

The missing files were just text (one was a config file and one was
a Perl script).

When I noticed the missing files, I re-ran the exact same command,
and the files were this time copied!

Because this has only happened twice (that I am aware of) in years
of use, I don't know how to replicate it.  But the files are
definitely being missed.

The only thing that's not "vanilla" about the setup is that the
source files reside on an NFS share.

An example of a file that didn't get updated last night: source file
(i.e. new, updated file) is 3413 bytes, timestamp Apr 16 09:15.
Destination (i.e. old, outdated file) is 3405 bytes, timestamp Apr 3
15:52.  These are the timestamps as reported by "ls".

$ rsync --version
rsync  version 2.6.3  protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, 
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Both source and destination sites are Linux kernel 2.6.x, CentOS
4.x.

I know this is a fairly old version of rsync, but I searched
bugzilla and the list archives, and couldn't find anything relevant.

Anyone have any thoughts or ideas?

Thanks,
Matt