rsync fails with "Permission denied" errors on random files over NFS

[Thomas Damgaard]

Hi

I use rsync for daily backups from a mailserver.

Every day the mailserver pushes backup to the backupserver.
Cron runs the command:
rsync -q -zrlpt --specials --chmod=Du+rwx \
--rsh 'ssh -F /root/ssh/config -i /root/ssh/private.key -o
UserKnownHostsFile=/root/ssh/known_hosts'  \
--files-from=/root/include \
--exclude-from=/root/exclude \
--link-dest=../previous \
/ backupuser at backupserver:backup/current

The rsync command is run as the root user on the mailserver.

/root/include contains the line:
/data

/root/exclude is empty at this moment.


The problem is that almost every day, I get an e-mail from cron saying
that rsync has failed.
It gives errors like these:
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240839853.V11I228039M884633.mail:2,RS"
failed: Permission denied (13)
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240830282.V11I228031M150189.mail:2,S"
failed: Permission denied (13)
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S"
failed: Permission denied (13)

I have checked that the files are ineed readable on the mailserver.
Here is ls -l from one of them:
-rw------- 1 mail mail 560383 2009-04-27 13:08
/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S


The files that gives the error is appearently changing randomly. One
day there can be one or two files with an error. Other times several
hundred.
I have seen it give errors on files in mail boxes that is currently
unused, so it cannot be because the file is currently used by some
other application (e.g. the imap server).

I noticed that it is readlink that gives the error. I'm not sure why
readlink is used in this case. This is a regular file and not a
symlink.


I hope you have some suggestions on how to debug this further.


Details on the servers's file systems:

Server backupserver:
/data is mounted like this:
nfs:/data/nfs/nfs0/data/xx/backup on /data type nfs
(rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)


Server mailserver:
/data is mounted like this:
nfs:/data/nfs/nfs0/data/xx/mail on /data type nfs
(rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)


Med venlig hilsen/Kind regards
Thomas Damgaard Nielsen
http://thomasdamgaard.dk