Mysterious (bogus?) rsync(d) errors -- (code 23) at main.c(1045)

[Josh Berkus]

Folks,

(1) Please pardon me if this is something stupid, but I've googled like mad 
and not figured it out (the other reports of this error don't seem to apply 
to me)
(2) I would be thrilled to hire an rsync expert to troubleshoot this for me, 
so offer if you are one.

The setup:
I have 2 identical RHAS3u4 servers, running linux 2.6.10.
On a nightly basis, we are rsyncing all of the database files on both servers 
which are in identical file locations.   The same users, with the same UIDs, 
exist on both servers
rsyncd.conf is further down this message

On trying sync'ing by both the "push" and "pull" methods (i.e. either by 
copying from the local directories to the other server's rscynd, or the other 
way), we are getting mysterious failures to rsync. 
The command line used is:
/usr/bin/rsync --perms --owner --group --times --recursive --safe-links 
--delete-after  server2::postgres-data /pgdata

Which transfers a whole bunch of data, and then gives these failures:
chown . : Operation not permitted
skipping non-regular file "pgdata/base/17521/pgsql_tmp"
skipping non-regular file "pgdata/pg_xlog"
chown . : Operation not permitted
rsync error: some files could not be transferred (code 23) at main.c(1045)

I really can't figure out what could be causing this; all of the files being 
copied are owned by the "postgres" user.  What is it trying to chown?   I've 
turned off all database processes to prevent the files from being modified 
during transfer.  And nothing is being logged to /var/log/messages for the 
error.  And it appears that all files actually transferred.  To make it even 
more mysterious, rsync works sometimes.

Any help is appreciated; we're already past deadline!

Rsyncd.conf 
--------------------
       uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid

       [postgres-data]
                uid = postgres
                gid = dba
               path = /pgdata
               comment = whole pgdata area
               read only = false
               write only = false

       [postgres-log]
                uid = postgres
                gid = dba
                path = /wal
                comment = Postgresql WAL
                read only = false
                write only = false


-- 
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco