Rsync major slowdown problem

[Larry Gingras]

I'm getting very strange issue with rsync on one of my servers. I spent
many days trying to figure it out myself, playing with options, settings
etc but had no luck and now only thing is left for me to write to this
mailing list.
I'm using rsync to backup data from one server to another. And I see
data is transferred extremely slowly. It seems to halt for a long period
of time, after send some data in small burst. But the most strange is
that both send and receive socket buffers on both ends are full all the
time, though all rsync instances are permanently in "select" system call
and consume 0% CPU time.
No playing with options solves problem. If I set big tcp buffers (about
512KB) it runs good for a litttle while but then same slowdown with full
buffers.
Now some details that might be usefull:
I'm using latest (3.0.6) rsyns on both ends. Before I have old rsync
(2.8.X something like that) on receiving server and there was different
problem: data copied with good speed but after some hours of copying it
dropes with "broken pipe" error on both ends. The most strange that both
of these issues (old and new one) affects only one server, but I have
few other servers copying data to this one backup server and no any
problems with those. OS same, settings (sysctls) same. However affected
server run many other software and no any problems was seen with those.
I have FreeBSD 7.2 (amd64) on all of the source servers and FreeBSD 6.2
(i386) on destination (backup) server.

I tried to debug receiving side and I see that it actually reads data
from socket but I still wonder how can buffer be full all the time if
all the time process in select state and awaiting data from that socket?!