too many connections on one module?

kimor79 at sonic.net kimor79 at sonic.net
Thu Apr 27 19:13:02 GMT 2006


I have ran into an odd problem with rsync and modules. Here are the details:

There are about 2500 clients (of various rsync and OS versions beyond my
control) syncing every hour to one server running rsync as a daemon.

There are 4 modules. 3 are different branches of the same code. These 3
modules each contain about 50k files. Most of the files are <1k or even
empty, with a few files being above 10M and the total size being about
100M. The other module contains about 180 files, 30M total size.

About 1500 clients rsync from the 4th module and have no problem. About 50
clients rsync from the 2nd and 3rd module and also have no problem.

The remaining 1000 rsync from the first module doing two passes. The first
pass is the top level dir excluding the dir containing about 45k of the
files. This is relatively fast and no problems are seen.

The second pass is the problem. On this pass the clients are syncing about
10 files from the previously excluded dir only. For some reason rsync
sessions take an excessively long time to complete (hours as opposed to
minutes). Eventually the connections pile up and load on the server has
gotten as high as 900!

The clients that rsync to the 2nd and 3rd module are doing the same thing
except they don't take a long time (even when the load is high). They take
a few minutes at most on the second pass.

Have I reached some per module limit?

Thanks
Kimo



More information about the rsync mailing list