rSync Issue with lots of small files

[Chris Dunlop]

G'day David,

David A. Soussan <das <at> dascomputerconsultants.com> writes:
> First posting, and I'm a n00b with rSync. I was having issues with rSync
> from a PC (3.0.7 either via DeltaCopy or cwRsync) to a Freenas box
> (3.0.6). I've since replicated and isolated the problem attempting to
> sync PC to PC with various flavors of rSync on both sides.
> 
> The executive summary is "daemon side processes are hanging up with 100%
> cpu utilization when sending many small files."
> 
> I can replicate this 100% of the time.
> 
> What I've done:
> 
> Created 2 directories with 20K files in each, 17 bytes long numbered
> 1.txt through 20000.txt. This is the source sync directory.

You may have resolved your problem by now, but for the record
and in case anyone else ends up here chasing the same issue...

The short answer is: use cwRsync 4.0.4 (or later).

The problem is, or was, with cygwin rather than rsync, and
specifically, in the communication between rsync and ssh.

I could duplicate this problem when using cwRsync 4.0.3 on the
sending side (using a single directory with 10,000 empty files).
I wasn't able to duplicate the problem when running direct to
a remote rsync daemon, i.e. not running via ssh.

I haven't been able to duplicate the problem with cwRsync 4.0.4,
which includes Cygwin 1.7.2-2, which in turn includes fixes
in the problem area:
 
  o Fix some hangs and a potential crash using pipes and FIFOs.
  o Fix multiple socket problems:
    o Potential blocking of non-blocking sockets.
    o Non-working renaming of AF_UNIX socket files.
    o Using MSG_PEEK on UDP sockets leads to data loss.

See: http://www.itefix.no/i2/node/12243

Hopefully that's the end of the reports of hangs when running
rsync under Windows!

Cheers,

Chris.