Possible Bug or limitation in Cygwin 1.7 and Rsync and file number limit

lists at trcintl.com lists at trcintl.com
Thu Jun 18 16:17:33 GMT 2009


I'm not sure if this is a bug or a limitation that can be worked around with a setting somewhere, but I have found a problem with cygwin 1.7 while using rsync.  I have been using rsync and cygwin v1.5 for quite some time.  I recently started testing cygwin v1.7 and I ran into a problem with an apparent limit on the number of files cygwin 1.7 along with rsync can process in a directory.  First let me say that I don't store this many files in a directory manually but rather we have a program that does.

I have not been able to identify the exact number of files, but it seems to be somewhere over 6,000 in a single directory.  To help sort this out, I whipped up the following perl code that creates 9,000 dummy files and puts them in a c:\temp directory.  Code is as follows if anyone would like to try and reproduce this:

my $path="c:/temp/";
my $number_of_files=9000;
my $base_file_name="file";
my $count=1;
my $file_name = $path . $base_file_name . $count;
while ($count <= $number_of_files) {
  open FILE, ">$file_name" or die $!;
  $count++;
  $file_name = $path . $base_file_name . $count;
  print FILE "1";
  close(FILE);
}

For those familiar with rsync, here are the options I am using in rsync and by the way, I am currently reproducing this by copying from and to the same workstation, however this same issue has been reproduced on more than one machine.  This doesn't appear to be an rsync problem but just the same, I have tried several different options for rsync and none seem to make a difference.

--recursive
--human-readable
--delete
--times
--ignore-errors
--stats
--progress
-v -v

If I then run rsync 3.0.5 using cygwin 1.7, the processor will spike to 100% utilization and it will process several thousand files before it just locks up with the processor still spiked at 100%.  No errors are shown, it just hangs.  If I then delete the directory I am using rsync to copy to and repeat the same procedure with the same version of rsync only this time using cygwin 1.5.25-15, the processor will again spike to 100% however the copy will finish as expected and everything returns to normal.  Again, I have been able to repeat this on more than one computer.

One more interesting thing.  If I first using rsync and cygwin v1.5 to copy the files and THEN come back and use rsync and cygwin 1.7, it make it all the way through.  

Is this a bug in cygwin 1.7 or is there some setting that will allow me to work around this limitation?

Thank you.


- Kyle



More information about the rsync mailing list