[Bug 13913] sync a folder with large amount of files

[samba-bugs at samba.org]

https://bugzilla.samba.org/show_bug.cgi?id=13913

--- Comment #3 from Dave Gordon <dg32768 at zoho.eu> ---
What filesystems are used at each end?
Also, when you say "hangs", how long have you left it in the "hung" state?

In preparation for trying to reproduce this issue, I created a new directory on
an ext4 filesystem and started to populate it, using the following bash
command:
$ for i in {01..323187} ; do echo $i > $i ; done
which would result in sequentially creating 323187 distinct files, each with
different contents.

I expected this to take a little while, but what I observed was that the first
214301 files appeared in the directory very quickly (<30 seconds, maybe only 10
or 15 seconds), but there was then a long pause during which no new files
appeared. Thus, the file creation process seemed to have stalled at this point.
However, after a long pause (>2 minutes) the rest (108886) of the files
suddenly appeared in the directory, in an interval of probably no more 10
seconds. Hence the average rate of file creation over a series of ten-second
intervals appeared to vary from ~10000/sec down to almost zero.

So I suspect that the ext4 filesystem at least may have some limit on how many
blocks/inodes/directory entries can be queued for writeback to disk at a time,
or that while the updated directory is being written back it may be locked
against further updates, possibly for quite a long time.

Thus, the "hang" may be in the underlying filesystem on the receiver, rather
than in rsync itself. There may however be differences in the behaviour of
rsync (e.g. order and timing of file operations) across different versions that
either trigger or avoid triggering the particular case in the filesystem code.

Can you run the receiving program in the hanging case under truss/strace or
similar, and thus collect information on the timing of the file create calls
over the duration of the program run?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.