[linux-cifs-client] Slow Folder Traversal on One system, Fast on Another

[Jeff Layton]

On Tue, 19 Jan 2010 17:58:19 -0800
Tanthrix <tanthrix at gmail.com> wrote:

> Jeff Layton wrote:
> > The two kernels are using different infolevels for the FIND_FIRST call
> > (not too surprising since this code has changed significantly in 10
> > releases).
> >
> > The main difference though is that the later kernel is doing a
> > QPathInfo against all of the inodes, and the earlier one isn't. The
> > readdir code doesn't trigger this, so I suspect that the time
> > difference is due to differences in userspace. i.e., the later release
> > appears to be doing more than just readdir() type calls when you call
> > 'ls -R'.
> >
> > One question...when you do the "ls" on ubuntu, is the output colorized?
> > If so, can you run:
> >
> >     $ unalias ls
> >
> > ...and then in the same shell, redo your 'ls -R' test on the ubuntu
> > box. Does it go faster?
> 
> Jeff,
> Indeed it does! In fact, it is now identically fast to my older system, 
> which strangely enough also gives colorized output from the ls -R 
> command without using QPathInfo. In any event, great detective work!
> 
> Now, the dillemma: As you may imagine, my original problem was not the 
> slow down when doing ls -R, but instead, a slowdown with an application 
> I use. More specifically, MythTV, and its plugin MythVideo which for my 
> use is basically a glorified file browser with an OSD that allows me to 
> play back media files on the TVs in the house.
> 
> Each time I open this program, something I do frequently, it reads the 
> full list of files on my CIFS mounted share to show me all my home 
> videos, pictures, movies, etc.. And as you have already guessed, it too 
> is causing a QPathInfo for each file, causing serious slowdown compared 
> to my older systems.
> 
> So I guess my question is this: Is there anything I can do from a CIFS 
> client perspective to prevent this problem? Or am I completely at the 
> mercy of the code in this application? And if that is the case, would it 
> be possible to use the earlier version of the CIFS client on my current 
> system, or is that something built into the kernel that would be 
> non-trivial/impossible to swap out?
> 
> Thanks for all the help, it's much appreciated!

It seems strange (and is probably a bug) that the older system doesn't
cause all of those QPathInfo calls. I suppose the ls on that system
might be able to use the d_type field in the dirent to determine a type
and decides that it doesn't need to stat() each entry. Or it could be a
bug and that older kernel didn't do an on the wire call for each
stat(). You might consider stracing both colorized 'ls' commands and
see what they're actually doing at a system call level.

As far as preventing the "problem", it's hard for me to say since it
all sort of depends on what the program is doing at the system call
level. I suspect that it too is probably doing a stat() on each
directory entry. In that case, no there's little you can do to help
this. A stat() system call currently means at least one call out on
the wire per file.

One thing you can try is turning off querying for server inode numbers
(mount with -o noserverino). It'll mean that you can't properly detect
hardlinks but it'll prevent one on-the-wire call per stat() syscall in
your case.

-- 
Jeff Layton <jlayton at redhat.com>