Filesystem question

[James Sutherland]

On Fri, 13 Oct 2000, David Collier-Brown wrote:

> On Fri, 13 Oct 2000, David Collier-Brown wrote:
> > > For big directories, yes. A mere 10,000 files/directory will drag
> > > Solaris and Linux down significantly.
> 
> >James Sutherland wrote:
> > You mean the linear search through a directory will slow Solaris
> > down? If that's the problem, I don't think a Samba-side solution is the
> > right one. Have you tried experimenting with Solaris directory caching??
> 
> My caches are auto-growing properly, so they're ok. What I did try was
> logging (described earlier) which radically improved metadata update
> operations.

The question is, though, how well does access time on cache hits scale
with directory size?

> > In which case, why try to fix this issue in Samba? It's clearly a Solaris
> > issue, IMO.
> 
> As I said before, Solaris is slow because it has to be general.  
> Samba **can** be faster, because it can use the best algorithm for the
> specific operations it does.  That doesn't mean it will be: this will
> take some careful analysis and measurement of the directory operations
> in NetBench.

Hrmm... I don't entirely buy that. Looking up a filename in a directory
is pretty specific: does Solaris do it well or not?

> > I'd try adding the cache Solaris-side first: this should provide the same
> > benefits, but avoid the issues raised earlier and benefit other
> > applications too.
> 
> That's been done: we just saw a 30-odd percent improvement in
> directory speed going to the next Solaris 8 quarterly CD. This will be
> continued by the filesystem group(s), driven by reports from the
> customers.

Interesting... what aspect of directory access?

> In the case of directory operations, this was driven by customers
> using PC NetLink and Samba, who detected non-linear performance
> degradation on overloaded disks.  This produced poor results from
> smbtorture and NetBench (for all UFSs, actually). This will be of
> interest to the Linux ext2 folks, too, although they have quite a
> different algorithm...

Right now, ext2 has a very poor structure for directory lookups; ReiserFS
and one of the modified ext2 variants improves on this dramatically. How
about your UFS?? (I would just look at the source, but at $75, you're
being undercut by Win2k :-)


James.