Samba Speed Question - Please Help!
Justen Marshall
justen at al.com.au
Thu Oct 5 00:21:49 GMT 2000
Hi
Thanks for all your replies! They have been very interesting and
informative.
I'll clarify a few things regarding my situation.
- Yes we are using Irix, version 6.5.8 (the latest) to be specific,
and the drive is definately XFS.
- I've heard that the Ciprico disk volume we are using is factory
optimised for small numbers of large files (eg streaming video)
rather than the reverse. But I don't really think that's a concern
here. Copying small or large files around on the local machine
gives good speed (in line with our expectations for the drive) but
without the huge overhead in CPU usage (virtually no CPU is used
at all).
- Thanks Mike for the suggestion about ReiserFS, I'll investigate
that. My reason for using an SGI was primarily that we had one
available :) If something else suits the job better, I'm only too
happy to go that way.
- My "10,000 small files" example was a worst-case scenario. In my
test they WERE all in one directory, however we rarely go above
2,500 files in any one directory in real-world usage. I did the
10,000 file copy two ways, once with a dos command like "copy *",
and another time with a (CygWin) 'find -exec copy' command. The
find command was MUCH slower, primarily due to spawning a shell to
do each copy individually. The "copy *" was the fastest, and
surprisingly (to me) had very little startup time required.
- I made the mistake of using the CygWin "cp" command the first
time... being used to typing Unix commands as I am... THAT baby
never even copied one file, it spent so long resolving the
wildcard! (And it put an smbd at 90-100% CPU for at least a few
minutes till I killed it).
- Robert D, I really like the ram-disk-lock-directory suggestion and
I will implement that regardless of what else I do.
- Rob T, I agree with you that poor performance on Unix wrt lots of
files in one dir is due to searching the directory space. That is
what I was referring to in my original post about the ngetdents()
funciton call being over-used... it's the one that has to do that
work, and it is called SO MANY TIMES that I am not surprised that
Samba is using up a huge amount of CPU time. Caching the result of
that function would be great, and Samba seems to do that to a
point - if you stat a file, then immediately stat it again, you
get a cached copy of the file info. However, if you stat another
file in the meantime, then come back and stat the first file, it
reads the whole directory again... dang. My greatest hope was that
there would be a caching mechanism for that info that was disabled
by default, which I could enable and thus drastically reduce our
CPU use!
I hope that gives a better idea of what I was trying to do!
If there are any Samba developers who would like to have a chat with
me regarding this issue, I would be very happy to talk with you (and I
can certainly arrange for a pizza voucher or two :)
In the mean time, I have lots of things to try. Thanks again for all
the help!
Justen
--
.-----------------------.------------.--------------------------.
| Justen Marshall | | e-mail: justen at al.com.au |
| Technical Director | | phone: +61-2-9383-4831 |
| Animal Logic Pty Ltd | | fax: +61-2-9383-4801 |
|-----------------------^------------^--------------------------|
| Athiesm is a non-prophet organization. |
`---------------------------------------------------------------'
More information about the samba
mailing list