Samba - CPU and memory usage - Proposed solution(?)

[Nikos Balkanas]

Hi,

Please note that I am leaving samba general list. Too much traffic.

In my original thread, I indicated that this is a special option for big
professional systems. If something from the optimization spills to general
samba architecture, all the better.

In this case this is exactly what you want to avoid. We don't want *any* CRM
agent or user to be able to do a del *.*.  If all files in a directory need
to be deleted or moved, this should be done by the Unix administrator, who
has responsibility of the system and can easily do a "rm -rf *" on the Unix
filesystem.

The version I worked and developed the patch was 2.2.22 (I am fairly certain
the same holds for 2.2.24).

I am under the opinion that it is not samba's responsibility to verify a
missing file. This should be done from the Unix administrator on the Unix
filesystem. Having implemented it without a problem for ~1 yr, and having to
deal with ~10  missing files, I can verify that whenever samba didn't find
it, it wasn't on the Unix filesystem.

Please take heed from Windows XP. It serves 1 million files in a directory
without breaking a sweat. The only way to do it is with "stat". If it
doesn't find the particular file, it reports it missing, it doesn't hoag the
server.

Please also avoid bloating samba. One of the reasons that I didn't port the
patch to 2.2.24, was that, although only a security update, had so many new
buffers, that basically I had to redo everything from the start. Please make
an effort to reduce buffers, not increase them. Mike's idea for a minimal
samba server sounds very tempting, even in the general case.

Cheers,
Nikos


----- Original Message -----
From: "Andrew Tridgell" <tridge at osdl.org>
To: "Michael B Allen" <mba2000 at ioplex.com>
Cc: "Nikos Balkanas" <nbalk at hol.gr>; <samba at lists.samba.org>;
<samba-technical at lists.samba.org>
Sent: Monday, January 24, 2005 4:40 AM
Subject: Re: Samba - CPU and memory usage - Proposed solution(?)


> Mike,
>
>  > If the path requested does not exist the server MUST search the files
>  > within each directory and do a case insensitive comparison on each to
>  > prove that the file does not exist. However, if the file does exist,
>  > usually the case provided is correct. So as an optimization you can
just
>  > stat(2) for that component and only do the directory listing search if
>  > it fails. Additionally if you simply stat(2) the entire path requested
>  > at the very beginning and it exists you can skip the multiple stats and
>  > directory searches altogether.
>  >
>  > I'm not terribly familar with Samba but apparently the version you're
>  > using lacks some of these optimizations.
>
> I'm pretty sure Samba3 does have these optimisations (it did last time
> I looked), and certainly Samba4 does, but unfortunately there are
> cases where the client makes calls that don't allow this sort of
> optimisation.
>
> Perhaps the most common case is where a user on the client does "del *.*".
> In that case the client asks for a directory listing, then it asks to
> delete each file one at a time, but it provides the mangled (8.3) name
> for the file in the SMBunlink call rather than the full name.
>
> Samba tries to optimise this by keeping a reverse lookup cache from
> 8.3 name to long name (the so called "mangled cache"). As long as the
> cache doesn't fill, then this works well. If the cache does fill then
> things are not so nice. The code then falls back to a full directory
> search to do the mapping from 8.3 to full name. For the "del *.*" case
> this turns the whole operation into something that is O(n^2), which
> gets very very slow for large n.
>
> I am a proposed solution for this that I have on my todo list for
> Samba4. The solution is to make the cache size dynamic, but use a
> timer event to shrink the cache per connection when its not needed. So
> no matter how big the directory is Samba will always keep the whole
> directory in cache for a few seconds (probably 30 seconds or so), but
> will then shrink it after that time. It will also have to have a
> "number of directories in cache" limit to prevent DOS attacks on this
> mechanism.
>
> Cheers, Tridge