Question(s) about smbd in respect to preadv2

[Milosz Tanski]

That's a very helpful response Michael,

On Sat, Jan 24, 2015 at 4:12 PM, Michael Adam <obnox at samba.org> wrote:
> Hi Milosz,
>
> On 2015-01-24 at 15:46 -0500, Milosz Tanski wrote:
>> I'm at a bit of a cross roads with testing preadv2 using samba and how
>> samba could benefit from the syscall. I need to better understand the
>> samba architecture today and where it's going. I spent a few hours
>> yesterday and today better to understand samba; I also looked at
>> Jermey's SambaXP 2014 talk. There's a few clarifications I need.
>>
>> I did some preliminary testing using preadv2 to perform. The first
>> tests I ran were using the source3 smbd server. And I compared the
>> sync, pthreadpool and pthreadpool + preadv2. Just a simple rand read
>> test with one client with a cached data.
>>
>> The results were that sync was fastest (no surprise there).
>> Pthreadpool + preadv2 was about 8% slower then sync. Plain old
>> pthreadpool 26% slower. So not a bad win there. Additionally, it looks
>> like the vfs_pread_send and vfs_pread_recv have a bit more overhead
>> over plain old vfs_pread in the code path so it's possible to get that
>> 8% even closer to the sync case.
>
> That is definitely very intresting.

You can find more details of the test and changes in this thread:
https://lkml.org/lkml/2015/1/21/407

>
>> So far, so good... but what struct me is that I don't really
>> understand why samba uses the pthreadpool if forks() for each client
>> connection? Why bother.
>
> Right, currently each tcp connection corresponds to an
> smbd child proces, but there may be several clients behind
> one tcp connection (in the case of a windows terminal
> server) or a multi-threaded client. Therefore it is
> important not to block on single syscalls. Hence the
> pthreadpool which encapsulates syscalls in threads.
>
>> From looking at the code in for source3, source4 and internet readings
>> (like Jeremy's talk). It looks like there's an ambition to not have a
>> process per connection model in the future. Am I correct in that
>> assumption?
>
> Vaguely, yes. One example is the work in process
> to implement SMB3 multi-channel. This is solved
> in our current apporoach by fd-passing tcp-connections
> to one smbd process that already serves the same
> SMB3 CLient GUID. Hence we have a per-client-GUID
> single-process model, which makes the fanning-out
> by use of threads even more important.

Originally I built preadv2 for lower webapps that essentially are
single threaded apps (for networking) with a threadpool for diskio.
The synchronization / communication of the threadpool introduces extra
latencies into requests (esp. smaller sized requests) in the full
cached case. And in the case of mixed use of cached and uncached
accesses it gets worse because fast request that shouldn't have to
wait get stuck in the queue after slow / large requests.

I ended up lucking into using samba as a test case for the kernel
folks they don't fully understand that use case and more familiar with
use cases of an app like samba.

>
> See e.g.
> http://blog.obnox.de/demo-of-smb3-multi-channel-with-samba/
> for a demo and some explanation what's going on.
> This work will hopefully be completed and integrated
> into upstream very soon!

I actually ran into the video while doing some googling about the
architecture of samba.

>
>> Is the hope to have one process with a main (or few) thread handling
>> the most of the incoming network IO and then a largish threadpool for
>> performing most of the accentual disk IO work?
>>
>> That's the kind of use case that preadv2 was conceived for. Otherwise
>> I'm not sure if it's the right fit and I'm not quite sure what's the
>> point of the threadpool is.
>
> That definitely sounds very interesting for this future
> mode of operation! We may also work towards a general
> single-process model. The per client-GUID approach is
> going half-way there.

The per client guid mode sounds like it gets you 90% of the way there.

>
> Cheers - Michael
>

Thanks again.


-- 
Milosz Tanski
CTO
16 East 34th Street, 15th floor
New York, NY 10016

p: 646-253-9055
e: milosz at adfin.com