A mechanism to throttle credits granted when a Samba server is under load

[Richard Sharpe]

On Mon, Feb 11, 2019 at 12:32 PM Volker Lendecke
<Volker.Lendecke at sernet.de> wrote:
>
> On Mon, Feb 11, 2019 at 12:01:15PM -0800, Richard Sharpe via samba-technical wrote:
> > To that end, I suggest we add a new VFS call (heh) called something
> > like SMB_VFS_GATE_CREDITS_GRANTED that would be added in
> > smb2_set_operation_credit between these lines of code:
> >
> >         credits_granted = MIN(credits_granted, credits_possible);
> >
> > +      gated_credits = SMB_VFS_GATE_CREDITS_GRANTED(credits_granted, ...)
> > +      credits_granted = MIN(credits_granted, gated_credits);
> >
> >         SSVAL(outhdr, SMB2_HDR_CREDIT, credits_granted);
> >
> > The VFS function might monitor things like /proc/sys/fs/aio_nr for
> > some max allowable number or might also look at CPU load, or might,
> > with enough info (like credit_charge, etc) also track the number of
> > credits granted to this smbd over time, but that is all for future
> > thought.
> >
> > Any thoughts?
>
> Is the VFS really the right place for this? What we need is a global
> crediting system. The difficulty here is a mechanism to grant the
> credits across all smbds. Every smbd has to acquire some credits from
> a common pool. If more smbds come in, the common pool will have less
> available per smbd. Then you can externally manage the pool based on
> load. Taking credits from a central pool is just a small tdb
> operation, the problem though is crash recovery. How do you get back
> the credits that an smbd has retrieved and not given back because it
> did not shut down properly?

Yes, I agree with these and other criticisms. I did wonder at whether
it should be in the VFS, but today that is only place easily
accessible for OEMs etc ...

There is an additional issue that we do not want to run the client
completely out of credits because I have seen clients just drop
connections when they run out of credits under some circumstances.

The more I think about the original problem encountered the more I
think it is not strictly a crediting issue in the following sense.

I think what has happened is that the server has been driven beyond
the knee in the latency curve, but because we send PENDING responses
with extra credits as soon as a READ or WRITE request as been placed
on the AIO queue, we allow the client to send even more requests and
drive us deeper past the knee. I have seen SETATTR requests taking 2
minutes when that server is in this situation.

I suspect the simple solution in the case I am dealing with may be to
turn off AIO and let the requests become synchronous. Sure, writes
will go into the page cache until that fills up, but after that it
tends to become self throttling. And reads will be fine from the
start.

--
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)(传说杜康是酒的发明者)