[Samba] Possible memory leak in

[Ralph Boehme]

Am 14.09.21 um 20:42 schrieb Jeremy Allison:
> On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba wrote:
>>
>> Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba:
>>> What I have noticed from looking at the Processes tab in System
>>> Monitor on the server: The server is on a 10GB network.  When I start
>>> a chia plot (~106Gb file) transfer from a client computer that is
>>> also connected to the 10Gb network the transfer starts off at
>>> ~1000Mb/s until the samba process is larger
>> than
>>> 32.5GiB. The largest I have seen is 39GiB.   The transfer rate then
>>> drops down to the hdd write speed of ~170Mb/s.  That samba process
>>> never
>> gets any
>>> smaller than 32.5GiB.  Even after the transfer is complete.
>>>
>>> If a client computer that is connected with a 1Gb nic starts a
>>> transfer, another samba process is spawned. That transfer stays
>>> around 130Mb/s
>> to 150
>>> Mb/s and the samba process grows to ~9GiB. After that transfer is
>>> complete the process drops down to ~5MiB.
>>>
>>> After a day of plotting and transferring ~100 plot files to the
>>> server
>> from
>>> those two computers there are still only the two samba processes
>>> running. Plotting has been stopped for several hours  and one is
>>> 32.5Gib, the other is 4.9Mib.
>>
>> the pool-usage of the large process you'd sent me privately contains:
>>
>> $ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l
>> 6468
>>
>> which means there are 6468 SMB2 WRITE requests pending at the server.
>>
>> This is likely triggered because
>>
>> - the client is sending data much faster then the server is writing data
>> to disk
>>
>> - Samba returns SMB2 Async Interim Response for every received WRITE 
>> request
>>
>> - These grant SMB2 credits to the client
>>
>> - As a result the client is not throttled by a crediting window
>>
>> Iirc we've seen this before some time ago and iirc a Windows server
>> behaves either doesn't send interim async responses or stops sending
>> them at some point.
>>
>> To me this looks like a design flaw in the crediting logic, but we can't
>> change that so we have to adjust our write processing code to handle 
>> this.
>>
>> Until that fix materializes (which may take some tim) you could disable
>> aio as a workaround in smb.conf by setting:
>>
>> aio read size = 0
>> aio write size = 0
> 
> Hmmm. Can't we create some back-pressure on the crediting algorithm
> by forcing writes to go synchronous after the queue async io events
> reaches a (parameterized) size ?
> 
> We already count fsp->num_aio_requests, so we have a per-fd
> count. We could add a per-process count ? Or maybe start
> forcing reads/writes to go synchronous once the time between
> request/response goes over a certain amount ?

first we have to check how Windows handles this. Iirc they don't send 
interim async responses. Problem solved. But my memory may not serve me 
right.

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba/attachments/20210914/c4c0b9c7/OpenPGP_signature.sig>