[Samba] Possible memory leak in
slow at samba.org
Tue Sep 14 20:04:16 UTC 2021
Am 14.09.21 um 20:42 schrieb Jeremy Allison:
> On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba wrote:
>> Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba:
>>> What I have noticed from looking at the Processes tab in System
>>> Monitor on the server: The server is on a 10GB network. When I start
>>> a chia plot (~106Gb file) transfer from a client computer that is
>>> also connected to the 10Gb network the transfer starts off at
>>> ~1000Mb/s until the samba process is larger
>>> 32.5GiB. The largest I have seen is 39GiB. The transfer rate then
>>> drops down to the hdd write speed of ~170Mb/s. That samba process
>> gets any
>>> smaller than 32.5GiB. Even after the transfer is complete.
>>> If a client computer that is connected with a 1Gb nic starts a
>>> transfer, another samba process is spawned. That transfer stays
>>> around 130Mb/s
>> to 150
>>> Mb/s and the samba process grows to ~9GiB. After that transfer is
>>> complete the process drops down to ~5MiB.
>>> After a day of plotting and transferring ~100 plot files to the
>>> those two computers there are still only the two samba processes
>>> running. Plotting has been stopped for several hours and one is
>>> 32.5Gib, the other is 4.9Mib.
>> the pool-usage of the large process you'd sent me privately contains:
>> $ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l
>> which means there are 6468 SMB2 WRITE requests pending at the server.
>> This is likely triggered because
>> - the client is sending data much faster then the server is writing data
>> to disk
>> - Samba returns SMB2 Async Interim Response for every received WRITE
>> - These grant SMB2 credits to the client
>> - As a result the client is not throttled by a crediting window
>> Iirc we've seen this before some time ago and iirc a Windows server
>> behaves either doesn't send interim async responses or stops sending
>> them at some point.
>> To me this looks like a design flaw in the crediting logic, but we can't
>> change that so we have to adjust our write processing code to handle
>> Until that fix materializes (which may take some tim) you could disable
>> aio as a workaround in smb.conf by setting:
>> aio read size = 0
>> aio write size = 0
> Hmmm. Can't we create some back-pressure on the crediting algorithm
> by forcing writes to go synchronous after the queue async io events
> reaches a (parameterized) size ?
> We already count fsp->num_aio_requests, so we have a per-fd
> count. We could add a per-process count ? Or maybe start
> forcing reads/writes to go synchronous once the time between
> request/response goes over a certain amount ?
first we have to check how Windows handles this. Iirc they don't send
interim async responses. Problem solved. But my memory may not serve me
Ralph Boehme, Samba Team https://samba.org/
SerNet Samba Team Lead https://sernet.de/en/team-samba
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 840 bytes
Desc: OpenPGP digital signature
More information about the samba