[Samba] authentication performance with 4.7.6 -> 4.7.8 upgrade (was: Re: gencache.tdb size and cache flush)

[Peter Eriksson]

I’m going to try to upgrade from 4.7.6 to 4.7.7 on one of our servers soon and see if things break or not. With 4.7.6 things are stable at least.

Our file servers are in a Microsoft Windows domain (consisting of 6 Microsoft Windows 2016 AD servers).


The graphs (logarithmic time scale) below are login times from a probe station that times a connect using “smbclient” with a Kerberos ticket and basically just quits directly. It shows five of our Samba servers in one graph (so not so easy to read :-)

Samba 4.7.6 (2018-09-04, right now, around 400-500 users per server):



Samba 4.7.8 (2018-08-29):



The probe software has a 10 seconds timeout so the “spikes” are probably/basically connection attempts that timed out. We probe all servers every minute. The holes in the graphs are 10 hours apart from last reboot (07:00 every day) -17:00, 03:00) and then nobody could connect basically (or the attempts took more than a minute so the whole session was aborted - and thus no data recorded in the RRD databases).

- Peter


> On 4 Sep 2018, at 04:15, Andrew Bartlett via samba <samba at lists.samba.org> wrote:
> 
> On Wed, 2018-08-29 at 15:36 +0200, Peter Eriksson via samba wrote:
>> For what it’s worth you are not alone in seeing similar problems with Samba and gencache. 
>> 
>> Our site has some 110K users (university with staff & students (including former ones), and currently around 2000 active (SMB) clients connecting to 5 different Samba servers (around 400-500 clients per server). When we previously just let things “run” gencache.tdb would grow forever and authentication login performance would start to deteriorate after a little while (would take more than 10 seconds). So we now delete it (and locks/locking.tdb that also tends to grow forever) and restart our samba processes every morning at 7 am - which gives us much more stable performance.
>> 
>> - Servers with 256GB of RAM, 10Gbps ethernet interfaces and around 110TB of disk per server.
>> - FreeBSD 11.2-p2
>> - Samba 4.7.6 with some local patches to allow (much) bigger socket listening queues in order to handle the case of many clients connecting at the same time.
>> 
>> (We are trying to upgrade to a more recent Samba but 4.7.8 and 4.7.9 gave us horrible authentication performance every 10:th hour where the servers basically denied clients to login for about 2 hours so we had to back down to 4.7.6 again).
> 
> I realise testing in production is difficult, but is there any chance
> you can pin down where between 4.7.6 and 4.7.8 it broke?  There are not
> that many changes between, and while some appear authentication related
> nothing stands out. 
> 
> Also, do you run Samba as an AD DC, or are these file servers in a
> windows domain?
> 
> Thanks,
> 
> Andrew Bartlett
> 
> -- 
> Andrew Bartlett
> https://samba.org/~abartlet/
> Authentication Developer, Samba Team         https://samba.org
> Samba Development and Support, Catalyst IT   
> https://catalyst.net.nz/services/samba
> 
> 
> 
> 
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba