[Samba] Apparent large memory leak with encryption + SMB3_00 or SMB3_02

[Jeremy Allison]

On Fri, Jun 19, 2020 at 04:07:03PM +0100, Andrew Nicholson via samba wrote:
> Hello,
> 
> I've recently set up Samba (4.12.3) on Arch Linux as the target for Time
> Machine backups for a couple of Macs. Shortly thereafter I started seeing
> OOMs whenever a backup would start. I stumbled upon disabling encryption on
> the server (i.e. changing "smb encrypt" from "required" to "off") to
> prevent this issue.
> 
> After further digging, I'm able to reproduce this issue using smbclient on
> the server machine with either SMB3_00 or SMB3_02 as the max protocol with
> the server configured to require encryption. Uploading a file increases the
> RSS of the smbd process by roughly the size of the uploaded file.
> 
> My minimal smb.conf and relevant smbstatus output are below. Have I missed
> something in the configuration? Or is this an issue with the AES-128-CCM
> encryption? I noticed that SMB3_10 or SMB3_11 do not suffer from the memory
> leak and use AES-128-GCM. Am happy to troubleshoot further as I would like
> to re-enable encryption if possible.

I believe this is already known. It's a bug in gnutls
which we started using for (most) of our encryption.

I believe it's already been fixed upstream, but the
folks involved should be able to comment more.