[Samba] Apparent large memory leak with encryption + SMB3_00 or SMB3_02

[Andrew Nicholson]

Hello,

I've recently set up Samba (4.12.3) on Arch Linux as the target for Time
Machine backups for a couple of Macs. Shortly thereafter I started seeing
OOMs whenever a backup would start. I stumbled upon disabling encryption on
the server (i.e. changing "smb encrypt" from "required" to "off") to
prevent this issue.

After further digging, I'm able to reproduce this issue using smbclient on
the server machine with either SMB3_00 or SMB3_02 as the max protocol with
the server configured to require encryption. Uploading a file increases the
RSS of the smbd process by roughly the size of the uploaded file.

My minimal smb.conf and relevant smbstatus output are below. Have I missed
something in the configuration? Or is this an issue with the AES-128-CCM
encryption? I noticed that SMB3_10 or SMB3_11 do not suffer from the memory
leak and use AES-128-GCM. Am happy to troubleshoot further as I would like
to re-enable encryption if possible.

Thank you,
Andrew

[global]
   smb encrypt = required

[data]
   path = /mnt/data
   writable = yes


Samba version 4.12.3
PID     Username     Group        Machine
     Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
837051  anichols     anichols     10.0.1.201 (ipv4:10.0.1.201:44812)
     SMB3_02           AES-128-CCM          AES-128-CMAC

Service      pid     Machine       Connected at
Encryption   Signing
---------------------------------------------------------------------------------------------
data         837051  10.0.1.201    Fri Jun 19 03:58:16 PM 2020 BST
AES-128-CCM  AES-128-CMAC

No locked files