[RFC] Using system libraries for crypto in samba

[Simo]

Hello List,
during SambaXP we had some discussion about moving away from custom
crypto code and toward using system provided crypto libraries instead.

Aside from the burden of maintaining your own crypto one of the
advantages of system libraries is auditing (against things like side
channel attacks) and hardware acceleration (libraries like OpenSSL and
GNUTLS/Nettle have AES-NI support for example).

So I started looking in what it would take to provide a small shim layer
in samba to access either library so that the choice is a compile time
option.

The top commit in this [1] branch has a Work In Progress implementation
of such an interface (fully functional and with tests for OpenSSL).

I have looked at both OpenSSL and GNUTLS to devise an interface that
could abstract both, then actually implemented it for OpenSSL (which I
knew would be the most challenging due to the much less cleaner
interfaces) to see what could actually be done.

CCM support in OpenSSL has some annoying restrictions (for example it
can do only one-shot encryption/decryption, chunking is not supported),
and I am not sure what will be required in GNUTLS as apparently CCM
landed less than 6 months ago, and I do not have support for it even in
F22 yet.

Beyond the CCM oddities, one other thing that stands out is that current
samba code uses in place encryption while these libraries always assume
separate (but still statically-allocated buffers).
At least for GCM I do not think this would be a huge problem, but I'd
like your opinions before I put any other effort into this.

[1]
https://git.samba.org/?p=idra/samba.git;a=shortlog;h=refs/heads/crypto

HTH,
Simo.

P.S: note that I added CCM tests, as I couldn't find any vectors in the
current code. NIST provides a lot of vectors (both PASS and FAIL ones),
and I selected a few for CCM to test that CCM would work, given all the
quirks the OpenSSL library requires for this mode. Perhaps we should use
those for the current implementation too.
-- 
Simo Sorce