NTLM authentication with NTLM2 session security

[eglass1 at comcast.net]

Hello,

I work with Chris Hertel and Michael Allen on the jCIFS project.  As a sidebar
to this, I have been putting together documentation on NTLM authentication:

http://davenport.sourceforge.net/ntlm.html

I just figured out something that had been confounding me for awhile, and I
hadn't found this documented anywhere, so I figured I'd post it here
(as well as adding it to my own documentation).  For all I know this could
be well-known information that I just wasn't able to locate, so I apologize
in advance if this is the case.

When the client and server have agreed upon the "Negotiate NTLM2 Key"
NTLMSSP flag (0x00080000), but LMCompatibilityLevel is set to 0 (i.e., *not*
using NTLMv2 authentication), the LM and NTLM responses in the Type 3 messages
are changed as follows:

The LM Response contains an 8-byte client nonce, null-padded to 24 bytes.

The NTLM response is still calculated "normally" (i.e., using keys derived
from the NT hash to DES-encrypt a value three times); but instead of encrypting
the server challenge, it uses:

head(MD5(challenge + nonce), 8)

That is, it concatenates the server challenge with the 8-byte client nonce from
the LM response, takes the MD5 digest, and uses the first 8 bytes of the result
as the data in the encryption operation.

The intent of this is presumed to be prevention against server-based
precomputed dictionary attacks on the NTLM response (similar to the LMv2
response).


Eric