[jcifs] Issues with lmCompatibility and NTLMfilter

[Eric Glass]

>  Now these flags are defined in NtlmFlags.java, but are never used
> throughout jcifs.  In fact, the flags segment is constructed from
> Type2Message.getDefaultFlags() which knows about 4 or 5 flags altogether.
> My thinking is that with lmcompatibilitylevel=4, IE wants to do NTLMv2,  but
> it does not see any of the appropriate flags returned in the type2 message,
> and quits.  Will returning these flags help IE to complete the negotiation
> and send the type3 authorization?
>

There is nothing in the NTLM handshake that indicates whether the
server accepts/supports NTLMv2 authentication (NTLM2 key indicates if
NTLM2-style signing/sealing is supported, but that is independent of
LMv2/NTLMv2 authentication).  If there is an incompatibility on that
level, you would typically see a Type 3 response and then a failure on
the backend.

Look under the following key in the registry (on the client desktops):

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\MSV1_0

There should be two values under there, NtlmMinClientSec and
NtlmMinServerSec; what are the values of these in your environment?
If the client is not responding to the challenge, that typically
implies that it is not satisfied with the NTLMSSP context that has
been established; this registry setting specifies the set of
"mandatory" flags that the host will demand when acting as a client
and a server, respectively.  If you specify 0x00000000 for both, it
should effectively be "wide open"; that may get you a bit further.

With LMCompatibilityLevel set to 4, however, you will also need to (in
the filter configuration) set "jcifs.smb.lmCompatibility" = 4.  This
will allow jCIFS to appropriately handle the NTLMv2/LMv2 Type 3
response from the client (once it starts receiving them).  At that
point you can start troubleshooting compatibility between jCIFS and
the backend domain controller.


Eric