[cifs-protocol] RE: Regarding String2key for random trust keys ([MS-ADTS] 7.1.6.8.1.2)

[John Dunning]

Hello Andrew,
    In your email you stated:

"The problem is that if Microsoft treats this buffer as 'UCS2' when converting to UTF8 (for kerberos keys), then any random buffer is valid input.  However, where that buffer to contain UTF16 sequences (as typed by a user as a password), it would not map to the correct UTF8 sequence for conversion into the kerberos key.

Similarly, if the buffer is UTF16, then what should we do when windows machines set a random stream of bytes as the password?

This is a critical interopability issue, as AES keys become more used.

Similarly, this affects the creation of ARCFOUR keys from UTF8 input on non-windows machines (should the characters be expanded to UCS2 or UTF16 prior to the MD4?)"

Based on this information along with the data that you sent we have an interim response for you. We are also looking into how to best reflect this information in our documentation. Please let me know if the below information is sufficient to get you unblocked.

ASSUMPTIONS:

1) The password string generator will produce a set of random USHORT values between 0x0001 and 0xFFFF, with 0x0000 being converted to 0x0001 there is a slight alteration to the amount of randomness in the result, though I doubt it is significant.

2) WideCharToMultiByte is used for the conversion with CP_UTF8 for the code page and without WC_ERR_INVALID_CHARS passed as a flag.

FACTS:

1) WideCharToMultiByte will convert any instance of 0xD800 - 0xDBFF (high surrogate) without an immediately following 0xDC00 - 0x0xDFFF (low surrogate) to U+FFFD (OBJECT REPLACEMENT CHARACTER).

2) It will so the same with any low surrogate that was not preceded by a high surrogate.

3) It will NOT care about noncharacters or other reserved characters in Unicode, they will be converted as is.

4) The "string" (munged as in #1 and #2) will be algorithmically converted to UTF-8 using the definition of UTF-8 at unicode.org. No separate protocol document was generated as the Unicode definition was satisfactory.

Since that the byte buffer provided in the mail contains several unpaired low surrogates and I think one unpaired high surrogate in it as well, the resultant string after conversion will have had some USHORT values converted to 0xFFFD prior to be converted to UTF-8.

Thanks
John Dunning
Senior Escalation Engineer Microsoft Corporation
Email: johndun at microsoft.com
Tele: (469)775-7008

-----Original Message-----
From: Andrew Bartlett [mailto:abartlet at samba.org]
Sent: Tuesday, September 23, 2008 1:45 PM
To: John Dunning; Darryl Welch
Cc: cifs-protocol at samba.org; pfif at tridgell.net
Subject: RE: Regarding String2key for random trust keys ([MS-ADTS] 7.1.6.8.1.2)

On Tue, 2008-08-19 at 22:30 +1000, Andrew Bartlett wrote:
> On Tue, 2008-08-19 at 14:48 +1000, Andrew Bartlett wrote:
> > On Fri, 2008-08-15 at 09:32 -0700, John Dunning wrote:
> > > Hello Andrew,
> > >     I wanted to ask you if you have taken a look at Section 3 of
> > > RFC
> > > 3629 which may be of help for this problem.
> >
> > Is that the expected target string format for string2key operations?
> >
> > > If you have and it didn't help then we need to get more
> > > information on how you are actually doing the conversion. For
> > > example are you using your own function or a canned one?
> >
> > We use our own implementation of iconv() for the UTF16 -> UTF8
> > translations.
> >
> > http://gitweb.samba.org/?p=samba.git;a=blob;f=source/lib/charset/ico
> > nv.c;h=4f4bc8fd2da70c9f9d5bb75b2ee0f946516c996a;hb=v4-0-test#l589
> >
> > It (rightly) rejects the random data as not being valid UTF16 input.
> >
> > As far as I can tell, it is not possible for random bytes to simply
> > be described as UTF16 (and then converted to another charset), so I
> > suspect we will need a filter or modified function.
>
> Talking with tridge about this problem, perhaps the problem is that
> these buffers are not really 'Unicode' (by the convention of this
> document, ie UTF-16).  If the buffers were instead UCS2 and rules
> about illegal and reserved ranges were ignored, then the standard UTF8
> Huffman encoding were applied, would this result in the same UTF8
> string as Micorsoft uses for it's input into the AES and DES string2key functions?
>
> For reference, MS-ADTS 7.1.6.8.1.1 describes it this way:
>
> TRUST_AUTH_TYPE_CLEAR AuthInfo byte field contains a cleartext
> password, encoded as a Unicode string.

I just wanted to try and see where this has got to, as I've not seen any communication on the issue for over a month.  This is currently one of my two top priority issues, along with the failure to join trusted domain.

The problem is that if Microsoft treats this buffer as 'UCS2' when converting to UTF8 (for kerberos keys), then any random buffer is valid input.  However, where that buffer to contain UTF16 sequences (as typed by a user as a password), it would not map to the correct UTF8 sequence for conversion into the kerberos key.

Similarly, if the buffer is UTF16, then what should we do when windows machines set a random stream of bytes as the password?

This is a critical interopability issue, as AES keys become more used.

Similarly, this affects the creation of ARCFOUR keys from UTF8 input on non-windows machines (should the characters be expanded to UCS2 or UTF16 prior to the MD4?)

Andrew Bartlett
--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Red Hat Inc.                  http://redhat.com