convert_string + librpc/ndr (via openchange) == fail

[Stefan (metze) Metzmacher]

Am 17.05.2011 09:21, schrieb sean finney:
> Hi Jeremy, Andrew,
> 
> All of this should be prefaced, btw, with the fact that I'm not
> incredibly intimate with the samba4/openchange internals, though
> I hope I'm showing enough willingness to get my hands dirty and
> help resolve this :)
> 
> On Mon, May 16, 2011 at 04:41:10PM -0700, Jeremy Allison wrote:
>> You're mistaking the meaning of LIBNDR_FLAG_STR_ASCII. It doesn't
>> actually mean ascii (as in 7-bit clean characters) - it actually
>> means what we call in Samba "dos charset". Look in librpc/ndr/ndr_string.c:
>>
>>  47         if (flags & LIBNDR_FLAG_STR_ASCII) {
>>  48                 chset = CH_DOS;
>>  49                 byte_mul = 1;
>>  50                 flags &= ~LIBNDR_FLAG_STR_ASCII;
>>  51         }
>>
>> Note that it sets chset to CH_DOS - which in Samba is set
>> by default to CP850 (the Windows equivalent to iso8859-1).
> 
> Right, but the problem is that the codepage may vary from property
> to property (it's not global, and they are created by third party
> (i.e. different mail clients)).  I think the expectation is that the
> client application (i.e. outlook, evolution-mapi, openchangeclient, 
> libmapi etc) grabs the string, and looks up the codepage (which is
> stored in one of a couple additional properties on the item), and does
> the conversion itself if necessary.
> 
>> My guess is that OpenChange isn't setting this up in the
>> same way as Samba on configure, so you're getting the fallback
>> default of "ascii" here.
>>
>> See the code in source3/configure.in that selects CP850 IBM850
>> as the default names for this conversion.
> 
> I'll double check with my local build to see if I can specify a
> default codepage, but this would be just papering over the issue
> AFAICT, since if someone sends mail in another null-terminated 8-bit
> codepage that doesn't match up there will still be failures.
> 
> And yes, the proper solution would be to have everything be unicode, and
> openchange has pretty good support for putting/fetching unicode properties
> when instructed to do so.  But I'm not sure there's a way to get exchange
> servers to do any kind of transparent server-side conversion of existing
> properties, and unless that's the case we're kinda stuck since we can't
> control what people put in their emails :)
> 
> On Tue, May 17, 2011 at 10:28:35AM +1000, Andrew Bartlett wrote:
>> A few things happened over the past few months:
>>
>>  The allow_bad_char change (but this was Samba4's historic behaviour)
> 
> And I suspect it was this one.  I can bisect it if you think it's
> necessary.
> 
>> If OpenChange wants the raw bytes from these 'strings', then it should
>> be possible to parse these as uint8_t buffers.  However, it is generally
>> useful to handle strings as strings, and conversion into the correct
>> character set is important.
> 
> I think this may be necessary, though the code in question is being
> generated from idl files + pidl, and i don't know if there's a way in idl
> to say "variable size null terminated uint8 buffer" without calling it
> "string".

If not we need to create one...

>> If you know the character set globally, then setting the dos charset
>> will help, but I'm guessing this isn't as simple as that.  As we have
>> the iconv handle code (and tests using the handles), I think the correct
>> fix is to reinstate that onto the ndr structures, but in an optional way
>> (understanding the limitations around enclosed structures etc).
> 
> Like I said above, this isn't a global setting, it can different between
> different MAPI items depending on the configuration of the senders, AFAICT.

Then using uint8_t array seems to be the only option.

Or the code page also encoded somewhere in the request?

metze

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20110517/ff1d111e/attachment.pgp>