convert_string + librpc/ndr (via openchange) == fail

sean finney seanius at
Tue May 17 01:21:08 MDT 2011

Hi Jeremy, Andrew,

All of this should be prefaced, btw, with the fact that I'm not
incredibly intimate with the samba4/openchange internals, though
I hope I'm showing enough willingness to get my hands dirty and
help resolve this :)

On Mon, May 16, 2011 at 04:41:10PM -0700, Jeremy Allison wrote:
> You're mistaking the meaning of LIBNDR_FLAG_STR_ASCII. It doesn't
> actually mean ascii (as in 7-bit clean characters) - it actually
> means what we call in Samba "dos charset". Look in librpc/ndr/ndr_string.c:
>  47         if (flags & LIBNDR_FLAG_STR_ASCII) {
>  48                 chset = CH_DOS;
>  49                 byte_mul = 1;
>  50                 flags &= ~LIBNDR_FLAG_STR_ASCII;
>  51         }
> Note that it sets chset to CH_DOS - which in Samba is set
> by default to CP850 (the Windows equivalent to iso8859-1).

Right, but the problem is that the codepage may vary from property
to property (it's not global, and they are created by third party
(i.e. different mail clients)).  I think the expectation is that the
client application (i.e. outlook, evolution-mapi, openchangeclient, 
libmapi etc) grabs the string, and looks up the codepage (which is
stored in one of a couple additional properties on the item), and does
the conversion itself if necessary.

> My guess is that OpenChange isn't setting this up in the
> same way as Samba on configure, so you're getting the fallback
> default of "ascii" here.
> See the code in source3/ that selects CP850 IBM850
> as the default names for this conversion.

I'll double check with my local build to see if I can specify a
default codepage, but this would be just papering over the issue
AFAICT, since if someone sends mail in another null-terminated 8-bit
codepage that doesn't match up there will still be failures.

And yes, the proper solution would be to have everything be unicode, and
openchange has pretty good support for putting/fetching unicode properties
when instructed to do so.  But I'm not sure there's a way to get exchange
servers to do any kind of transparent server-side conversion of existing
properties, and unless that's the case we're kinda stuck since we can't
control what people put in their emails :)

On Tue, May 17, 2011 at 10:28:35AM +1000, Andrew Bartlett wrote:
> A few things happened over the past few months:
>  The allow_bad_char change (but this was Samba4's historic behaviour)

And I suspect it was this one.  I can bisect it if you think it's

> If OpenChange wants the raw bytes from these 'strings', then it should
> be possible to parse these as uint8_t buffers.  However, it is generally
> useful to handle strings as strings, and conversion into the correct
> character set is important.

I think this may be necessary, though the code in question is being
generated from idl files + pidl, and i don't know if there's a way in idl
to say "variable size null terminated uint8 buffer" without calling it

> If you know the character set globally, then setting the dos charset
> will help, but I'm guessing this isn't as simple as that.  As we have
> the iconv handle code (and tests using the handles), I think the correct
> fix is to reinstate that onto the ndr structures, but in an optional way
> (understanding the limitations around enclosed structures etc).

Like I said above, this isn't a global setting, it can different between
different MAPI items depending on the configuration of the senders, AFAICT.

> That is, revert some of f9ca9e46ad24036bf00cb361a6cef4b2e7e98d7d, but in
> a way that makes the iconv handle strictly optional (ie a set function,
> rather than a function argument). 
> That handle would either change the CH_DOS handle at runtime for a
> specific NDR pull/push (a reasonable approach I think), or we can add a
> new CH_SPECIFIED or similar mapping onto another flag. 

I don't really have a strong opinion (or even much of an idea) on how
this should be fixed, but

> I hope this background helps as we do our best to continue supporting
> OpenChange.  It has not been our intention to break OpenChange, and we
> are very willing to assist in trying to find the right solution for both
> projects.

I hope the subject didn't come across as overly snarky, just wanted to
draw some attention to the matter :)


More information about the samba-technical mailing list