Challenges in merging the charcnv code

[Andrew Bartlett]

I've hit a rather nasty challenge while trying to merge the charcnv code
between Samba3 and Samba4.

The fundamental problem starts with this:

     if (allow_badcharcnv) {
                /* Not implemented yet */
                return false;
     }

In Samba3, many of the pull_ and push_ string functions set this flag,
and expect to map invalid character sequences to an _

In Samba4, these functions do not.   Now what behaviour should I
implement, and indeed should this be a property of the character set
conversions at all?  

The Samba3 approach (almost any character set conversion can accept bad
input) is over-broad:  Should an attempt to upload a file with the name
in the wrong character set succeed, but with a munged destination name?
Should protocol strings, usernames and passwords be munged?

The Samba4 approach in this area is much more limited:  There is a need
in Samba4 to munge invalid UTF16 in the same way that Microsoft does,
when performing a String2Key operation.  Invalid UTF16 sequences are
mapped to the character 0xFFFD, when the character set of UTF16_MUNGED
is specifically selected.

This is why I wonder if perhaps it should instead be a pre-filter in the
cases where we actually need this?  Perhaps by the time we release what
comes after Samba 3.4 we can expect that servers will correctly report
the character set of the data they hold?

Whatever happens, I don't want to end up with a solution where we have
'if in Samba3' checks in the core character set routines. 

Could we at least agree to always munge to 0xFFFD?

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Red Hat Inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba-technical/attachments/20090325/261584d7/attachment.bin