Challenges in merging the charcnv code

[Jeremy Allison]

On Wed, Mar 25, 2009 at 02:20:12PM +1100, Andrew Bartlett wrote:
> I've hit a rather nasty challenge while trying to merge the charcnv code
> between Samba3 and Samba4.
> 
> The fundamental problem starts with this:
> 
>      if (allow_badcharcnv) {
>                 /* Not implemented yet */
>                 return false;
>      }
> 
> In Samba3, many of the pull_ and push_ string functions set this flag,
> and expect to map invalid character sequences to an _
> 
> In Samba4, these functions do not.   Now what behaviour should I
> implement, and indeed should this be a property of the character set
> conversions at all?  

Yes, sorry. Please keep existing behavior, changing this will
cause problems for existing deployments. Don't check in code
changes that do this please.

> This is why I wonder if perhaps it should instead be a pre-filter in the
> cases where we actually need this?  Perhaps by the time we release what
> comes after Samba 3.4 we can expect that servers will correctly report
> the character set of the data they hold?

Nope. That will never happen. People don't always know.

> Whatever happens, I don't want to end up with a solution where we have
> 'if in Samba3' checks in the core character set routines. 
> 
> Could we at least agree to always munge to 0xFFFD?

No, sorry. That's not compatible with existing behavior. It will
break existing deployments. Now you could argue that they're already
broken, and we should log every time when we're forced to do this
to get people to fix it, but no changes here please.

Jeremy.