Doubts about Samba's unicode translation tables

[Xavi Hernandez]

Hi Jeremy,

On Fri, Apr 19, 2024 at 5:56 PM Jeremy Allison <jra at samba.org> wrote:

> On Fri, Apr 19, 2024 at 11:04:40AM +0200, Xavi Hernandez via
> samba-technical wrote:
> >Hi all,
> >
> >I'm currently trying to integrate Samba with CephFS, and one of the
> >important things to improve is to access CephFS files in an insensitive
> way
> >without needing to scan the entire directory from smbd.
> >
> >During this work I've found that Samba does the case insensitive
> comparison
> >using a couple of UTF16 translation tables (one that converts to uppercase
> >and another that converts to lowercase).
> >
> >Looking at how NTFS does the same thing I've found that it also uses a
> >UTF16 table stored in the $UpCase special NTFS file located in the root of
> >the volume.
> >
> >The first question is why Samba uses two tables while Windows only
> requires
> >one ?
> >For what purpose is the lowercase translation table in Samba used ?
> >Is the Samba's case-insensitive comparison method actually equal to
> Windows
> >?
> >
> >I've also extracted the $UpCase file from a Windows 11 machine and I've
> >found that the Samba's uppercase table is very similar but not identical
> >(there are 339 different values). Is this expected ?
>
> Posting the differences would be very interesting. Thanks !
>

In all cases except one, the Samba table translates the unicode character
to itself, while the Windows table does some other translation.

The exception is character U+03C2, which is translated to itself by the
Windows table, but to U+03A3 by the Samba table.

I've attached a file with all the differences.

Regards,

Xavi
-------------- next part --------------
Differences in uppercase translation tables for Samba and Win11.

'------' means that there's no translation, it's mapped to itself.

Unicode   Samba  Win11
U+0180 -> ------ U+0243
U+0195 -> ------ U+01F6
U+019A -> ------ U+023D
U+019E -> ------ U+0220
U+01BF -> ------ U+01F7
U+01F9 -> ------ U+01F8
U+0219 -> ------ U+0218
U+021B -> ------ U+021A
U+021D -> ------ U+021C
U+021F -> ------ U+021E
U+0223 -> ------ U+0222
U+0225 -> ------ U+0224
U+0227 -> ------ U+0226
U+0229 -> ------ U+0228
U+022B -> ------ U+022A
U+022D -> ------ U+022C
U+022F -> ------ U+022E
U+0231 -> ------ U+0230
U+0233 -> ------ U+0232
U+023C -> ------ U+023B
U+0242 -> ------ U+0241
U+0247 -> ------ U+0246
U+0249 -> ------ U+0248
U+024B -> ------ U+024A
U+024D -> ------ U+024C
U+024F -> ------ U+024E
U+0250 -> ------ U+2C6F
U+0251 -> ------ U+2C6D
U+026B -> ------ U+2C62
U+0271 -> ------ U+2C6E
U+027D -> ------ U+2C64
U+0280 -> ------ U+01A6
U+0289 -> ------ U+0244
U+028C -> ------ U+0245
U+0371 -> ------ U+0370
U+0373 -> ------ U+0372
U+0377 -> ------ U+0376
U+037B -> ------ U+03FD
U+037C -> ------ U+03FE
U+037D -> ------ U+03FF
U+03C2 -> U+03A3 ------
U+03D7 -> ------ U+03CF
U+03D9 -> ------ U+03D8
U+03DB -> ------ U+03DA
U+03DD -> ------ U+03DC
U+03DF -> ------ U+03DE
U+03E1 -> ------ U+03E0
U+03F2 -> ------ U+03F9
U+03F8 -> ------ U+03F7
U+03FB -> ------ U+03FA
U+0450 -> ------ U+0400
U+045D -> ------ U+040D
U+048B -> ------ U+048A
U+048D -> ------ U+048C
U+048F -> ------ U+048E
U+04C6 -> ------ U+04C5
U+04CA -> ------ U+04C9
U+04CE -> ------ U+04CD
U+04CF -> ------ U+04C0
U+04ED -> ------ U+04EC
U+04F7 -> ------ U+04F6
U+04FB -> ------ U+04FA
U+04FD -> ------ U+04FC
U+04FF -> ------ U+04FE
U+0501 -> ------ U+0500
U+0503 -> ------ U+0502
U+0505 -> ------ U+0504
U+0507 -> ------ U+0506
U+0509 -> ------ U+0508
U+050B -> ------ U+050A
U+050D -> ------ U+050C
U+050F -> ------ U+050E
U+0511 -> ------ U+0510
U+0513 -> ------ U+0512
U+0515 -> ------ U+0514
U+0517 -> ------ U+0516
U+0519 -> ------ U+0518
U+051B -> ------ U+051A
U+051D -> ------ U+051C
U+051F -> ------ U+051E
U+0521 -> ------ U+0520
U+0523 -> ------ U+0522
U+1D79 -> ------ U+A77D
U+1D7D -> ------ U+2C63
U+1EFB -> ------ U+1EFA
U+1EFD -> ------ U+1EFC
U+1EFF -> ------ U+1EFE
U+1F80 -> ------ U+1F88
U+1F81 -> ------ U+1F89
U+1F82 -> ------ U+1F8A
U+1F83 -> ------ U+1F8B
U+1F84 -> ------ U+1F8C
U+1F85 -> ------ U+1F8D
U+1F86 -> ------ U+1F8E
U+1F87 -> ------ U+1F8F
U+1F90 -> ------ U+1F98
U+1F91 -> ------ U+1F99
U+1F92 -> ------ U+1F9A
U+1F93 -> ------ U+1F9B
U+1F94 -> ------ U+1F9C
U+1F95 -> ------ U+1F9D
U+1F96 -> ------ U+1F9E
U+1F97 -> ------ U+1F9F
U+1FA0 -> ------ U+1FA8
U+1FA1 -> ------ U+1FA9
U+1FA2 -> ------ U+1FAA
U+1FA3 -> ------ U+1FAB
U+1FA4 -> ------ U+1FAC
U+1FA5 -> ------ U+1FAD
U+1FA6 -> ------ U+1FAE
U+1FA7 -> ------ U+1FAF
U+1FB3 -> ------ U+1FBC
U+1FC3 -> ------ U+1FCC
U+1FF3 -> ------ U+1FFC
U+214E -> ------ U+2132
U+2184 -> ------ U+2183
U+2C30 -> ------ U+2C00
U+2C31 -> ------ U+2C01
U+2C32 -> ------ U+2C02
U+2C33 -> ------ U+2C03
U+2C34 -> ------ U+2C04
U+2C35 -> ------ U+2C05
U+2C36 -> ------ U+2C06
U+2C37 -> ------ U+2C07
U+2C38 -> ------ U+2C08
U+2C39 -> ------ U+2C09
U+2C3A -> ------ U+2C0A
U+2C3B -> ------ U+2C0B
U+2C3C -> ------ U+2C0C
U+2C3D -> ------ U+2C0D
U+2C3E -> ------ U+2C0E
U+2C3F -> ------ U+2C0F
U+2C40 -> ------ U+2C10
U+2C41 -> ------ U+2C11
U+2C42 -> ------ U+2C12
U+2C43 -> ------ U+2C13
U+2C44 -> ------ U+2C14
U+2C45 -> ------ U+2C15
U+2C46 -> ------ U+2C16
U+2C47 -> ------ U+2C17
U+2C48 -> ------ U+2C18
U+2C49 -> ------ U+2C19
U+2C4A -> ------ U+2C1A
U+2C4B -> ------ U+2C1B
U+2C4C -> ------ U+2C1C
U+2C4D -> ------ U+2C1D
U+2C4E -> ------ U+2C1E
U+2C4F -> ------ U+2C1F
U+2C50 -> ------ U+2C20
U+2C51 -> ------ U+2C21
U+2C52 -> ------ U+2C22
U+2C53 -> ------ U+2C23
U+2C54 -> ------ U+2C24
U+2C55 -> ------ U+2C25
U+2C56 -> ------ U+2C26
U+2C57 -> ------ U+2C27
U+2C58 -> ------ U+2C28
U+2C59 -> ------ U+2C29
U+2C5A -> ------ U+2C2A
U+2C5B -> ------ U+2C2B
U+2C5C -> ------ U+2C2C
U+2C5D -> ------ U+2C2D
U+2C5E -> ------ U+2C2E
U+2C61 -> ------ U+2C60
U+2C65 -> ------ U+023A
U+2C66 -> ------ U+023E
U+2C68 -> ------ U+2C67
U+2C6A -> ------ U+2C69
U+2C6C -> ------ U+2C6B
U+2C73 -> ------ U+2C72
U+2C76 -> ------ U+2C75
U+2C81 -> ------ U+2C80
U+2C83 -> ------ U+2C82
U+2C85 -> ------ U+2C84
U+2C87 -> ------ U+2C86
U+2C89 -> ------ U+2C88
U+2C8B -> ------ U+2C8A
U+2C8D -> ------ U+2C8C
U+2C8F -> ------ U+2C8E
U+2C91 -> ------ U+2C90
U+2C93 -> ------ U+2C92
U+2C95 -> ------ U+2C94
U+2C97 -> ------ U+2C96
U+2C99 -> ------ U+2C98
U+2C9B -> ------ U+2C9A
U+2C9D -> ------ U+2C9C
U+2C9F -> ------ U+2C9E
U+2CA1 -> ------ U+2CA0
U+2CA3 -> ------ U+2CA2
U+2CA5 -> ------ U+2CA4
U+2CA7 -> ------ U+2CA6
U+2CA9 -> ------ U+2CA8
U+2CAB -> ------ U+2CAA
U+2CAD -> ------ U+2CAC
U+2CAF -> ------ U+2CAE
U+2CB1 -> ------ U+2CB0
U+2CB3 -> ------ U+2CB2
U+2CB5 -> ------ U+2CB4
U+2CB7 -> ------ U+2CB6
U+2CB9 -> ------ U+2CB8
U+2CBB -> ------ U+2CBA
U+2CBD -> ------ U+2CBC
U+2CBF -> ------ U+2CBE
U+2CC1 -> ------ U+2CC0
U+2CC3 -> ------ U+2CC2
U+2CC5 -> ------ U+2CC4
U+2CC7 -> ------ U+2CC6
U+2CC9 -> ------ U+2CC8
U+2CCB -> ------ U+2CCA
U+2CCD -> ------ U+2CCC
U+2CCF -> ------ U+2CCE
U+2CD1 -> ------ U+2CD0
U+2CD3 -> ------ U+2CD2
U+2CD5 -> ------ U+2CD4
U+2CD7 -> ------ U+2CD6
U+2CD9 -> ------ U+2CD8
U+2CDB -> ------ U+2CDA
U+2CDD -> ------ U+2CDC
U+2CDF -> ------ U+2CDE
U+2CE1 -> ------ U+2CE0
U+2CE3 -> ------ U+2CE2
U+2D00 -> ------ U+10A0
U+2D01 -> ------ U+10A1
U+2D02 -> ------ U+10A2
U+2D03 -> ------ U+10A3
U+2D04 -> ------ U+10A4
U+2D05 -> ------ U+10A5
U+2D06 -> ------ U+10A6
U+2D07 -> ------ U+10A7
U+2D08 -> ------ U+10A8
U+2D09 -> ------ U+10A9
U+2D0A -> ------ U+10AA
U+2D0B -> ------ U+10AB
U+2D0C -> ------ U+10AC
U+2D0D -> ------ U+10AD
U+2D0E -> ------ U+10AE
U+2D0F -> ------ U+10AF
U+2D10 -> ------ U+10B0
U+2D11 -> ------ U+10B1
U+2D12 -> ------ U+10B2
U+2D13 -> ------ U+10B3
U+2D14 -> ------ U+10B4
U+2D15 -> ------ U+10B5
U+2D16 -> ------ U+10B6
U+2D17 -> ------ U+10B7
U+2D18 -> ------ U+10B8
U+2D19 -> ------ U+10B9
U+2D1A -> ------ U+10BA
U+2D1B -> ------ U+10BB
U+2D1C -> ------ U+10BC
U+2D1D -> ------ U+10BD
U+2D1E -> ------ U+10BE
U+2D1F -> ------ U+10BF
U+2D20 -> ------ U+10C0
U+2D21 -> ------ U+10C1
U+2D22 -> ------ U+10C2
U+2D23 -> ------ U+10C3
U+2D24 -> ------ U+10C4
U+2D25 -> ------ U+10C5
U+A641 -> ------ U+A640
U+A643 -> ------ U+A642
U+A645 -> ------ U+A644
U+A647 -> ------ U+A646
U+A649 -> ------ U+A648
U+A64B -> ------ U+A64A
U+A64D -> ------ U+A64C
U+A64F -> ------ U+A64E
U+A651 -> ------ U+A650
U+A653 -> ------ U+A652
U+A655 -> ------ U+A654
U+A657 -> ------ U+A656
U+A659 -> ------ U+A658
U+A65B -> ------ U+A65A
U+A65D -> ------ U+A65C
U+A65F -> ------ U+A65E
U+A663 -> ------ U+A662
U+A665 -> ------ U+A664
U+A667 -> ------ U+A666
U+A669 -> ------ U+A668
U+A66B -> ------ U+A66A
U+A66D -> ------ U+A66C
U+A681 -> ------ U+A680
U+A683 -> ------ U+A682
U+A685 -> ------ U+A684
U+A687 -> ------ U+A686
U+A689 -> ------ U+A688
U+A68B -> ------ U+A68A
U+A68D -> ------ U+A68C
U+A68F -> ------ U+A68E
U+A691 -> ------ U+A690
U+A693 -> ------ U+A692
U+A695 -> ------ U+A694
U+A697 -> ------ U+A696
U+A723 -> ------ U+A722
U+A725 -> ------ U+A724
U+A727 -> ------ U+A726
U+A729 -> ------ U+A728
U+A72B -> ------ U+A72A
U+A72D -> ------ U+A72C
U+A72F -> ------ U+A72E
U+A733 -> ------ U+A732
U+A735 -> ------ U+A734
U+A737 -> ------ U+A736
U+A739 -> ------ U+A738
U+A73B -> ------ U+A73A
U+A73D -> ------ U+A73C
U+A73F -> ------ U+A73E
U+A741 -> ------ U+A740
U+A743 -> ------ U+A742
U+A745 -> ------ U+A744
U+A747 -> ------ U+A746
U+A749 -> ------ U+A748
U+A74B -> ------ U+A74A
U+A74D -> ------ U+A74C
U+A74F -> ------ U+A74E
U+A751 -> ------ U+A750
U+A753 -> ------ U+A752
U+A755 -> ------ U+A754
U+A757 -> ------ U+A756
U+A759 -> ------ U+A758
U+A75B -> ------ U+A75A
U+A75D -> ------ U+A75C
U+A75F -> ------ U+A75E
U+A761 -> ------ U+A760
U+A763 -> ------ U+A762
U+A765 -> ------ U+A764
U+A767 -> ------ U+A766
U+A769 -> ------ U+A768
U+A76B -> ------ U+A76A
U+A76D -> ------ U+A76C
U+A76F -> ------ U+A76E
U+A77A -> ------ U+A779
U+A77C -> ------ U+A77B
U+A77F -> ------ U+A77E
U+A781 -> ------ U+A780
U+A783 -> ------ U+A782
U+A785 -> ------ U+A784
U+A787 -> ------ U+A786
U+A78C -> ------ U+A78B