[Samba] Mount unix samba 4 share to osx client without mangled file names

[Markus Doits]

Hello,

I have a unix server (arch linux) with samba 4.1.14. The share has files
inside which where created directly on the server. Linux and OSX
(10.10.1) clients should access the share and store files.

When I mount this samba share into the osx client (10.10.1), files
created directly on the server with special names like `File with
"quotes"` are displayed mangled (`C7XSWH~V`) on the osx client.

When I create a file with special chars on the OSX client mount, it
displays correctly in OSX but on the unix server it is display with a
weird symbol like `File with quotes`.

I assume it is because OSX expects the filenames to be in utf-8 nfd
form, but the filenames on the server server are stored in utf-8 nfc form.

Am I right about this? Is this the problem?

Basically I want to have a working samba share on my linux server where
users from OSX and unix clients can access and store files, and special
chars in file names should be allowed. Is there any way to do this cross
platform?

Do I have to tell osx to simply use utf-8 nfc encoding for the samba
share? Or is there some special samba server/client config?

I already tried things like explicitly setting the encoding in smb.conf
on the unix server (withtout *really* knowing what I do), with no luck:

----
unix charset = utf-8
dos charset = utf-8

# --> error in log:
# invalid DOS charset: 'dos charset' must not be UTF8, using (default
# value) CP850 instead

# or like suggested in SO comment
http://serverfault.com/questions/636956/mount-unix-samba-4-share-to-osx-client-without-mangled-file-names

dos charset = utf-8-mac

# --> error in log:
# dos charset 'utf-8-mac' unavailable - using ASCII
----

Is there a way to set up the samba server in a heterogeneous environment
(osx and linux)?

Thanks for your help!

Markus