Rename deadlock possible with Samba and ZFS On Linux in case insensitive mode

[Richard Sharpe]

Hi folks,

We have just run into a problem with Samba and ZFS On Linux so I
thought I would let people know what the issue is.

We run ZFS On Linux in case-insensitive mode. During an FSCT test we
started seeing deadlocks. We tracked the deadlocks down to renames and
lock_rename in the Linux kernel deadlocking while trying to take the
node-specific i_mutex twice.

This happened because two path names were used in the rename,
Internally we had a path like a/some-path/SomeOtherPath/SomeFile.xyz
for the source of the rename and windows gave us
A/some-path/SomeOtherPath/SomeOtherFile.xyz for the destination of the
rename.

The only difference was the case of the first component.

Since ZFS On Linux currently does not call d_add_ci in zpl_lookup when
the file system is Case Insensitive it results in two dentries in the
file pointing to the same inode and will cause deadlocks on rename.

A bug has been filed against the ZoL github repository:

        https://github.com/zfsonlinux/zfs/issues/4136

You can work around it to some extent in Samba but a full solution
will be a problem and there appears to be a simple solution in ZFS On
Linux.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)