[Samba] io_uring cause data corruption

[Jeremy Allison]

On Mon, Apr 27, 2020 at 11:21:35PM +0200, A L wrote:
> On 2020-04-27 18:45, Jeremy Allison via samba wrote:
> > On Mon, Apr 27, 2020 at 10:27:17AM +0200, A L via samba wrote:
> > > On 2020-04-26 19:46, Jeremy Allison via samba wrote:
> > > > On Sun, Apr 26, 2020 at 11:51:42AM +0200, A L via samba wrote:
> > > > > * Connected from a Windows 10 computer over 1G ethernet. *
> > > > > Copy data using Windows Explorer and FastCopy(1) from the
> > > > > Samba share to a local disk. * Verify the sha-256 sum on the
> > > > > files. From what I can see there is data corruption on many
> > > > > of the files. Sha-256 does not match. I copied the same
> > > > > files many times and the data corruption occurs within
> > > > > minutes. The total data set is about 800GB.
> > > > Can you do checksums on file fragments so we can discover at
> > > > what offset (if non-zero) the corruption occurs.
> > > Yes, I will check this. I saw a patch on the kernel mailing list
> > > about possible corruptions in during re-scheduling. I wonder if this
> > > is the problem I am hitting. I'll make some more tests with this
> > > patch. https://www.spinics.net/lists/io-uring/msg01706.html
> > Oh, that might explain it. I won't do further work until you can confirm
> > the Samba corruptions happen with this kernel patch also.
> Hello again,
> 
> I set up the following test case:
> * Linux 5.7-rc3 (with the patch from previous mail)
> * samba-4.12.1
> * gcc-9.3.0
> * liburing-0.6
> * glibc-2.30-r8
> 
> =================================
> Test 1)
> Copy 10 10GB files.
> 1) ddrescue -s 10G -v -f /dev/urandom 0.bin

Rather than using /dev/urandom can you use a
constant (non-zero) byte value ? That way we can spot
the first position where it changes in the file.

> 2) for((i=1;i<=10;i+=1)); do cp --reflink=always 0.bin $i.bin; done
> 3) sha256sum *.bin > sha256sum.txt
> 4) Windows 10, file explorer, copy the 10 files to a local disk D:\test\
> 5) Verify local files in D:\test with sha256sum
> 6) sha256sum was correct.
> 7) redid step 4 and 5. Now sha256sum was wrong, but all 10 files had the
> same (but wrong) csum!

The io_uring module has a vfs_io_uring:num_entries parameter
which by default is set to 128. The readv/writev requests are
queued inside smbd, but the number of outstanding io_uring
requests are set by io_uring_queue_init(vfs_io_uring:num_entries,...).
Can you try changing that value to see if the corruption point
moves ?

There is also a setting vfs_io_uring:sqpoll, currently set
to false. Try flipping that to true to see if it changes
the problem.