Server-side copy with sendfile system call

[David Disseldorp]

On Fri, 23 May 2014 16:46:36 +0800, Teng-Feng Yang wrote:

> I make the function fallback to read/write procedure when src=dest and
> pass the smb2.ioctl smbtorture test.

Great!

> However, I still can reproduce the performance issue mentioned before
> by the following steps.
> 
> 1. Create a 10G test file by dd:
> dd if=/dev/zero of=zero_10G.img bs=1G count=10

...

> 3. Recopy this file into the same directory from remote Windows server.
> From the second time we do the server-side copy, the throughput number
> will bouncing up and down between 120MB/s and 0MB/s repeatedly as
> mentioned in previous message. The copy_chunk_slow.log was gathered
> across this copy.

The wire traffic looks okay, but the FSCTL_SRV_COPYCHUNK round-trip time
is huge for some of the requests. I've attached a copy-chunk IOPS graph,
that shows the stalls captured in copy_chunk_slow.log. Each of the flat
~0 IOPS sections is your client waiting for the server to respond to
a FSCTL_SRV_COPYCHUNK request.

Looking at the first slow IO (frame 209), the round-trip time for the
request is 6.64 seconds! It requests a transfer of 16MB from the src
to dest in 1MB chunks, and aside from the offsets it doesn't differ
from the request immediately prior (frame 206) that completed in 0.13
seconds.

Further information is needed to find out what your server is doing
while processing the stalled requests.

> If you still need any information, please feel free to let me know.

I'd suggest using "strace -T  -p <pid_of_smbd_for_this_client>" to check
whether the waits are caused by large sendfile syscall delays in kernel.

What kind of server HW/OS/kernel version are you using to run this test?

Cheers, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: copychunk_sendfile_slow.png
Type: image/x-apple-ios-png
Size: 14626 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140523/1de897c3/attachment.bin>