Has anyone seen writev on FreeBSD return EINVAL after a write to a socket even though the data has hit the wire?

[Richard Sharpe]

On Thu, May 2, 2013 at 1:06 PM, David Collier-Brown <davec-b at rogers.com> wrote:
> On 05/02/2013 12:11 PM, Jeremy Allison wrote:
>> On Wed, May 01, 2013 at 08:52:52PM -0700, Richard Sharpe wrote:
>>> Hi folks,
>>>
>>> I am seeing a problem with SMB2 and writev's of responses, usually
>>> large responses, like a 64kiB read response. writev is returning
>>> EINVAL even though the data has actually hit the wire. I have verified
>>> this by looking at the capture and comparing the SMB sequence number
>>> in the header of the packet on the wire with what is in memory pointed
>>> to by the iovec sent to writev.
>>>
>>> It is most strange, and I am unable to think how this might happen.
>> At this point you need kernel tracing to find the bug
>> in FreeBSD. You might want to ping Julian about this.
>>
>> Jeremy.
>>
> You may want to log the base-address/length sets used, and look for
> anything that will trigger the EINVAL, possibly iuncluding alignment.
> My leaky memory says Solaris merely does poorly on unaligned pointers,
> while SunOS 4 suffered from problems...

This turned out to be an issue in the ixgbe driver (for Intel's 10GbE
card). For some reason it was setting the max DMA size to 65535.

While the FreeBSD TCP stack never sends a TCP + IP packet larger that
65536 bytes, including headers, you still have to allow for the
Ethernet headers (including VLAN tags.)

By firstly reducing SO_SNDBUF to 32768 and then changing the driver to
allow a little more, things are fine. That is, I performed two
separate experiments to demonstrate that I was on the correct track.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)