Fwd: backing up to a Drobo CIFS VFS: Error -11 sending data on socket to server, Bufferbloat?

[Richard Sharpe]

On Fri, May 12, 2017 at 10:17 AM, Robert Kudyba <rkudyba at fordham.edu> wrote:
> I posted the graph at storm.cis.fordham.edu/~rkudyba/wireshark.pdf

OK. There looks like some gaps where there is little or no write traffic.

>>> The earliest message you posted suggests that the client (Linux) was
>>> complaining of no activity for 120 seconds, which suggests that the
>>> Drobo stopped responding at some point. Thus I would expect to see one
>>> or more writes that did not get responses, followed by a connection
>>> drop and reconnect.
>
> Some random messages from tshark not exactly sure what I'm looking for
> as the time stamps aren't easy to interpret:
> 41674 217.172644 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] Write AndX Response, FID: 0x4ca4, 65536 bytes
> 41702 217.177652 localserver → drobo SMB 14546 Write AndX Request,
> FID: 0x4ca4, 65536 bytes at offset 54329344[TCP segment of a
> reassembled PDU]
> 41726 217.183095 drobo → localserver SMB 117 Write AndX Response, FID:
> 0x4ca4, 65536 bytes
>  4156 136.143502 drobo → localserver SMB 252 Trans2 Response<unknown>
>  4254 161.145741 drobo → localserver SMB 117 [TCP ZeroWindow] Write
> AndX Response, 65536 bytes
>  4260 161.147530 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
>  4566 161.219832 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] Write AndX Response, 65536 bytes
>  4594 161.225586 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
>  4690 162.010514 drobo → localserver SMB 117 [TCP ZeroWindow] Write
> AndX Response, 65536 bytes
>  4696 162.011564 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
>  4697 162.011830 drobo → localserver SMB 204 Trans2 Response<unknown>
>  1056   4.654442 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] [TCP Previous segment not captured] Write AndX Response,
> 65536 bytes
>  1057   4.660219 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] Write AndX Response, 65536 bytes
>  1082   4.808276 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] [TCP Previous segment not captured] Write AndX Response,
> 65536 bytes
>   2718  33.792075 localserver → drobo SMB 14546 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 167563264[TCP segment of a
> reassembled PDU]
>  2723  33.795265 localserver → drobo SMB 20338 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 167628800[TCP segment of a
> reassembled PDU]
>  2726  33.806707 drobo → localserver SMB 117 [TCP ACKed unseen
> segment] Write AndX Response, 65536 bytes
>  2727  33.806736 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
>  2730  33.807123 localserver → drobo SMB 17442 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 167694336[TCP segment of a
> reassembled PDU]
>  2736  33.811345 localserver → drobo SMB 8754 Write AndX Request, FID:
> 0x38ff, 65536 bytes at offset 167759872[TCP segment of a reassembled
> PDU]
>  2739  33.814173 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
>  2744  33.818007 localserver → drobo SMB 13098 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 167825408[TCP segment of a
> reassembled PDU]
>  41544 300.578408 localserver → drobo SMB 5858 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 2089480192[TCP segment of a
> reassembled PDU]
> 41548 300.582822 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
> 41551 300.584432 localserver → drobo SMB 13098 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 2089545728Write AndX Request, FID:
> 0x38ff, 4096 bytes at offset 2089828352
> 41555 300.589855 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
> 41558 300.590108 localserver → drobo SMB 13098 Write AndX Request,
> FID: 0x38ff, 65536 bytes at offset 2089611264[TCP segment of a
> reassembled PDU]
> 41559 300.594149 drobo → localserver SMB 117 Write AndX Response, 65536 bytes
> 41560 300.594438 drobo → localserver SMB 129 Write AndX Response, 4096 bytes
> 41584 300.602694 drobo → localserver SMB 129 [TCP ACKed unseen
> segment] Write AndX Response, 65536 bytes

Hmmm, there seem like gaps but that could be an artifact of you
selecting stuff at random. You are probably not going to find the
problem that way.

One thing you can do is to search for all those SMB requests where
there is no response. Of course since there are packet drops you might
find a few of those, but you can search for (!smb2.request_in) &&
(smb2.flags.response == 0) and that will show you all the requests for
which there is no response.

Another thing you can do is look for where the connection drops.
Search on (tcp.flags.reset == 1) || (tcp.flags.fin == 1) select one,
and then filter for all packets on the connection (tcp,port ==
portnum) where portnum is not 445 so you get just one TCP connection
and then you can look at what happened at the end of that tcp
connection.

This last approach might be the best if the problem is that the Linux
CIFS module is forgetting to send data at some point and we are not
looking for a Write request that the Drobo did not respond to.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)