[clug] Diagnosing SSH performance problems [SEC=UNCLASSIFIED]

[Daniel Pittman]

Michael Cohen <scudette at gmail.com> writes:

> The problem occurs frequently enough to even be documented on wikipedia
> http://en.wikipedia.org/wiki/Path_MTU_discovery

That documents the behaviour I mentioned:

    This can result in connections that complete the TCP three-way handshake
    correctly, but then hang when data is transferred. This state is referred
    to as a "black hole connection".

That is, you get a *hang* when you use large packets, not a connection that
limps along at ~ 9600bps, as the OP documented:

>>>> So I have an SSH problem where pushing data over an SSH connation gets
>>>> me less than 9600 baud and numerous halts and timeouts. Pulling from the
>>>> same host just rips along as you'd expect.

> It is not related to noise or saturation and has nothing to do with
> window scaling - usually just an incorrectly configured firewall.

If the symptoms matched a PMTU black hole, I would absolutely agree with you.
In this case, though, what is seen is that one direction is fast, and the
other direction is slow, but both directions complete successfully.

So, unless there has been a stealth implementation of RFC 4821 PMTU discovery,
and it is bad enough that it causes dreadful performance, I don't think these
symptoms explain what the OP is seeing.


The PMTU issue you name are no less real, of course.  Just not what is
happening (in my opinion) in this particular case. ;)

        Daniel

> On Thu, Dec 10, 2009 at 10:59 PM, Daniel Pittman <daniel at rimspace.net> wrote:
>> Michael Cohen <scudette at gmail.com> writes:
>>
>>> What you describes sounds like your machine is configured for too high an
>>> MTU for the link.
>>
>> Are you assuming a noisy link, such as an analog modem without error
>> correction, or something like that?
>>
>> Otherwise incorrect MTU settings should result in either PMTU discovery, or a
>> stall as the client sends over-large packets, and those packets are never
>> delivered.
>>
>>> Try to lower your MTU and see if that fixes it. If that is the issue ensure
>>> your firewall does not block ICMP MTU discovery type messages so that your
>>> system is able to discover the appropriate MTU.
>>
>> Mmmm.  At the network layer, I would more suspect a firewall that strips the
>> window scaling option, and a Linux that sets window_scale to 2 so that the
>> connection limps along rather than stalling forever.[1]
>>
>> Alternately, something else such as high packet loss or latency on the link
>> might show these results; it could also be asymmetric load, with one direction
>> close to saturation, resulting in high latency in one direction only.[2]
>>
>>> On Thu, Dec 10, 2009 at 5:18 PM, Roppola, Antti - BRS
>>> <Antti.Roppola at daff.gov.au> wrote:
>>>> Hi all,
>>>>
>>>> So I have an SSH problem where pushing data over an SSH connation gets
>>>> me less than 9600 baud and numerous halts and timeouts. Pulling from the
>>>> same host just rips along as you'd expect.
>>>>
>>>> Nothing obvious seen using "ssh -v". Nothing relevant leaps out at me in
>>>> sshd_config.
>>>>
>>>> To me this suggests the issue is deeper down in the transport, network,
>>>> data link of physical layer. Viz, I can't think of anything in OpenSSH
>>>> that could be mis-configured to give asymmetric symptoms.
>>>>
>>>> Does this seem like a reasonable hypothesis?
>>
>> Yes, to me.  I can't really envision a scenario that results in asymmetric
>> behaviour when *only* OpenSSH is responsible.
>>
>>        Daniel
>>
>> Footnotes:
>> [1]  When the option was first introduced, this was set to 7, resulting in
>>     effectively a zero window when the option was stripped; now it default
>>     to 2 by default, and provides bad-but-functional connectivity.
>>
>> [2]  When you only send acks from your machine to theirs, performance stays
>>     good.  When you send bulk data, however, you are exposed to the
>>     transmission delays.
>>
>> --
>> ✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
>>               ♽ made with 100 percent post-consumer electrons
>> --
>> linux mailing list
>> linux at lists.samba.org
>> https://lists.samba.org/mailman/listinfo/linux
>>

-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons