[clug] multiple Ethernet ports

[Daniel Pittman]

Hugh Fisher <hugh.fisher at anu.edu.au> writes:
> Adrian Blake wrote:
>
>> This is not specifically a Linux question.
>> I am moving data via a switch from one place to another. The rate is 1.3
>> Gbps on 2 physical Ethernet ports into a network switch.  To that switch is
>> connected a server to store that data. The server has 4 physical Ethernet
>> ports.
>>
>> How do I write code to read all four ports concurrently and keep the
>> correct order of the packets?

Carefully?  Um, seriously, this is going to be relatively hard, but the
obvious answer is to transmit a packet stream with a sequence identifier
in it to the far end, then do reassembly.

You *will* get out of sequence packets, so you will have to buffer, but
that shouldn't be stupidly hard given you are already talking about a
significantly event driven, asynchronous solution.

That, obviously, involves multiple concurrent TCP streams and
appropriate routing.  Um, or you could add your own retransmission
mechanism and use UDP, allowing you to avoid TCP entirely at the cost of
reinventing parts of it.

> SCTP with multi-homing should do it for you.

Depending on how your application plays with it, yes, and probably less
expensive than building the same solution yourself at the application
layer.

Actually, on checking, it doesn't look like Linux has any support for
multihoming for performance — only for fail-over between multiple
paths.  So, clearly, I am reading the wrong documentation. :)

Can you point me to documentation on how to get it to increase
performance using multiple paths?  I would be interested to learn more.


Oh, and Adrian, there is at least one userspace SCTP implementation in a
library, which you may find valuable to use rather than aiming for an
in-kernel version.


However, the right question to ask is not the one you asked.  The right
question is "how do I increase the data transfer rate using multiple
Ethernet links between two machines"?

The answer to that question is much, much simpler, and allows you to
hide the complexity from the application which can simply continue to
use regular TCP, or whatever, in a single stream to handle this for you.


The way you achieve that is "bonding", which your OS and switch should
support, and which allows you to treat multiple independent Ethernet
links as one single faster link.

    http://www.linuxfoundation.org/en/Net:Bonding
    http://www.linuxhorizon.ro/bonding.html

That is going to be the lowest cost, most effective mechanism to deliver
the increased performance you need: it involves zero application
changes, only network configuration.

You *need* the balance-rr mode, however, together with a suitable switch
configuration; no other mode, including 802.3ad, supports a single
stream using more than a single channel worth of bandwidth.


That will save you, at least, from needing to rewrite away from a single
TCP stream until you need more than a couple of adapters worth of
bandwidth.

Regards,
        Daniel