[clug] multiple network ports on one machine

[Daniel Pittman]

Adrian <adrian.blake at ieee.org> writes:

> A special purpose machine , a correlator, has 24 cards cramed packed
> with fast Xilinx FPGAs. At the end of a 30 second period each card in
> turn will dump the contents of its buffers onto the network. A total
> of 23, 721, 984 Kbps or per board (24 boards) = 988, 416 Kbps = 0.989
> Gbps. With packetisation and other overheads this will exceed the
> capacity of a single Gigabit port.

*nod*

> Fortunately there are 4 Ethernet ports per card, so I can use 2 at the
> same time. Via a network switch this is feed into a computer(s) which
> has 4 gigabit Ethernet ports. So there are two streams of data to be
> written to disk(s).

Right.

> Each port on the FPGA card can address a specific destination Ethernet
> address on the computer. The data will be sent as UDP packets.

OK.

> If I start 2 UDP servers will each be associated with the different
> Ethernet address (physical ports)?

No.  UDP services are associated with *IP* addresses, not Ethernet
addresses.  There is no direct relationship.

IP addresses have some relationship with Ethernet addresses, if your
computer has Ethernet cards, but the relationship is complicated,
especially if they are on the same subnet, or the same physical segment.

How are your network ports on the server configured?
What switch hardware are you using?


Anyway, when it comes time you want to either (a) use some sort of
bonding on the links that presents two cables as one logical link, or
(b) assign a distinct address to each cable.

For case (a) you run a single UDP listener.  (More or less. :)

For case (b) you listen to a different IP address for each stream.


> Is the solution to use different port numbers, i.e. bind IP address
> x.x.x.y:aaaa and bind x.x.x.z:bbbb.

Not as such, no, if the problem you are trying to solve is "how do I get
both cables active at the same time."

To achieve that, bonding or careful network setup.  We can help with
either, but both are complicated and vendor specific.


> I know that the next bottleneck will be writing to disk, but I will
> consider that later.
>
> This data dumping occurs every 30 sec for 12 hours ..... lot of data.

That concerns me: UDP is an unreliable protocol, so packets drop if
there is congestion.

For a constant 1Gbit data stream[1] where packets are sent at the same
rate all the time this isn't too bad: just ensure you have more network
capacity than data, and that your reader keeps up, and all is well.

For a dump of data every 30 seconds you have a *bursty* traffic stream,
with peaks and troughs of load.

Are you sure that your equipment is going to handle that peak, not just
the average load, without dropping data?

Heck, are you sure that your peak data rate is only going to be 1Gbit,
come to that?

Regards,
        Daniel

Footnotes: 
[1]  I am discounting packetization overhead here for convenience,
     because it doesn't much change the overall picture.