[clug] multiple network ports on one machine

[David Cottrill]

This sounds like an interesting experiment in it's own right - what is it
correlating?

Can you send the data from each FPGA connection to separate servers for
later analysis/combining (one network port each - solves many problems)?
Even virtual servers would simplify that aspect.

Given the prodigious computational possibilities of FPGAs, can more
pre-processing be done to cut down or smooth bandwidth?
David


On Tue, Mar 31, 2009 at 4:39 PM, Daniel Pittman <daniel at rimspace.net> wrote:

> Adrian <adrian.blake at ieee.org> writes:
>
> > A special purpose machine , a correlator, has 24 cards cramed packed
> > with fast Xilinx FPGAs. At the end of a 30 second period each card in
> > turn will dump the contents of its buffers onto the network. A total
> > of 23, 721, 984 Kbps or per board (24 boards) = 988, 416 Kbps = 0.989
> > Gbps. With packetisation and other overheads this will exceed the
> > capacity of a single Gigabit port.
>
> *nod*
>
> > Fortunately there are 4 Ethernet ports per card, so I can use 2 at the
> > same time. Via a network switch this is feed into a computer(s) which
> > has 4 gigabit Ethernet ports. So there are two streams of data to be
> > written to disk(s).
>
> Right.
>
> > Each port on the FPGA card can address a specific destination Ethernet
> > address on the computer. The data will be sent as UDP packets.
>
> OK.
>
> > If I start 2 UDP servers will each be associated with the different
> > Ethernet address (physical ports)?
>
> No.  UDP services are associated with *IP* addresses, not Ethernet
> addresses.  There is no direct relationship.
>
> IP addresses have some relationship with Ethernet addresses, if your
> computer has Ethernet cards, but the relationship is complicated,
> especially if they are on the same subnet, or the same physical segment.
>
> How are your network ports on the server configured?
> What switch hardware are you using?
>
>
> Anyway, when it comes time you want to either (a) use some sort of
> bonding on the links that presents two cables as one logical link, or
> (b) assign a distinct address to each cable.
>
> For case (a) you run a single UDP listener.  (More or less. :)
>
> For case (b) you listen to a different IP address for each stream.
>
>
> > Is the solution to use different port numbers, i.e. bind IP address
> > x.x.x.y:aaaa and bind x.x.x.z:bbbb.
>
> Not as such, no, if the problem you are trying to solve is "how do I get
> both cables active at the same time."
>
> To achieve that, bonding or careful network setup.  We can help with
> either, but both are complicated and vendor specific.
>
>
> > I know that the next bottleneck will be writing to disk, but I will
> > consider that later.
> >
> > This data dumping occurs every 30 sec for 12 hours ..... lot of data.
>
> That concerns me: UDP is an unreliable protocol, so packets drop if
> there is congestion.
>
> For a constant 1Gbit data stream[1] where packets are sent at the same
> rate all the time this isn't too bad: just ensure you have more network
> capacity than data, and that your reader keeps up, and all is well.
>
> For a dump of data every 30 seconds you have a *bursty* traffic stream,
> with peaks and troughs of load.
>
> Are you sure that your equipment is going to handle that peak, not just
> the average load, without dropping data?
>
> Heck, are you sure that your peak data rate is only going to be 1Gbit,
> come to that?
>
> Regards,
>        Daniel
>
> Footnotes:
> [1]  I am discounting packetization overhead here for convenience,
>     because it doesn't much change the overall picture.
>
> --
> linux mailing list
> linux at lists.samba.org
> https://lists.samba.org/mailman/listinfo/linux
>