[clug] Linux routing problem - two interfaces on same media segment, one seems isolated

[Daniel Pittman]

steve jenkin <sjenkin at canb.auug.org.au> writes:

> I've inherited some machines to admin and run into a knotty
> routing/network problem - and was hoping somebody on the list might
> have had this problem or know how to fix it.

Wow.  That is a nasty situation to inherit, because it is almost
certainly /never/ going to work the way whoever built it originally
expected.

> I've setup systems before with multiple interfaces and this should be
> easy... I'm missing something, but don't know what. 

There is probably one key thing you are missing if you have two
interfaces on a single physical segment:

Linux implements the "weak end host model", in which the addresses of
the system are a property of the *host*, not of the *interface*.  

This means, for example, that it is going to ARP for all your IP
addresses on a single physical device per physical segment.

This also means that you are not going to get, for your machine, any of
load balancing, improved performance, or improved reliability, by
connecting multiple physical NICs to a single physical segment.
(Without additional effort.)

[...]

> I've read & tried a bunch of stuff on this problem - this link is good
> <http://linux-ip.net/html/ether-arp.html>

Yes, that looks like a good summary of the issue.

> I've followed the advice on 'ARP flux' and turned on 'arp-filter'.
> That's supposed to make an interface only set arp addresses on an
> interface that are routable on that interface. Still no go.

[...]

> Anyone got any clues???

Sure: either pull one of the physical cards, so you only have a single
card per physical segment, or use bonding so that the cards can both
function as if they were a single physical card for the segment.

So ... if you want improved performance or redundancy, look into bonding
and ensure that your switches support it.

If you want a simple life then pull one of the cards physically out of
the segment.

Regards,
        Daniel

No, seriously, most anything else is just going to make you sad.