[clug] Linux routing problem - two interfaces on same media segment, one seems isolated

[Daniel Pittman]

Paul Wayper <paulway at mabula.net> writes:
> Daniel Pittman wrote:
> | steve jenkin <sjenkin at canb.auug.org.au> writes:
> |> I've setup systems before with multiple interfaces and this should be
> |> easy... I'm missing something, but don't know what.
> |
> | There is probably one key thing you are missing if you have two
> | interfaces on a single physical segment:
> |
> | Linux implements the "weak end host model", in which the addresses of
> | the system are a property of the *host*, not of the *interface*.
> |
> | This means, for example, that it is going to ARP for all your IP
> | addresses on a single physical device per physical segment.
> |
> | This also means that you are not going to get, for your machine, any of
> | load balancing, improved performance, or improved reliability, by
> | connecting multiple physical NICs to a single physical segment.
> | (Without additional effort.)
>
> Dan, I don't think Steve's after NIC bonding here.
>
> As I understand it, he has two networks, public and private.  All
> machines have connections to both the public and private networks via
> NICs.  The complication is that the networks are on the same physical
> switch, and two of the machines have one NIC (each) and are using
> multi-homing to provide two interfaces, one with a public IP and one
> with a private one.

Yes -- I didn't think he was, either, but it wasn't entirely clear what
the goals of the original designer were, or if that was useful
information.

> I'm assuming that Steve's objective here is not to have redundancy or
> load balancing.  It's to allow the machines to talk to eachother on
> one network that the internet connection can't see, and another to
> talk to the internet.

As far as I know, yes.

> The fact that the machines are using a switch which has access to the
> internet to talk privately is odd, but that's what VLANs were invented
> for and we can assume that they either plan to or are using VLANs on
> the switch to ensure traffic privacy.

Actually, no: I don't think we can assume that they are planning on
using a VLAN to support this, or they already would have done so -- or
at least stated as much.

> Some day, one imagines, they might get second NICs for the ones which
> only have one, and on that day they can buy a second switch and have
> the private network completely separate.

Yup.  That would work just fine.  (Having two cables tagged into their
own VLAN by the switch, effectively giving two physical segments, would
also work, because it would give two physical segments.)

> This is why I don't think your suggestion here will work:
>
> | Sure: either pull one of the physical cards, so you only have a single
> | card per physical segment, or use bonding so that the cards can both
> | function as if they were a single physical card for the segment.
>
> Unfortunately, Steve, I don't have any idea why you might be having
> this problem.  I would check your hardware and your kernel log,
> though.

It is because he has two physical Ethernet cards on one physical
segment, and Linux handles that in a way that is not entirely intuitive
to many people.

Either of those solutions: bonding, or ignoring one NIC, will solve the
problem.  Hardware is the /last/ resort, if the issues continue once the
software level problems are solved.

> I bought a machine with some accursed Sky2-based GigE NICs on-board
> and they were so dodgy I had to finally put a separate GigE PCI card
> in and turn the on-board ones off.  Seriously, folks, Sky2 cards are
> teh crapzors.

Which brand of motherboard was this, and can you give more detail of the
issues?  I don't know of any problems with Sky2 hardware, and I would
rather not have to find out by putting my own (or a client's) money on
the table. :)

Regards,
        Daniel