[clug] Linux routing problem - two interfaces on same media segment, one seems isolated

[steve jenkin]

I've inherited some machines to admin and run into a knotty
routing/network problem - and was hoping somebody on the list might have
had this problem or know how to fix it.

I've setup systems before with multiple interfaces and this should be
easy... I'm missing something, but don't know what. The next place to
look is the switches, but we've just replaced them & the problem
persisted :-(

The servers sit on a public and private (management) IP network.
But both networks are carried over a single media segment (one big switch).

Two servers have a single interface connected, multi-homed (two IP
addresses, one each network). They worked fine & continue to work.

A third server had the two networks on different interfaces, but still a
single media segment. (eth1 is public & works fine, eth0 is private).

When I cycled (ifdown/up) 'eth0' with the private network (.15), it lost
connectivity with all other devices on the private network - but tcpdump
shows there is still the normal arp etc 'chatter' - the interface is UP
and the kernel is aware of it.

Recently I've connected an interface of another host to the switch and
tried to give it a number (.80) on the private IP network - and it too
cannot see any other devices (including .15), even though it has only
one connection to this media segment.


The messages I see from ping & nmap -sP are:
  "Destination Host Unreachable"
which means the kernel doesn't know how route to the address.

tcpdump on both interfaces confirms nothing (arp/icmp) is going out, but
as you'd expect, both interfaces are seeing traffic.

I've read & tried a bunch of stuff on this problem - this link is good
<http://linux-ip.net/html/ether-arp.html>

I've followed the advice on 'ARP flux' and turned on 'arp-filter'.
That's supposed to make an interface only set arp addresses on an
interface that are routable on that interface. Still no go.


When I force the ARP addresses for the two multi-homed hosts (.1 & .2)
into the arp cache, ping goes silent - the kernel can now find a route,
but I see no arp or icmp packets on either interface.

Anyone got any clues???

Bringing an interface up with an address automatically adds that subnet
into "main" route table in the kernel, with that IP as the gateway...
How can that *not* work?

cheers
steve


"uname -a" for the 3 systems [old kernels]:

172.1.1.1:
  Linux  2.6.8-24.13-smp #1 SMP Fri Mar 18 10:19:42 UTC 2005 i686 i686
i386 GNU/Linux

172.1.1.2:
  Linux 2.6.8-24.13-smp #1 SMP Fri Mar 18 10:19:42 UTC 2005 i686 i686
i386 GNU/Linux

172.1.1.15:
 Linux  2.6.12-10-686 #1 Wed Feb 7 03:56:37 UTC 2007 i686 GNU/Linux


These commands run on host with 2 interfaces active (.15)

root at host:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   Iface
202.1.1.0       0.0.0.0         255.255.255.224 U       eth1
172.16.1.0      0.0.0.0         255.255.255.0   U       eth0
0.0.0.0         202.1.1.1       0.0.0.0         UG      eth1


[Show routing cache - nothing in there for 172.16.1 network]

root at host:~# netstat -C -ee -rn|grep -v 202.1.1
Kernel IP routing cache
Source          Destination     Gateway         Flags Metric Ref    Use
Iface    MSS   Window irtt  TOS HHRef HHUptod     SpecDst
172.16.1.129  224.0.0.1       224.0.0.1       ml    0      0        0 lo
      0     0      0     0   -1    0     172.16.1.15


root at host:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


root at host:~# for if in eth0 eth1; do echo "$if" `cat
/proc/sys/net/ipv4/conf/$if/arp_filter`; done
eth0 1
eth1 1


######## test before setting ARP manually ########

root at host:~# ping 172.16.1.2
PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
>From 172.16.1.15 icmp_seq=2 Destination Host Unreachable
>From 172.16.1.15 icmp_seq=3 Destination Host Unreachable
>From 172.16.1.15 icmp_seq=4 Destination Host Unreachable

--- 172.16.1.2 ping statistics ---
7 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5999ms

root at host:~# ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
>From 172.16.1.15 icmp_seq=1 Destination Host Unreachable
>From 172.16.1.15 icmp_seq=2 Destination Host Unreachable
>From 172.16.1.15 icmp_seq=3 Destination Host Unreachable

--- 172.16.1.1 ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5000ms



######## set ARP manually ########

root at host:~# arp -s -i eth0 172.16.1.1 00:30:48:73:7E:A2
root at host:~# arp -s -i eth0 172.16.1.2 00:30:48:73:7E:3E

root at host:~# arp -an
172.16.1.2  at 00:30:48:73:7E:3E [ether] PERM on eth0
202.1.1.2   at 00:30:48:73:7E:3E [ether] on eth1
172.16.1.1  at 00:30:48:73:7E:A2 [ether] PERM on eth0
202.1.1.1   at 00:30:48:73:7E:A2 [ether] on eth1


######## test after setting ARP manually ########

root at host:~# ping -c 3 172.1.1.2
PING 172.1.1.2 (172.1.1.2) 56(84) bytes of data.

--- 172.1.1.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms








-- 
Steve Jenkin, Info Tech, Systems and Design Specialist.
0412 786 915 (+61 412 786 915)
PO Box 48, Kippax ACT 2615, AUSTRALIA

sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin