ctdb with Infiniband

[Shuichi Ihara]

Hi Stefan,

Thanks for sending patch and I saw this just was landed in the CTDB mainstream.
I've confirmed this patch works well and ib{0,1..} interfaces can be
detected and up them if it's still not running.
BTW, I got the following error messages that seems ARP requests can't
send Infiniband based H/W address.
It's different length between Infiniband H/W address and MAC address.

2010/08/01 13:30:55.928960 [ 3687]: common/system_linux.c:120 not an
ethernet address family (0x20)
2010/08/01 13:30:55.928985 [ 3687]: server/ctdb_takeover.c:240 sending
of arp failed on iface 'ib0' (Invalid argument)
2010/08/01 13:30:56.906027 [ 3687]: common/system_linux.c:120 not an
ethernet address family (0x20)
2010/08/01 13:30:56.906053 [ 3687]: server/ctdb_takeover.c:2657
sending of gratious arp on iface 'ib0' failed (Invalid argument)
2010/08/01 13:30:57.033025 [ 3687]: common/system_linux.c:120 not an
ethernet address family (0x20)
2010/08/01 13:30:57.033051 [ 3687]: server/ctdb_takeover.c:240 sending
of arp failed on iface 'ib0' (Invalid argument)
2010/08/01 13:30:57.911046 [ 3687]: common/system_linux.c:120 not an
ethernet address family (0x20)
2010/08/01 13:30:57.911070 [ 3687]: server/ctdb_takeover.c:2657
sending of gratious arp on iface 'ib0' failed (Invalid argument)
2010/08/01 13:30:58.138021 [ 3687]: common/system_linux.c:120 not an
ethernet address family (0x20)
2010/08/01 13:30:58.138047 [ 3687]: server/ctdb_takeover.c:240 sending
of arp failed on iface 'ib0' (Invalid argument)

As a quick workaround, we could Ignore sending ARP to Infiniband for
IPoIB. I've attempted an attached patch and confirmed fine as well.
I would land this patch to CTDB mainstream unless there are any objections.

Thanks
Ihara

2010/7/30 Stefan (metze) Metzmacher <metze at samba.org>:
> Hi Ihara,
>
>> If I am misunderstanding, please let me know. --enable-infiniband
>> means for Cluster Interlink of CTDB?
>> I was tying to ib0 interface one of TCP/IP network interface for
>> public network of CTDB, but was getting
>> PARTIALLYONLINE and couldn't become OK on 'ctdb status'.
>
> The problem is the interface name.
>
> See /etc/ctdb/events.d/10.interface doesn't know how to monitor
> ib* interfaces. It was missing some calls to setup the interface as "up".
>
> The attached patch should fix that.
>
> metze
>
>> Thanks
>> Ihara
>>
>> 2010/7/29 ronnie sahlberg <ronniesahlberg at gmail.com>:
>>> Hi,
>>>
>>> The bad news is that the infiniband support has experienced bit-rot
>>> and is not maintained. So you can not use it. You have to use TCP
>>> today.
>>>
>>>
>>> The good news is that we managed to get the ctdb protocol reasonably
>>> fast and non-chatty   so there is no real need for the low-latency
>>> infiniband transport for the inter-ctdb communications.
>>>
>>>
>>>
>>> Basically, the ctdb protocol is fast enough with GbE (or 10GbE) and
>>> TCP that the cifs serving bottlenecks lies elsewhere.
>>>
>>>
>>>
>>> regards
>>> ronnie sahlberg
>>>
>>>
>>>
>>> On Thu, Jul 29, 2010 at 12:09 PM, Shuichi Ihara <ihashu at gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I'm building ctdb (1.0.114) with --enable-infiniband option, but
>>>> getting the following errors.
>>>>
>>>> Compiling utils/ping_pong/ping_pong.c
>>>> Linking bin/ping_pong
>>>> Linking bin/ctdbd
>>>> ib/ibw_ctdb_init.o: In function `ctdb_flush_cn_queue':
>>>> /root/ctdb/ib/ibw_ctdb_init.c:140: undefined reference to `DLIST_REMOVE'
>>>> ib/ibw_ctdb_init.o: In function `ctdb_ibw_queue_pkt':
>>>> /root/ctdb/ib/ibw_ctdb_init.c:174: undefined reference to `DLIST_ADD_AFTER'
>>>> collect2: ld returned 1 exit status
>>>> make: *** [bin/ctdbd] Error 1
>>>>
>>>> I've tested with OFED-1.4.x and OFED-1.5.x for Infiniband stuff, but
>>>> got a same error.
>>>> Any idea to fix this build issue?
>>>>
>>>> Thanks
>>>> Ihara
>>>>
>>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ignore-Infiniband-ARP.patch
Type: application/octet-stream
Size: 518 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20100801/46d2d006/attachment.obj>