infiniband: ?

Alexander Bokovoy ab at samba.org
Sat Nov 18 22:04:40 GMT 2006


Tridge,

tridge at samba.org wrote:
> Alexander,
> 
>> So making a datagram-based or TCP/IP- (IPoIB) like layer might be a
>>  better option. I'm still looking at the possibilities to get
>> something common to IB and Myrinetso that both commonly used
>> high-speed networks would be effectively useful for CTDB.
> 
> Remember that the key is latency, not ease of use of the API. When we
>  asked Christoph Raisch about IPoIB he said to forget it - its just
> way too slow. Instead he recommended the verbs API. To make that work
> with our epoll/select we would need a small hack to verbs to expose
> the channel->fd file descriptor so we can add it to the list of fd's
> that we watch.
> 
> If you look inside the ibv_get_cq_event() function then I think
> you'll get the idea. Basically we will add a function ibv_get_fd()
> which returns the channel->fd to us, then add that to our epoll fd
> list, then when epoll tells us that the fd is readable we will call
> the verbs API functions to get the next IB event.
Yes, been there, saw that. :-)
We'll see. I'm currently going through your prototype code and let Peter
to handle IB prototype. I can test CTDB/IB scalability up to 64 nodes at
one site.

> That should hopefully give us the low latency of verbs, while
> allowing us to be properly event driven. Of course, this assumes it
> all works :)
Yes, this is a major assumption. :-)


-- 
/ Alexander Bokovoy
Samba Team                      http://www.samba.org/
ALT Linux Team                  http://www.altlinux.org/
Midgard Project Ry              http://www.midgard-project.org/


More information about the samba-technical mailing list