infiniband: ?

[tridge at samba.org]

Alexander,

 > So making a datagram-based or TCP/IP- (IPoIB) like layer might be a
 > better option. I'm still looking at the possibilities to get something
 > common to IB and Myrinetso that both commonly used high-speed networks
 > would be effectively useful for CTDB.

Remember that the key is latency, not ease of use of the API. When we
asked Christoph Raisch about IPoIB he said to forget it - its just way
too slow. Instead he recommended the verbs API. To make that work with
our epoll/select we would need a small hack to verbs to expose the
channel->fd file descriptor so we can add it to the list of fd's that
we watch.

If you look inside the ibv_get_cq_event() function then I think you'll
get the idea. Basically we will add a function ibv_get_fd() which
returns the channel->fd to us, then add that to our epoll fd list,
then when epoll tells us that the fd is readable we will call the
verbs API functions to get the next IB event.

That should hopefully give us the low latency of verbs, while allowing
us to be properly event driven. Of course, this assumes it all works :)

Cheers, Tridge