Any guides to setting up IPoIB for ESXI ?

DBayPlaya2k3

Member
Nov 9, 2012
72
4
8
Houston
Hello does anyone have any guides to setting up Infiniband for ESXI 5.1 or later (current on 5.5).

I would like to explore this technology more as its a cheap altentive to 10GE but I have not found any good guides out there explaining whats required ect.

Some folks around here talk about requiring a subnet manager as well.

Thanks guys
 

poto

Active Member
May 18, 2013
199
61
28
I am also a beginner, but found the infiniband setup reasonably straightforward, thanks to the experts here and on the Mellanox/Vmware communities. Here are some links I found helpful:

Homelab Storage Network Speedup with .... Infiniband - ESX Virtualization
Custom firmware enables Windows 2012 IPoIB RDMA for Mellanox OEM Infiniband Cards - www.openida.com
Does vSphere OFED 1.8.1 compatible with vSphere... | Mellanox Interconnect Community
Infiniband@home : votre homelab à 20Gbps - Hypervisor.fr

There are also many helpful discussions on this site if you search through the 10gb networking threads.

I bought 3ea Mellanox MHRH2A-XSR HCA and qsfp copper cables for around $300 total. These are direct connected between 2 esxi host w/direct storage and backup file server. You install drivers on esxi host using putty/winscp, Win server os has inbox drivers. You can avoid expense of switch by direct-connecting HCA, but you have to provide software subnet manager. If you are using esxi 5.1/5.0, there is an open source subnet vib referenced in the links above that will allow direct connection between esxi hosts - cheap and fast vmotion results. I upgraded to esxi 5.5 because I wanted the >2tb vmdk ability. The open source subnet is not currently compatible with esxi 5.5, but you can also use multi-nic vmotion until a compatible subnet mgr is available for esxi 5.5.
Good luck!