Infiniband & ESXi shared storage confusion

[explosivelobster]

This post is a bit rambling so apologies in advance, but the amount of knowledge on these forums is too hard to resist! I'm trying to provision some SSD shared storage for 2 (soon to be 3) ESXi hosts, the kit I have at my disposal is:

• 2x Dell Poweredge R720 (ESXi host)
• Dell Powervault NX3200 (storage server)
• Sun Oracle Flash Accelerator 3.2TB (rebranded Samsung 1725 PCIe NVMe) - 2 of these that live in the storage server, I just want to present these as raw devices
• ConnectX-3 cards for all hosts (flashed to latest FCBT OEM F/W) - dual port in server, single port in ESXi hosts
• Mellanox IS5022 8 port unmanaged infiniband switch
• Brocade ICX-6610 switch
• QSFP+ / SFP+ cables to string everything together

My ideal setup would be to use the CX3 cards in IB mode with the IS5022, leaving the ICX for VM / vMotion networks, but if that proves to be difficult or impossible I would be open to running on the ICX using a 40GbE link and iSCSI, but that would be the last resort. I'd also like to avoid purchasing any more kit if possible! I have some experience with iSCSI but have not used IB before.

I found this article on iSER on ESXi 6.7 (vSphere with iSER - How to release the full potential of your iSCSI storage! - VMware VROOM! Blog) but I was a bit concerned that they are using the CX3 pro cards, mine are not the pro version and I don't understand the difference.

My initial attempts have been with ESOS running as a KVM VM with the NVMe SSDs & CX3 card via PCIe passthrough, I've activated opensm and can establish a link to the IB switch, but now I'm a bit lost. ESOS is loading the ib_srpt driver so I imagine this is SRP? I would like to use iSER but the information I've found is confusing, the examples I've found seem to be using IPoIB which I've not configured and am not familiar with...

Another complication is the ESXi nodes are currently on 6.7 which isn't officially supported on the Dell 12th gen machines - if this is a problem happy to drop to 6.5.

What's the best way to string this lot together and get it working??

Any advice / links / suggestions welcome, because I'm very lost!

Thanks

 

[Rand__]

so i don't have all the answers, but

1. CX3 vs pro: https://forums.servethehome.com/index.php?threads/mellanox-354a-pro-40.25167/#post-233427
2. esx 6.7 only supports Ethernet, not IB.

If you would want IB you need something (a local VM ideally) to translate IB to VMWare/TcpIP but I have not seen a guide on that (probably easy on a single node, complicated in a HA manner )

If you can use ETH (not with your 5022), see
https://forums.servethehome.com/index.php?threads/cx3-iser.23964/ & https://forums.servethehome.com/index.php?threads/zfs-iser-san.23521/

 

[explosivelobster]

Thanks that's good to know - so iSER over 40GbE should still be possible then!

In which case I have another question, the CX3 card in the server was originally set to Eth mode but ESOS still claims it with the ib_srpt driver - that's not right is it? How do I tell it to use 40GbE?

 

[Rand__]

usually you should be able to use something like mlx_config, which is part of the MFT tools iirc,
see
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

 

[explosivelobster]

usually you should be able to use something like mlx_config, which is part of the MFT tools iirc,
see
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

Yep I've used that to set Eth mode on both interfaces

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        IB(1)           ETH(2)
         LINK_TYPE_P2                        IB(1)           ETH(2)

Will do some more poking around in ESOS. Thanks again!

 

[Rand__]

(Note: Cards like the ConnectX-3 that support Ethernet and IB personalities are Ethernet by default until the IB module is loaded.)

so just dont load the IB module on ESOS?

 

[explosivelobster]

(Note: Cards like the ConnectX-3 that support Ethernet and IB personalities are Ethernet by default until the IB module is loaded.)

so just dont load the IB module on ESOS?

I've not explicitly loaded it, it appears to load by default on 1.3.9 - trying to see where I stop that happening!

 

[Rand__]

No idea if that old guide is still applicable...
Marc's Adventures in IT Land: Building & Using a Highly Available ESOS Disk Array

"Edit the ‘/etc/infiniband/openib.conf’ IB driver configuration file, and set the following two lines (on both hosts):

IPOIB_LOAD=yes
SET_IPOIB_CM=yes"