This post is a bit rambling so apologies in advance, but the amount of knowledge on these forums is too hard to resist! I'm trying to provision some SSD shared storage for 2 (soon to be 3) ESXi hosts, the kit I have at my disposal is:
My ideal setup would be to use the CX3 cards in IB mode with the IS5022, leaving the ICX for VM / vMotion networks, but if that proves to be difficult or impossible I would be open to running on the ICX using a 40GbE link and iSCSI, but that would be the last resort. I'd also like to avoid purchasing any more kit if possible! I have some experience with iSCSI but have not used IB before.
I found this article on iSER on ESXi 6.7 (vSphere with iSER - How to release the full potential of your iSCSI storage! - VMware VROOM! Blog) but I was a bit concerned that they are using the CX3 pro cards, mine are not the pro version and I don't understand the difference.
My initial attempts have been with ESOS running as a KVM VM with the NVMe SSDs & CX3 card via PCIe passthrough, I've activated opensm and can establish a link to the IB switch, but now I'm a bit lost. ESOS is loading the ib_srpt driver so I imagine this is SRP? I would like to use iSER but the information I've found is confusing, the examples I've found seem to be using IPoIB which I've not configured and am not familiar with...
Another complication is the ESXi nodes are currently on 6.7 which isn't officially supported on the Dell 12th gen machines - if this is a problem happy to drop to 6.5.
What's the best way to string this lot together and get it working??
Any advice / links / suggestions welcome, because I'm very lost!
Thanks
- 2x Dell Poweredge R720 (ESXi host)
- Dell Powervault NX3200 (storage server)
- Sun Oracle Flash Accelerator 3.2TB (rebranded Samsung 1725 PCIe NVMe) - 2 of these that live in the storage server, I just want to present these as raw devices
- ConnectX-3 cards for all hosts (flashed to latest FCBT OEM F/W) - dual port in server, single port in ESXi hosts
- Mellanox IS5022 8 port unmanaged infiniband switch
- Brocade ICX-6610 switch
- QSFP+ / SFP+ cables to string everything together
My ideal setup would be to use the CX3 cards in IB mode with the IS5022, leaving the ICX for VM / vMotion networks, but if that proves to be difficult or impossible I would be open to running on the ICX using a 40GbE link and iSCSI, but that would be the last resort. I'd also like to avoid purchasing any more kit if possible! I have some experience with iSCSI but have not used IB before.
I found this article on iSER on ESXi 6.7 (vSphere with iSER - How to release the full potential of your iSCSI storage! - VMware VROOM! Blog) but I was a bit concerned that they are using the CX3 pro cards, mine are not the pro version and I don't understand the difference.
My initial attempts have been with ESOS running as a KVM VM with the NVMe SSDs & CX3 card via PCIe passthrough, I've activated opensm and can establish a link to the IB switch, but now I'm a bit lost. ESOS is loading the ib_srpt driver so I imagine this is SRP? I would like to use iSER but the information I've found is confusing, the examples I've found seem to be using IPoIB which I've not configured and am not familiar with...
Another complication is the ESXi nodes are currently on 6.7 which isn't officially supported on the Dell 12th gen machines - if this is a problem happy to drop to 6.5.
What's the best way to string this lot together and get it working??
Any advice / links / suggestions welcome, because I'm very lost!
Thanks
Last edited: