Anyone using IB over ConnectX-3 w/ ESXi 7.0U1?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
Hey,

I've never used an IB adapter before, but I was thinking of doing some direct-attached IB adapters for my 3 ESXi hosts now that I learned my 3 Intel 82598EB adapters I dug out of storage are now no longer supported. I do have VCSA if that matters.

I have 82599 10GbE for basic networking, but I wanted to use IB for a HA storage backend sync + HA heartbeat on something like drbd, ctl HA + hast, Starwind vSAN, something I can rig up on OmniOS, etc. (haven't decided which yet, if you have any war stories feel free to share).

I'm really cheap, but I'd like to get an adapter that has some longevity, given that VMware likes to depreciate hardware fairly expediently. I noticed there's lots of ConnectX-2 adapters still supported by ESXi, but I doubt for long (RIP my 82598s, *sniff*). So I was thinking of going with the MCX354A-FCCT Dual QDR+ 40/56, which looks like the best combination of supported + inexpensive I've found so far.

My questions:

  • Am I making a good choice for a cheapskate for adapter? Any other recommendations? What about this re-flashing I've been reading about, how extensive is that, and how hard is it?
  • What sort of protocol should I be using w/ ESXi (IPoIB, RDMA, etc.)?
  • I'm totally new to IB and have no idea what I'd be doing other than I know one of the machines needs a subnet manager - are multiple subnet managers possible? I'm trying to set up an HA environment, after all...
  • Any other guidance like what cables I should be using, etc. All the machines are in the same room but one is about 15ft from the others
  • I assume dual port cards are necessary for a DAC setup, for the "hub" machine (set up as a gateway + SN manager), at the very least - I was thinking 1x dual w/ 1x going to each of the two other machines. Would dual port cards be helpful for either of the other two machines?

Any guidance much appreciated. Thanks!
 

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
Did you ever move forward with this? I came here with all the same questions... :)
I bought a handful of Connectx-3s. The adapter's great. Two mellanox-branded CX354as and an EMC CX354a. They were all stock QCBT, I reflashed them to FCBT no problem. Re-flashing was easy.

ESXi only really supports ethernet for infrastructure. You can use SR-IOV + PVRDMA for very specific VMs (I think only Rhel/CentOS and Windows supported). So it's basically a 40GbE adapter, no other options.

Thankfully, ethernet is apparently a lot easier to work with. I believe they stopped supporting IB infrastructure in ESXi either 6.5 or 6.7 for that very reason - probably got tired of dealing with support tickets regarding it. They figure people who can afford VMware licenses can afford 100GbE adapters if they really need them.

If you're running over 10 feet, get an optical cable (they're not that expensive). I bought a couple of these Mellanox active cables in 2020, model 498386-B24 HP PN H1290 - the auction is still up and they still have some left: New Mellanox HP 498386-B24 5M 4x DDR/QDR QSFP IB Fiber Optic Cable (H1290) 993301511422 | eBay

They say they're for IB but they work for ethernet just the same. If you're going 10 ft or under, just get some copper DACs, make sure they're QSFP+ (and not plain QSFP). You'll probably have to research the exact cable via model number, since they don't often make the distinction in ads on places like ebay.

The whole experience hasn't been nearly as daunting as I thought it would be before I got into it. Just requires a bit of reading. Take a leap of faith and buy some stuff and you'll get the feel for it, a ton of it is cheap AF so it's not that risky.
 
  • Like
Reactions: tinfoil3d

lemmons

New Member
Oct 27, 2021
2
2
3
Thanks so much for the reply! I'm pushing the limits of some 2U builds at the moment, a pair of 5950X builds both running 7.0.2, and I'll admit I was running out of "leaps of faith". I'll go find some cheap gear and see if I can blow past 10Gb now.

Thanks again!
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
Thanks so much for the reply! I'm pushing the limits of some 2U builds at the moment, a pair of 5950X builds both running 7.0.2, and I'll admit I was running out of "leaps of faith". I'll go find some cheap gear and see if I can blow past 10Gb now.

Thanks again!
Oh I also forgot to mention I did flash the cards. I was super cheaping out. There are HP and EMC cards I am pretty sure were only factory configured for 10GbE I flashed with stock mellanox 40/56Gb and it went off without a hitch (kinda surprised with the HP model). I'll have to send you the models I got and what I tricked them into thinking they were when I get back to a computer, but it was surprisingly easy (was in Windows, command line). The NICs were like $20-$30 a piece, mind-blowingly cheap.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
So I found this doc about how you might be able to trick the ConnectX-3 pro into using ROCEv2:


That's in Linux, there's a possibility if you're using ESXi you may be able to do it, too (if you want to run this down): https://forums.servethehome.com/ind...to-run-roce-v2-on-cx-3-vpi-in-esxi-7-0.29628/

The exact NICs I bought were EMC and HP CX354A MCX354A-QCBT. These come from the factory as "QDR 40GB or 10GB", but can be flashed with MCX354A-FCBT firmware, which is "Dual FDR 56Gb/s or 40/56GbE". That means, you pay for one thing and get another with a tiny little bit of work.

I got 4 of them for $30-35 a piece, two of them had SFF-8088 to 8087 connectors which I use to run some external SAS channels back into the server for cards that have external SAS I want to use on internal drives.

Example here: Mellanox CX354A Connect-3 10Gb Dual Port Network Card+SAS, MCX354A-QCBT*136 | eBay
Or you can just buy a bracket with the expander on it (this one's an 8644): Full High Bracket with SAS Expander for CX354A MCX354A-QCBT MCX314A MCX354A | eBay

To be honest, I might get the PROs if I had it to do over, they have upgraded ASICs and are probably a better overall card. But they are 2-3x the price. Might as well get the ConnectX-4 LXs at that point, as they're only like $20-$40 more. You can flash the XCATs to ACAT (I think that's 4x 25GbE, if I'm not mistaken): https://forums.servethehome.com/index.php?threads/crossflash-hpe-640sfp28-to-mcx4121a-acat.28809/

There's a great article I just read on the CX4-LX card on servethehome, I'm pretty sure it's next on my list: https://www.servethehome.com/mellanox-connectx-4-lx-mini-review-ubiquitous-25gbe/

This dude said he flashed el cheapo CX3s with PRO (FCCT) firmware, but not sure if any improvement over FCBT with non-pro hardware (interesting that it works, though): https://forums.servethehome.com/index.php?threads/mellanox-connectx-4-re-flash-to-100gbe.19399/
 
Last edited:

xtantaudio

New Member
Apr 8, 2022
11
2
3
ESXi only really supports ethernet for infrastructure. You can use SR-IOV + PVRDMA for very specific VMs (I think only Rhel/CentOS and Windows supported). So it's basically a 40GbE adapter, no other options.

AveryFreeman,

I am attempting to do the same thing, picked up some connectX3 cards, installed them in my servers after flashing to the latest firmware.

I do have a few questions....I am taking from your statement above that ESXi doesnt support Infiniband after about version 6.X, and I am guessing that you switched the cards to Ethernet mode and connected them to a switch, correct ?

If that is the case, are you running an external QSFP switch for access to the rest of the network ? I had tried a breakout QSFP+ to SFP+ cable, but it doesnt seem to link up on my 10g switch for some reason..... however, it DID work on one of the 4 SFP+ ports before I flashed the new firmware.

I guess I just want to make sure I am on the right track before I go out and purchase another (ethernet) switch with QSFP ports.....
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
AveryFreeman,

I am attempting to do the same thing, picked up some connectX3 cards, installed them in my servers after flashing to the latest firmware.

I do have a few questions....I am taking from your statement above that ESXi doesnt support Infiniband after about version 6.X, and I am guessing that you switched the cards to Ethernet mode and connected them to a switch, correct ?

If that is the case, are you running an external QSFP switch for access to the rest of the network ? I had tried a breakout QSFP+ to SFP+ cable, but it doesnt seem to link up on my 10g switch for some reason..... however, it DID work on one of the 4 SFP+ ports before I flashed the new firmware.

I guess I just want to make sure I am on the right track before I go out and purchase another (ethernet) switch with QSFP ports.....
Hi, I'm not running a switch, I direct connected them in a star pattern. If you have 3x nodes, star pattern can give them all 1:1 access to one another. 3 would be the limit on such topography as far as I can tell.

If I were to run a switch, I'd keep it localized to the storage network, as I already had 10GbE + a switch for general LAN tasks when I set up the 40GbE. Was always intended as storage only (has dif subnet and is isolated and everything). There's no reason to "browse" the internet at 40GbE and storage network is best for host-to-host VM migration, etc.

I don't have any experience with breakout cables, but I have read they can be quirky (becoming active at different times, or other unexpectedness).
As far as the 10GbE working on SFP+ before flashing, maybe you were seeing that auto detect it was originally rated for (10GbE and 40GbIB). Anyway, the flashing was really easy and effective in my case, it was definitely easier than something like a SAS controller.

Good luck :)
 

DASHIP

New Member
May 4, 2016
15
0
1
54
Have y'all seen this VMWare article on using SR-IOV InfiniBand adapters with ESXi 7 or later?:

It's used for both cluster networking and storage, as well as vNIC's within the VM's for inter-VM networking, all at near bare metal speed. Pretty cool. With relatively cheap IB adapters and switches out there, this would make the hot setup for a lab cluster...

I also thought this article on setting up a PVRDMA cluster using 100G Infiniband with near bare metal performance (not quite as good as the above method, but pretty good) was interesting:

Here's a VMWare tech paper on the same:
 
Last edited:

kiranghag

New Member
Feb 4, 2024
1
0
1
lovely thread with almost everything needed on the topic.

one note of caution - if you are using SRIOV, the ESXi needs full memory allocation for the VMs. you cannot run VMs with memory more than available system ram.