Mellanox Connectx-3 w/ ESXi questions

AveryFreeman

VMXNET3 only supports >10G in Linux & Windows
Mar 17, 2017
241
21
18
39
Near Seattle
averyfreeman.com
Hey,

new to Mellanox cards, RDMA in general. Wasn't sure what really to expect with these, but knew they'd be faster than current 82599-based network

I've got 3 hypervisors and was going to use these for storage backend, sharing between the 3 probably gluster, or maybe 2 with drbd (still haven't decided - probably not ceph, not enough machines).

Anyway, ESXi 7.0 reports these vibs loaded:

Code:
nmlx4-core                     3.19.16.8-2vmw.701.0.0.16850804       VMW      VMwareCertified   2021-02-05
nmlx4-en                       3.19.16.8-2vmw.701.0.0.16850804       VMW      VMwareCertified   2021-02-05
nmlx4-rdma                     3.19.16.8-2vmw.701.0.0.16850804       VMW      VMwareCertified   2021-02-05
ntg3                           4.1.5.0-0vmw.701.0.0.16850804         VMW      VMwareCertified
Questions:

  1. ESXi says RoCE v1 possible but no RoCE v2 or nvme-of. Is RoCE v2 possible w/ connectx-3 cards, or am I hardware-limited to v1?
  2. I am using QSFP+ active fiber, and I flashed QCBT(10/40 en) cards to FCBT (40/46 en/ib), but if ESXi no longer supports IB does that mean I can't use the 56Gb IB unless I pass through the card to a VM? If I plan to use these through ESXi vNICs, should I have just left them as QCBTs (10/40Gb en)?
  3. Where do I find the FCBT 2.42 5000 firmware that comes in a TGZ file, so I can cross-flash? There's a link to 2.40 5030 in the forums, but I have no idea where to find the TGZ for latest version - I could only find the .bin file on the Mellanox website, and when I tried to sub 2.42 5000 for 2.40 5030 on link in forums, file not found and .htaccess index off on their server.
 

i386

Well-Known Member
Mar 18, 2016
2,319
640
113
31
Germany
ESXi says RoCE v1 possible but no RoCE v2 or nvme-of. Is RoCE v2 possible w/ connectx-3 cards, or am I hardware-limited to v1?
cx-3 only support roce v1, cx-3 pro and newer support roce v1 & v2
Where do I find the FCBT 2.42 5000 firmware that comes in a TGZ file, so I can cross-flash?
At some point mellanox stopped releasing the sources that allowed to create custom firmware. Use the binary from mellanox for flashing to the vanilla version.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

VMXNET3 only supports >10G in Linux & Windows
Mar 17, 2017
241
21
18
39
Near Seattle
averyfreeman.com
cx-3 only support roce v1, cx-3 pro and newer support roce v1 & v2

At some point mellanox stopped releasing the sources that allowed to create custom firmware. Use the binary from mellanox for flashing to the vanilla version.
Shit, re: RoCE v1/v2 I didn't see that in the datasheets, I knew I should have gotten the pros... welp, next time... I'm sure the cx-4s will be dirt cheap in a year or two, so I'll wait a bit and get those

Re: flashing w/ bin only - how do I crossflash, then? They're hardware QCBTs but I flashed them FCBT...
 

i386

Well-Known Member
Mar 18, 2016
2,319
640
113
31
Germany
I'm sure the cx-4s will be dirt cheap in a year or two
There are single port 100gbe cx-4 for ~$120 on ebay. It's "dirt cheap" for qsfp28/sfp28/current stuff :D
They're hardware QCBTs but I flashed them FCBT
It's the same asic, the firmware enables/disables certain speeds.
how do I crossflash, then?
See this post:
flint -d <device> -i mlnx_firmware.bin -allow_psid_change burn
 

AveryFreeman

VMXNET3 only supports >10G in Linux & Windows
Mar 17, 2017
241
21
18
39
Near Seattle
averyfreeman.com
There are single port 100gbe cx-4 for ~$120 on ebay. It's "dirt cheap" for qsfp28/sfp28/current stuff :D

It's the same asic, the firmware enables/disables certain speeds.

See this post:
Ah, the process I was following was using a .mlx file to compile a new .bin with a name specific to the OEM vendor. Let's see if I can find it...:

OK, this isn't the one I was reading, it's far more detailed. I'll have to go over this one.


This part is what I'm referring to, though:

Code:
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin
I'm glad they're fairly easy to work with. I guess one is an EMC and the other is a Mellanox, but they look exactly the same to my eye, I couldn't tell you which is which.
 

AveryFreeman

VMXNET3 only supports >10G in Linux & Windows
Mar 17, 2017
241
21
18
39
Near Seattle
averyfreeman.com
There are single port 100gbe cx-4 for ~$120 on ebay. It's "dirt cheap" for qsfp28/sfp28/current stuff :D

It's the same asic, the firmware enables/disables certain speeds.

See this post:
OK I went to that post, it's the same process I was following - requires the TGZ for custom firmware with the .mlx or .ini files. Looks like 2.40.5030 was the last release as detailed in thread from 3 years ago :/


NVM, I am just burning the 2.42 .bin file I downloaded from Mellanox without going through any of the other steps - looks like it's going to work ...

... yep, I flashed a QCBT to an FCBT, it says FW version is 2.42.5000 and the PSID went from 1090110018 to 1090110019...
 
Last edited: