Mellanox ConnectX-2 Firmware

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
CentOS 7, I haven't tried on a Ubuntu 18.04 machine.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,519
2,695
113
31
fohdeesha.com
works great on ubuntu/debian, you just need to install the linux headers and required packages with:

Code:
apt-get install gcc make dkms unzip linux-headers-$(uname -r)
 
  • Like
Reactions: BackupProphet

BackupProphet

Well-Known Member
Jul 2, 2014
835
309
63
Stavanger, Norway
kingmakers.no
Thanks @fohdeesha That worked on Ubuntu 16.04, installing the Mellanox OFED DKMS drivers do take a while though :p

Anyway after that I used @arglebargle's GDrive link for 2.10.0720
Code:
olav@flasher:~/fw-ConnectX2-rel-2_10_0720$ sudo mlxburn -dev /dev/mst/mt26448_pci_cr0 -fw fw-ConnectX2-rel.mlx
-I- Querying device ...
-I- Using auto detected configuration file: ./test-MNPA19-XTR.ini (PSID = MT_0F60110010)
-I- Generating image ...

    Current FW version on flash:  2.9.1000
    New FW version:               2.10.0720

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
-I- Image burn completed successfully.



olav@flasher:~/fw-ConnectX2-rel-2_10_0720$ sudo flint -d /dev/mst/mt26448_pci_cr0 query
Image type:            FS2
FW Version:            2.10.0720
FW Version(Running):   2.9.1000
FW Release Date:       12.3.2012
Rom Info:              type=PXE version=3.3.400 proto=VPI
Device ID:             26448
Description:      Port1            Port2
MACs:                      0002c94f65c4     0002c94f65c5
VSD:
PSID:                  MT_0F60110010
But after the reboot now I see nothing. The OS see the Mellanox card though, just not getting any contact...

After reinstalling OFED drivers again, it was back!
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
Hey, glad it worked for you; I figured having more copies of that firmware mirrored was a good thing. I left a note in the directory with an outline of the process I used to update all of my CX2 cards too.
 
  • Like
Reactions: BackupProphet

BackupProphet

Well-Known Member
Jul 2, 2014
835
309
63
Stavanger, Norway
kingmakers.no
  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
I'm not sure if this applies to you but supposedly the C-X2s support SR-IOV as well. If you're hosting VMs on that machine you might want to switch it on in the firmware and try it out. I've been meaning to but I don't have a machine to test in besides my NAS until monday.

There's a post from a Mellanox hater with instructions buried in the forums somewhere.

Yeah thank you for that. It seems to work great, though, enabling RDMA in Ubuntu was a little more work with loading the necessary kernel modules.

src: Working with RDMA in Ubuntu - RDMAmojo
Code:
# RDMA stack modules
sudo modprobe rdma_cm
sudo modprobe rdma_ucm
Mellanox has examples here: HowTo Enable, Verify and Troubleshoot RDMA | Mellanox Interconnect Community

And finally I benchmarked with qperf linux-rdma/qperf
Awesome, what's the difference in CPU utilization between iperf and qperf like? I'm running one of my CX3s in a Win10 desktop that I intended to eventually install ESXi or Proxmox on (and take advantage of RDMA) but I haven't had the time yet and I'm curious.

If you're going to be using IB you might give some thought to picking up a pair of HP CX3s, you'll be able to use 56Gb FDR IB (with an FDR cable.) for heavily reduced overhead.
 
Last edited:

Lyubomir Traykov

New Member
Jul 23, 2018
1
0
1
Hello, Is someone having a problem with Hyper-V 2016 Switch? My card has the latest 2.10.720 firmware, and Mellanox OFED 5.50 driver. When the card is not used in a vSwitch everything is working fine including the RDMA, but when i add it to a vSwitch i get DUP pings on the host and the VMs. If i use the windows provided driver, the DUP pings stops but the RDMA is not working. I tried everything but no luck.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
Have you tried 2.9.8350? I had weird failures configuring SR-IOV with 2.10.0720 on one of my cards and ended up needing to revert to 2.9.8350 to get that one (MHQH29C-XTR) working. I had no problems using 2.10.0720 on a a single port card (MPNA19B?).
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
The 2.11.xx version is for Large BAR applications and requires that your BIOS support allocating a larger than normal amount of memory to each virtual function. If you rummage the forum for CX-2 firmware threads you'll find the original post that discusses it.

You can try using it, but if your BIOS doesn't allocate resources properly you could end up not being able to boot with the card installed. 2.9.8350 is a safer bet, it's much more widely tested.
 

ViciousXUSMC

Active Member
Nov 27, 2016
227
125
43
39
Just bought 3 single port Mellanox ConnectX-2 (MHRH19B-XTR)
Been looking up pretty much every thread I can find on the matter as well as downloading every resource I can find.

This thread seems the most up to date.

A few things I am wondering about before I get to work.
1.) People said you can extract the newest firmware out of the WinOF install by looking in your appdata folder. For me this was not true for atleast the newest version 5.50 drivers. I extracted all the files including the additional MSI files that got dumped into the appdata directory and found no .MLX files.
2.) The connect X2 was last listed as supported on legacy drivers back in the 4.x days, but there were reports of newer drivers (4.9x I think) having the .ini entries in there thus it has support but not listed. I am unsure if that still holds true for the newest 5.50 drivers.
3.) I seem to be one of the few with this single port version I assume the same firmware works on it as the dual port version?
4.) Should I even bother if I just want 10gb ethernet and will not be using any of the advanced features like RDMA?

I'll be placing one of these in a Dell R710 running ESXi 6.5

2nd Card will go in my Windows 10 PC

3rd Card will go into a R510 that will become my 2nd NAS, still not sure what will be running on it (Proxmox, ESXi, Baremetal Freenas, or XPC-ng)
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,519
2,695
113
31
fohdeesha.com
what they're talking about extracting from winof is the firmware sources to compile the driver yourself, and there's 0 reason to do that, just download the latest already compiled firmware from mellanox's site

the latest 5.x windows drivers work with the connetx2, however the latest linux drivers from mellanox dropped cx2 support completely - however no linux distros I know of include this recent of a driver
 

i386

Well-Known Member
Mar 18, 2016
3,097
992
113
33
Germany
and there's 0 reason to do that, just download the latest already compiled firmware from mellanox's site
Actually it's done to get a "newer" firmware: that special package from hpe contains the firmware 2.10.x instead of the 2.9.x that's available from mellanox. (That 2.10.x firmware is/was necessary to enable rdma on the connect-x 2 ethernet nics.)
 

ViciousXUSMC

Active Member
Nov 27, 2016
227
125
43
39
Patrick has something that I downloaded, extracts from 5.4 and I'll use that. However there is a 5.5 now so not sure if drivers/firmware are newer.

Also the package I do not know if it works on all models or I need to extract manually the specific firmware I need for my cards.

Still a bit more research to be done.
 

TeleFragger

Member
Oct 26, 2016
152
20
18
49
Since this thread is ConnectX-2 cards.. I bought a CX4 VPI port style (if you read my HP ConnectX-1 thread)...
Those cards I cant get past HP fw 2.8 but these I was able to get to 2.10.. If anyone wants to know how...let me know.. but will take some senior remembering....
for me the weird part is Windows 10 still sees it as a ConnectX-1 card...

upload_2019-2-7_21-17-24.png
 

Attachments