Mellanox ConnectX-2 Firmware

adam_kf

New Member
Apr 17, 2017
17
5
3
36
I ordered a few Mellanox ConnectX2 ethernet cards off ebay a while ago (MNPA19-XTR) and have been using them for straight ethernet for a while now and they work fine.

With the release of Windows 2016, I'm playing with S2D and am trying to get these cards to work in an RDMA configuration. At the moment they detect as "RDMA capable":

Code:
PS C:\> Get-NetAdapterRdma

Name                      InterfaceDescription                     Enabled
----                      --------------------                     -------
RDMA1                     Mellanox ConnectX-2 Ethernet Adapter     True
but they dont seem to show up as RDMA capable from an SMB perspective (requirement for RDMA on S2D):

Code:
PS C:\> Get-SmbServerNetworkInterface

Scope Name                Interface Index RSS Capable RDMA Capable Speed   IpAddress
----------                --------------- ----------- ------------ -----   ---------
<SNIP>
FE80::DDF7:9F14:3280:5780 7               True        False        10 Gbps 10.91.255.2
I believe this may be due to the firmware revision on the card (currently 2.9.1200) needing to be upgraded to at least 2.9.8350, however according to Mellanox's dirver page, the ConnectX2 EN only goes up to 2.9.1200 (my current rev).

Does anyone know if these cards can be forcibly flashed with newer firmwares? I found this post on STH, but it seems to be specifically for ConnectX2 Infiniband cards.
Custom Firmware for Mellanox OEM Infiniband Cards - WS2012 RDMA
 

Marsh

Moderator
May 12, 2013
2,481
1,301
113
I have many ConnectX-2 EN cards, currently running 10gb SPF+ EN cards in a S2D cluster.

Yes, according my own notes:
Oct-2016
Flash MNPH29C-XTR cards to firmware 2.9.8350 for RDMA
Flash MNPA19-XTR cards to firmware 2.9.8350 for RDMA
 
  • Like
Reactions: T_Minus

i386

Well-Known Member
Mar 18, 2016
2,677
794
113
32
Germany
Weird the "newer" 2.9.8350 firmware is from decmebr 2011, the "older" 2.9.1200 from may 2012
 

Marsh

Moderator
May 12, 2013
2,481
1,301
113
Yep,
I passed age 60, my memory is fading. So I write a lot of notes for myself to read later.
Otherwise, I would have to spend at least an hour to performing the same experiment again.

Version 2.9.8350 is the right one to flash,
I remembered that I got the ~ $20 MNPH29C-XTR cards to work with RDMA in a WS2016 cluster, what a joy to see that happened.
 

Dan K.

New Member
Aug 30, 2016
14
0
1
35
I just picked up a couple of the connectx-2s. Where can I download the 2.9.8350 FW?

Thanks!
 

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Well, I tried flashing today using 2.9.8350 with no luck. Looks like I've bricked two of the cards. Working on the "unbricking" back to 2.9.1200 now :)


Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> mst status

MST devices:
------------
  mt26448_pci_cr0
  mt26448_pciconf0
Identified card correctly.

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query

Image type:          FS2
FW Version:          2.9.1200
Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
Device ID:           26448
Description:    Port1            Port2
MACs:                    0002c952d9fe     0002c952d9ff
VSD:
PSID:                MT_0F60110010
PSID = MT_0F60110010 (AKA Mellanox branded MNPA19-XTR)

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> mlxburn.exe -dev mt26448_pci_cr0 -fw fw-ConnectX2-rel.mlx

-I- Querying device ...
-I- Using auto detected configuration file: C:/Software/fw-ConnectX2-rel-2_9_8350/MNPA19_A1-A2.ini (PSID = MT_0F60110010)
-I- Generating image ...

    Current FW version on flash:  2.9.1200
    New FW version:               2.9.8350

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
-I- Image burn completed successfully.
Firmware applies successfully to the card and re-query confirms this:

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query

Image type:          FS2
FW Version:          2.9.8350
FW Version(Running): 2.9.1200
FW Release Date:     9.10.2011
Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
Device ID:           26448
Description:    Port1            Port2
MACs:                    0002c952d9fe     0002c952d9ff
VSD:
PSID:                MT_0F60110010
After reboot the card does not show up at all in device manager.

Any thoughts are greatly appreciated!
 

i386

Well-Known Member
Mar 18, 2016
2,677
794
113
32
Germany
Well, I tried flashing today using 2.9.8350 with no luck. Looks like I've bricked two of the cards. Working on the "unbricking" back to 2.9.1200 now :)


Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> mst status

MST devices:
------------
  mt26448_pci_cr0
  mt26448_pciconf0
Identified card correctly.

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query

Image type:          FS2
FW Version:          2.9.1200
Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
Device ID:           26448
Description:    Port1            Port2
MACs:                    0002c952d9fe     0002c952d9ff
VSD:
PSID:                MT_0F60110010
PSID = MT_0F60110010 (AKA Mellanox branded MNPA19-XTR)

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> mlxburn.exe -dev mt26448_pci_cr0 -fw fw-ConnectX2-rel.mlx

-I- Querying device ...
-I- Using auto detected configuration file: C:/Software/fw-ConnectX2-rel-2_9_8350/MNPA19_A1-A2.ini (PSID = MT_0F60110010)
-I- Generating image ...

    Current FW version on flash:  2.9.1200
    New FW version:               2.9.8350

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
-I- Image burn completed successfully.
Firmware applies successfully to the card and re-query confirms this:

Code:
PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query

Image type:          FS2
FW Version:          2.9.8350
FW Version(Running): 2.9.1200
FW Release Date:     9.10.2011
Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
Device ID:           26448
Description:    Port1            Port2
MACs:                    0002c952d9fe     0002c952d9ff
VSD:
PSID:                MT_0F60110010
After reboot the card does not show up at all in device manager.

Any thoughts are greatly appreciated!
Did you save the configuration file before flashing (flint -d /dev/mst/mt26448_pciconf0 dc)? You could try to build a custom firmware with this configuration and the melx file from the 2.9.8350 archive.
 

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Successfully unbricked using the other device (mt26448_pciconf0) and the stock 2.9.1200 from the Mellanox website:

Code:
flint -d mt26448_pciconf0 -i fw-ConnectX2-rel-2_9_1200-MNPA19_A1-A3-FlexBoot-3.3.400.bin burn
Did you save the configuration file before flashing (flint -d /dev/mst/mt26448_pciconf0 dc)? You could try to build a custom firmware with this configuration and the melx file from the 2.9.8350 archive.
I'll give that a try and let you know.
 
Last edited:

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Did you save the configuration file before flashing (flint -d /dev/mst/mt26448_pciconf0 dc)? You could try to build a custom firmware with this configuration and the melx file from the 2.9.8350 archive.
No luck unfortunately. Fimware flashes successfully but the NIC disappears after reboot, and starts reporting as an errored PCI device:

Code:
PS C:\> Get-PnpDevice -FriendlyName *mellanox*

Status     Class           FriendlyName                                                                     InstanceId
------     -----           ------------                                                                     ----------
Unknown    Net             Mellanox ConnectX-2 Ethernet Adapter                                             MLX4\CON...
Error      System          Mellanox ConnectX EN (MT26448) - PCIe 2.0 5GT/s, 10GigE Ethernet Adapter         PCI\VEN_...
 

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Argh, close but no cigar:

Code:
mlxburn.exe -dev mt26448_pci_cr0 -fw .\fw-ConnectX2-rel.mlx -conf .\MNPA19_A1-A2.ini
-I- Generating image ...
    Current FW version on flash:  2.9.1200
    New FW version:               2.10.720
-E- Image burn Failed: Failsafe operation failed: Bad FW image, configuration section should be placed in the end of the flash
-E- Image burn failed: child process exited abnormally
Looks to be some good info on that german page. Will dig deeper.
 

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Success! Here's what i did (in case it helps anyone else):

1) download the HP firmware image as suggested by i386 (Drivers & Software - HPE Support Center.)
2) start the install and copy the folder containing "fw-ConnectX2-rel.mlx" from my %temp% folder
3) Run the following commands to a) create a fw image and b) flash the controller:

Code:
PS C:\Software\{4986D353-357D-4518-AFFE-4A56D7D67417}> .\mlxburn.exe -fw .\fw-ConnectX2-rel.mlx -conf .\MNPA19_A1-A2.ini -wrimage firmware-image.bin

PS C:\Software\{4986D353-357D-4518-AFFE-4A56D7D67417}> .\mstflint.exe -d mt26448_pci_cr0 -i .\firmware-image.bin b

    Current FW version on flash:  2.9.1200
    New FW version:               2.10.720

Burning FW image without signatures  - OK
Restoring signature                  - OK
I can now see the adapter after reboot and it has the newer 2.10.720 firmware.

One of the servers now correctly identifies that network adapter as "RDMA Capable":
Code:
PS C:\> Get-SmbServerNetworkInterface
Scope Name              Interface Index RSS Capable RDMA Capable Speed   IpAddress
----------              --------------- ----------- ------------ -----   ---------
*                       11              True        True         10 Gbps 10.91.255.2
The other one does not, but i think that may be due to all the screwing around I've been doing with that machine. I'll re-image and see if she comes good!
 

Dan K.

New Member
Aug 30, 2016
14
0
1
35
Have you tested the full throughput of the cards on Windows? I'm getting about 550-600MB/s receive and 650-700MB/s sending on bare metal Windows 10. On a Windows 10 VM on ESXi with vmxnet3 I get full 10G up and down. Still trying to work that out. I tried running the various perf tuning profiles on the performance tab...
 

adam_kf

New Member
Apr 17, 2017
17
5
3
36
Still playing myself. When live migrating VM's I can saturate the link (full 10GBit). S2D is not spectacular but i've not tried optimizing too much yet... job for tomorrow.

Does RDMA show as enabled with all the powershell listed in the first post?
 

Dan K.

New Member
Aug 30, 2016
14
0
1
35
Nope it's not showing as enabled for me :\. I also reinstalled the 5.35 WinOF package after flashing the firmware. I should probably try a clean install. I'm also wondering if the Hyper-V adapter has anything to do with it.

Code:
PS C:\WINDOWS\system32> Get-NetAdapterRdma

Name                      InterfaceDescription                     Enabled
----                      --------------------                     -------
vEthernet (External Sw... Hyper-V Virtual Ethernet Adapter         False
Ethernet 3                Mellanox ConnectX-2 Ethernet Adapter     True


PS C:\WINDOWS\system32> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed   IpAddress
---------- --------------- ----------- ------------ -----   ---------
*          2               False       False        1 Gbps  192.168.1.6
*          2               False       False        1 Gbps  169.254.237.228
*          2               False       False        1 Gbps  fe80::f8fc:da41:ded4:ede4
*          6               False       False        1 Gbps  169.254.186.111
*          6               False       False        1 Gbps  fe80::c52c:c5f1:59fd:ba6f
*          16              True        False        10 Gbps 2601:40b:0:f466:2cf5:5850:28
*          16              True        False        10 Gbps 2601:40b:0:f466:4dec:f07c:60
*          16              True        False        10 Gbps 192.168.20.3
*          16              True        False        10 Gbps fe80::4dec:f07c:60d3:9ede
 

i386

Well-Known Member
Mar 18, 2016
2,677
794
113
32
Germany
To enable rdma you must run one of the mellanox cmdlets to change/enable roce v1 (newer winof versions have the default setting roce = disabled and older roce = roce v2, which was introduced with connectx-3 pro).
Can't remeber the command but it's somewhere on their website.

Btw I never got rdma running on windows clients (7, 10) with connectx-2 or connectx-3. Even following some microsoft tutorials which mention the connectx-3.