Mellanox ConnectX-2 Firmware

Discussion in 'Networking' started by adam_kf, Apr 18, 2017.

  1. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    I ordered a few Mellanox ConnectX2 ethernet cards off ebay a while ago (MNPA19-XTR) and have been using them for straight ethernet for a while now and they work fine.

    With the release of Windows 2016, I'm playing with S2D and am trying to get these cards to work in an RDMA configuration. At the moment they detect as "RDMA capable":

    Code:
    PS C:\> Get-NetAdapterRdma
    
    Name                      InterfaceDescription                     Enabled
    ----                      --------------------                     -------
    RDMA1                     Mellanox ConnectX-2 Ethernet Adapter     True
    but they dont seem to show up as RDMA capable from an SMB perspective (requirement for RDMA on S2D):

    Code:
    PS C:\> Get-SmbServerNetworkInterface
    
    Scope Name                Interface Index RSS Capable RDMA Capable Speed   IpAddress
    ----------                --------------- ----------- ------------ -----   ---------
    <SNIP>
    FE80::DDF7:9F14:3280:5780 7               True        False        10 Gbps 10.91.255.2
    I believe this may be due to the firmware revision on the card (currently 2.9.1200) needing to be upgraded to at least 2.9.8350, however according to Mellanox's dirver page, the ConnectX2 EN only goes up to 2.9.1200 (my current rev).

    Does anyone know if these cards can be forcibly flashed with newer firmwares? I found this post on STH, but it seems to be specifically for ConnectX2 Infiniband cards.
    Custom Firmware for Mellanox OEM Infiniband Cards - WS2012 RDMA
     
    #1
  2. Marsh

    Marsh Moderator

    Joined:
    May 12, 2013
    Messages:
    1,800
    Likes Received:
    795
    I have many ConnectX-2 EN cards, currently running 10gb SPF+ EN cards in a S2D cluster.

    Yes, according my own notes:
    Oct-2016
    Flash MNPH29C-XTR cards to firmware 2.9.8350 for RDMA
    Flash MNPA19-XTR cards to firmware 2.9.8350 for RDMA
     
    #2
    T_Minus likes this.
  3. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,325
    Likes Received:
    302
    Weird the "newer" 2.9.8350 firmware is from decmebr 2011, the "older" 2.9.1200 from may 2012
     
    #3
  4. Marsh

    Marsh Moderator

    Joined:
    May 12, 2013
    Messages:
    1,800
    Likes Received:
    795
    Yep,
    I passed age 60, my memory is fading. So I write a lot of notes for myself to read later.
    Otherwise, I would have to spend at least an hour to performing the same experiment again.

    Version 2.9.8350 is the right one to flash,
    I remembered that I got the ~ $20 MNPH29C-XTR cards to work with RDMA in a WS2016 cluster, what a joy to see that happened.
     
    #4
  5. Dan K.

    Dan K. New Member

    Joined:
    Aug 30, 2016
    Messages:
    12
    Likes Received:
    0
    I just picked up a couple of the connectx-2s. Where can I download the 2.9.8350 FW?

    Thanks!
     
    #5
  6. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Well, I tried flashing today using 2.9.8350 with no luck. Looks like I've bricked two of the cards. Working on the "unbricking" back to 2.9.1200 now :)


    Code:
    PS C:\Software\fw-ConnectX2-rel-2_9_8350> mst status
    
    MST devices:
    ------------
      mt26448_pci_cr0
      mt26448_pciconf0
    Identified card correctly.

    Code:
    PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query
    
    Image type:          FS2
    FW Version:          2.9.1200
    Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
    Device ID:           26448
    Description:    Port1            Port2
    MACs:                    0002c952d9fe     0002c952d9ff
    VSD:
    PSID:                MT_0F60110010
    PSID = MT_0F60110010 (AKA Mellanox branded MNPA19-XTR)

    Code:
    PS C:\Software\fw-ConnectX2-rel-2_9_8350> mlxburn.exe -dev mt26448_pci_cr0 -fw fw-ConnectX2-rel.mlx
    
    -I- Querying device ...
    -I- Using auto detected configuration file: C:/Software/fw-ConnectX2-rel-2_9_8350/MNPA19_A1-A2.ini (PSID = MT_0F60110010)
    -I- Generating image ...
    
        Current FW version on flash:  2.9.1200
        New FW version:               2.9.8350
    
    Burning FS2 FW image without signatures - OK
    Restoring signature                     - OK
    -I- Image burn completed successfully.
    Firmware applies successfully to the card and re-query confirms this:

    Code:
    PS C:\Software\fw-ConnectX2-rel-2_9_8350> flint -d mt26448_pci_cr0 query
    
    Image type:          FS2
    FW Version:          2.9.8350
    FW Version(Running): 2.9.1200
    FW Release Date:     9.10.2011
    Rom Info:            type=PXE version=3.3.400 devid=26448 proto=VPI
    Device ID:           26448
    Description:    Port1            Port2
    MACs:                    0002c952d9fe     0002c952d9ff
    VSD:
    PSID:                MT_0F60110010
    After reboot the card does not show up at all in device manager.

    Any thoughts are greatly appreciated!
     
    #6
  7. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Dan: After much searching, I found a copy here. Here's a OneDrive link: fw-ConnectX2-rel-2_9_8350.zip

    If you get it working, let me know
     
    #7
  8. Dan K.

    Dan K. New Member

    Joined:
    Aug 30, 2016
    Messages:
    12
    Likes Received:
    0
    Awesome, thanks! I'll give it a shot tomorrow.
     
    #8
  9. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,325
    Likes Received:
    302
    Did you save the configuration file before flashing (flint -d /dev/mst/mt26448_pciconf0 dc)? You could try to build a custom firmware with this configuration and the melx file from the 2.9.8350 archive.
     
    #9
  10. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Successfully unbricked using the other device (mt26448_pciconf0) and the stock 2.9.1200 from the Mellanox website:

    Code:
    flint -d mt26448_pciconf0 -i fw-ConnectX2-rel-2_9_1200-MNPA19_A1-A3-FlexBoot-3.3.400.bin burn
    I'll give that a try and let you know.
     
    #10
    Last edited: Apr 19, 2017
  11. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    No luck unfortunately. Fimware flashes successfully but the NIC disappears after reboot, and starts reporting as an errored PCI device:

    Code:
    PS C:\> Get-PnpDevice -FriendlyName *mellanox*
    
    Status     Class           FriendlyName                                                                     InstanceId
    ------     -----           ------------                                                                     ----------
    Unknown    Net             Mellanox ConnectX-2 Ethernet Adapter                                             MLX4\CON...
    Error      System          Mellanox ConnectX EN (MT26448) - PCIe 2.0 5GT/s, 10GigE Ethernet Adapter         PCI\VEN_...
     
    #11
  12. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,325
    Likes Received:
    302
    #12
  13. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Argh, close but no cigar:

    Code:
    mlxburn.exe -dev mt26448_pci_cr0 -fw .\fw-ConnectX2-rel.mlx -conf .\MNPA19_A1-A2.ini
    -I- Generating image ...
        Current FW version on flash:  2.9.1200
        New FW version:               2.10.720
    -E- Image burn Failed: Failsafe operation failed: Bad FW image, configuration section should be placed in the end of the flash
    -E- Image burn failed: child process exited abnormally
    Looks to be some good info on that german page. Will dig deeper.
     
    #13
  14. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Success! Here's what i did (in case it helps anyone else):

    1) download the HP firmware image as suggested by i386 (Drivers & Software - HPE Support Center.)
    2) start the install and copy the folder containing "fw-ConnectX2-rel.mlx" from my %temp% folder
    3) Run the following commands to a) create a fw image and b) flash the controller:

    Code:
    PS C:\Software\{4986D353-357D-4518-AFFE-4A56D7D67417}> .\mlxburn.exe -fw .\fw-ConnectX2-rel.mlx -conf .\MNPA19_A1-A2.ini -wrimage firmware-image.bin
    
    PS C:\Software\{4986D353-357D-4518-AFFE-4A56D7D67417}> .\mstflint.exe -d mt26448_pci_cr0 -i .\firmware-image.bin b
    
        Current FW version on flash:  2.9.1200
        New FW version:               2.10.720
    
    Burning FW image without signatures  - OK
    Restoring signature                  - OK
    I can now see the adapter after reboot and it has the newer 2.10.720 firmware.

    One of the servers now correctly identifies that network adapter as "RDMA Capable":
    Code:
    PS C:\> Get-SmbServerNetworkInterface
    Scope Name              Interface Index RSS Capable RDMA Capable Speed   IpAddress
    ----------              --------------- ----------- ------------ -----   ---------
    *                       11              True        True         10 Gbps 10.91.255.2
    The other one does not, but i think that may be due to all the screwing around I've been doing with that machine. I'll re-image and see if she comes good!
     
    #14
    Legotan, DaddyGrant and Marsh like this.
  15. Dan K.

    Dan K. New Member

    Joined:
    Aug 30, 2016
    Messages:
    12
    Likes Received:
    0
    Your instructions worked perfectly! I'm now on 2.10.720.
     
    #15
  16. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    #16
  17. Dan K.

    Dan K. New Member

    Joined:
    Aug 30, 2016
    Messages:
    12
    Likes Received:
    0
    Have you tested the full throughput of the cards on Windows? I'm getting about 550-600MB/s receive and 650-700MB/s sending on bare metal Windows 10. On a Windows 10 VM on ESXi with vmxnet3 I get full 10G up and down. Still trying to work that out. I tried running the various perf tuning profiles on the performance tab...
     
    #17
  18. adam_kf

    adam_kf New Member

    Joined:
    Apr 17, 2017
    Messages:
    12
    Likes Received:
    3
    Still playing myself. When live migrating VM's I can saturate the link (full 10GBit). S2D is not spectacular but i've not tried optimizing too much yet... job for tomorrow.

    Does RDMA show as enabled with all the powershell listed in the first post?
     
    #18
  19. Dan K.

    Dan K. New Member

    Joined:
    Aug 30, 2016
    Messages:
    12
    Likes Received:
    0
    Nope it's not showing as enabled for me :\. I also reinstalled the 5.35 WinOF package after flashing the firmware. I should probably try a clean install. I'm also wondering if the Hyper-V adapter has anything to do with it.

    Code:
    PS C:\WINDOWS\system32> Get-NetAdapterRdma
    
    Name                      InterfaceDescription                     Enabled
    ----                      --------------------                     -------
    vEthernet (External Sw... Hyper-V Virtual Ethernet Adapter         False
    Ethernet 3                Mellanox ConnectX-2 Ethernet Adapter     True
    
    
    PS C:\WINDOWS\system32> Get-SmbServerNetworkInterface
    
    Scope Name Interface Index RSS Capable RDMA Capable Speed   IpAddress
    ---------- --------------- ----------- ------------ -----   ---------
    *          2               False       False        1 Gbps  192.168.1.6
    *          2               False       False        1 Gbps  169.254.237.228
    *          2               False       False        1 Gbps  fe80::f8fc:da41:ded4:ede4
    *          6               False       False        1 Gbps  169.254.186.111
    *          6               False       False        1 Gbps  fe80::c52c:c5f1:59fd:ba6f
    *          16              True        False        10 Gbps 2601:40b:0:f466:2cf5:5850:28
    *          16              True        False        10 Gbps 2601:40b:0:f466:4dec:f07c:60
    *          16              True        False        10 Gbps 192.168.20.3
    *          16              True        False        10 Gbps fe80::4dec:f07c:60d3:9ede
     
    #19
  20. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,325
    Likes Received:
    302
    To enable rdma you must run one of the mellanox cmdlets to change/enable roce v1 (newer winof versions have the default setting roce = disabled and older roce = roce v2, which was introduced with connectx-3 pro).
    Can't remeber the command but it's somewhere on their website.

    Btw I never got rdma running on windows clients (7, 10) with connectx-2 or connectx-3. Even following some microsoft tutorials which mention the connectx-3.
     
    #20
Similar Threads: Mellanox ConnectX-2
Forum Title Date
Networking 10gbit interface compatibility (Intel X553 <-> Mellanox ConnectX-2) Aug 12, 2018
Networking Mellanox ConnectX-2 VPI vs ConnectX-2? May 30, 2018
Networking Mellanox Connectx-2 temperature (and alternatives?) May 2, 2018
Networking Mellanox ConnectX-2 in SFP mode? Apr 8, 2018
Networking Question about Mellanox Connectx-2 VPI and Cables Mar 26, 2018

Share This Page