Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

Churchill

Admiral
Jan 6, 2016
838
213
43
I did what @kiteboarder did on 3 cards and worked like a champ through the process.

My only question is how can I truly test the cards to see if they will appear as full Ethernet devices?
 
  • Like
Reactions: kiteboarder
M

Milos

New Member
Feb 22, 2018
25
2
3
Black hole with internet
kiteboarder said:
These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.
Click to expand...


Flashing is done , 3 cards all is ok thanks a lot.
 
  • Like
Reactions: kiteboarder
C

cuco

Member
Feb 13, 2018
57
4
8
39
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
 
M

mb300sd

Active Member
Aug 1, 2016
204
80
28
cuco said:
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
Click to expand...
Looks like MCX353A-FCBT to me. Change to ethernet mode in device manager (click the network adapter, View->Devices by connection, open the properties for the device 1 up from it)
 
C

cuco

Member
Feb 13, 2018
57
4
8
39
Ok this seems to have worked. I uses mighty fohdeesha's example.
Card is set to ethernet mode as well. I use the WinOF 5.50.51 drivers. But there is now WakeOnLan option.
When i read right 5.50.51 does not support WOL.
I tried to use the 5.35 with the latest firmware. but this does not seem to work. the system doesnt use the driver.
I am running on Server2019 and their only is a server 2016 5.35 driver. Any idea how to get WOL to work? Do i have to downgrade the firmware?
 
K

KC8FLB

Member
Aug 12, 2018
74
58
18
kiteboarder said:
These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.
Click to expand...
Thank you, worked wonderfully on a Win 10 box.
 
  • Like
Reactions: kiteboarder
S

saverio

New Member
Jun 24, 2020
9
1
3
Hello,
I have flashed a CX354A-QCBT to -FCBT using the guide in this thread. I tried to cross connect the two ports using an FS.com QSFP DAC, but the link is 10Gb, not 40. If I try to set it manually to 40000 with ethtool, it doesn't go up.


Could it be a cable issue?

Code: 
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:02:00.0
  Port1 MAC:        506b4b7cf4b1
  Port2 MAC:        506b4b7cf4b2
  Versions:         Current        Available
     FW             2.42.5000      2.42.5000
     PXE            3.4.0752       3.4.0752

  Status:           Up to date
Code: 
ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes
 
F

Freebsd1976

Active Member
Feb 23, 2018
404
73
28
saverio said:
Hello,
I have flashed a CX354A-QCBT to -FCBT using the guide in this thread. I tried to cross connect the two ports using an FS.com QSFP DAC, but the link is 10Gb, not 40. If I try to set it manually to 40000 with ethtool, it doesn't go up.


Could it be a cable issue?

Code: 
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:02:00.0
  Port1 MAC:        506b4b7cf4b1
  Port2 MAC:        506b4b7cf4b2
  Versions:         Current        Available
     FW             2.42.5000      2.42.5000
     PXE            3.4.0752       3.4.0752

  Status:           Up to date
Code: 
ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes
Click to expand...
Maybe or may not be , I have flash 5 qcbt to fcbt .
1 one of them only one port can work on 40/56 ,the second port work on 10g.
2. the other four card work as normal fcbt .
 
R

Roelf Zomerman

Active Member
Jan 10, 2019
149
28
28
blog.azureinfra.com
and for the Windows Users: (very similar to the opening post:)

#Download Tools: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_15_0_104.exe
#Download Firmware: http://www.mellanox.com/downloads/f...00-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Install the tools, then go to
CD .\Program Files\Mellanox\WinMFT

#Query the device name (use the 2nd name returned looking like: mt4099_pciconf0)
mdevices status

#Backup old firmware
flint -d mt4099_pciconf0 dc orig_fw.ini

#Burn new firmware
flint -d mt4099_pciconf0 -i c:\<folder>\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

#disable boot stuff
mlxconfig -d mt4099_pciconf0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d mt4099_pciconf0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d mt4099_pciconf0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d mt4099_pciconf0 set LEGACY_BOOT_PROTOCOL_P2=0

#remove the boot ROM
flint -d mt4099_pciconf0 --allow_rom_change drom

#check current settings
mlxconfig -d mt4099_pciconf0 query

#Reboot Machine
 
  • Like
Reactions: sadamahan, nedimzukic2, Luzer and 1 other person
D

dpsi

New Member
Dec 14, 2015
3
0
1
Just wanted to mention that I was able to do this succesfully with a Revision A6 649281-B21 using the A2-A5 firmware.
 
T

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Hi, I was able to flash two MCX354A-QCBT to MCX354A-FCBT firmware, but I seem to be getting just ~12Gbps speed when running ipref tests between a Linux NAS and Windows 10 client even though I confirmed that the firmware were flashed correctly, and the OSs reports 40Gbps link speed on both hosts. I'm using Mellanox MC2207130-001 cable that supports 40GbE. Does the cross-flashing actually not unlock 40GbE? Am I CPU-bottlenecked (the NAS is running an embedded CPU)? What could I be doing wrong?

Thanks a lot for your help,

ipref on the Linux NAS:

Code: 
-----------------------------------------------------------
Server listening on 1234
-----------------------------------------------------------
Accepted connection from 192.168.101.100, port 60195
[  5] local 192.168.101.1 port 1234 connected to 192.168.101.100 port 60197
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.27 GBytes  10.9 Gbits/sec
[  5]   1.00-2.00   sec  1.44 GBytes  12.3 Gbits/sec
[  5]   2.00-3.00   sec  1.45 GBytes  12.5 Gbits/sec
[  5]   3.00-4.00   sec  1.49 GBytes  12.8 Gbits/sec
[  5]   4.00-5.00   sec  1.43 GBytes  12.3 Gbits/sec
[  5]   5.00-6.00   sec  1.48 GBytes  12.7 Gbits/sec
[  5]   6.00-7.00   sec  1.42 GBytes  12.2 Gbits/sec
[  5]   7.00-8.00   sec  1.39 GBytes  11.9 Gbits/sec
[  5]   8.00-9.00   sec  1.33 GBytes  11.4 Gbits/sec
[  5]   9.00-10.00  sec  1.30 GBytes  11.2 Gbits/sec
[  5]  10.00-10.03  sec  39.9 MBytes  10.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.03  sec  14.0 GBytes  12.0 Gbits/sec                  receiver
Code: 
> .\iperf3.exe -c 192.168.101.1 -p 1234 -d
send_parameters:
{
        "tcp":  true,
        "omit": 0,
        "time": 10,
        "parallel":     1,
        "len":  131072,
        "client_version":       "3.1.3"
}
Connecting to host 192.168.101.1, port 1234
SO_SNDBUF is 212992
[  4] local 192.168.101.100 port 60197 connected to 192.168.101.1 port 1234
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec
[  4]   1.00-2.00   sec  1.45 GBytes  12.5 Gbits/sec
[  4]   2.00-3.00   sec  1.46 GBytes  12.5 Gbits/sec
[  4]   3.00-4.00   sec  1.48 GBytes  12.7 Gbits/sec
[  4]   4.00-5.00   sec  1.44 GBytes  12.3 Gbits/sec
[  4]   5.00-6.00   sec  1.48 GBytes  12.7 Gbits/sec
[  4]   6.00-7.00   sec  1.42 GBytes  12.2 Gbits/sec
[  4]   7.00-8.00   sec  1.37 GBytes  11.8 Gbits/sec
[  4]   8.00-9.00   sec  1.34 GBytes  11.5 Gbits/sec
send_results
{
        "cpu_util_total":       83.620775,
        "cpu_util_user":        34.689192,
        "cpu_util_system":      48.931583,
        "sender_has_retransmits":       0,
        "streams":      [{
                        "id":   1,
                        "bytes":        15074459648,
                        "retransmits":  -1,
                        "jitter":       0,
                        "errors":       0,
                        "packets":      0
                }]
}
get_results
{
        "cpu_util_total":       2.208995,
        "cpu_util_user":        0.031556,
        "cpu_util_system":      2.177375,
        "sender_has_retransmits":       -1,
        "streams":      [{
                        "id":   1,
                        "bytes":        15074459648,
                        "retransmits":  -1,
                        "jitter":       0,
                        "errors":       0,
                        "packets":      0
                }]
}
[  4]   9.00-10.00  sec  1.30 GBytes  11.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  14.0 GBytes  12.1 Gbits/sec                  sender
[  4]   0.00-10.00  sec  14.0 GBytes  12.1 Gbits/sec                  receiver

iperf Done.

Linux NAS:
Code: 
# mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:07:00.0
  Port1 MAC:        0002c9a595a1
  Port2 MAC:        0002c9a595a2
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found
Code: 
# ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no
Windows 10 client:
Code: 
> mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        0002c9e78d51
  Port2 MAC:        0002c9e78d52
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found
Code: 
> Get-NetAdapter | select interfaceDescription, name, status, linkSpeed

interfaceDescription                     name            Status      LinkSpeed
--------------------                     ----            ------      ---------
Mellanox ConnectX-3 Ethernet Adapter #4  Ethernet 11     Up          40 Gbps
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,572
1,025
113
I can confirm that the speed of crossflashed (Oracle branded QCBT -> FCBT) CX3's is not capped, so likely another issue with your system.

What's the product number of your CPU?

What are your other network settings (e.g. MTU)?
 
K

kiteboarder

Active Member
May 10, 2016
101
48
28
45
Just a guess, you may have one card in a system's pci-e slot that's only gen 2 x 4 lanes which would net you a max of 16 Gb/s even though the 40Gb link is active.
 
T

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Thanks a lot for the note.

Re: network settings:
I started looking into MTU settings, they're all default at 1500. Learning how to change them and will report back how things are. I assume I want them to be like 9000? I also changed windows driver to tune for single port traffic and that seems to have added 1-2Gbps

Re: CPU
Windows client: Intel Core i7-6700K
Linux NAS (Synology DS1621+): AMD Ryzen V1500B

Re: PCI-E
Windows client: PCI-E 3.0 x8
Linux NAS (Synology DS1621+): PCI-E 3.0 x4

Doing some napkin math, PCI-E 3.0 x4 should at least support one port at 40Gbps?
 
K

kiteboarder

Active Member
May 10, 2016
101
48
28
45
PCI-E gen 3 is 8 Gb/s per lane, so right there you're down to 32 Gb/s max theoretical. And even then it won't be that high due to a little bit of coding overhead.

I don't recommend changing the MTU unless it's a direct computer to computer link.

I also recommend being honest with what your NAS can do read/write wise. Unless it can really do more than 16 Gb/s (2 GB/s) then don't worry about your link performance - it's good enough. ;-)
 
  • Like
Reactions: P0rt4lN3T
T

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Thank you for the note.

kiteboarder said:
I don't recommend changing the MTU unless it's a direct computer to computer link.
Click to expand...
I'm doing a direct link right now but plan to move to a switch at some point. I'm curious how MTU matters for computer/computer vs. through a switch?

kiteboarder said:
I also recommend being honest with what your NAS can do read/write wise. Unless it can really do more than 16 Gb/s (2 GB/s) then don't worry about your link performance - it's good enough. ;-)
Click to expand...
My plan was to add NVMe SSDs as caches in the NAS and hoping the caches will make it comparable to at least a SATA SSD. This way, I can just use the NAS as if it's an internal drive with drive redundancy built-in. I'm running ipref to ensure the network won't be a bottleneck since the NVMe SSDs have 2.0GB/s+ sequential performance.
 
I

i386

Well-Known Member
Mar 18, 2016
4,387
1,622
113
35
Germany
IPerf on windows is not the same as iperf
tryingtorunservers said:
but I seem to be getting just ~12Gbps speed when running ipref tests between a Linux NAS and Windows 10 client
Click to expand...
That's because the windows implementation of iperf is not optimized.
Boot linux from a live cd/usb on the windows host and try iperf or boot windows on both machines and use ntttcp for network testing.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
cuco said:
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
Click to expand...
Did you manage to convert the Fujitsu?
 
You must log in or register to reply here.
Top Bottom