Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

dmerillat

New Member
Jan 8, 2022
12
3
3
DaveLTX said:
Like I said, its the wrong board. 283-B21 and onwards are Rev C (meaning C1-C5 etc) and there exists no mellanox firmware for that.
Any Rev A firmware does not work. I tried myself and the only way is to dump the firmware on them and extract them then modify the ini but I haven't been able to do that myself. Maybe someone can chime in

On the flipside, they dont need flashing like the 281 cards to enable 40G, they already have it enabled
Click to expand...
The 649283 is the 10g card, it has to be cross-flashed to a 649282 for 40g.

Code: 
[PS_INFO]
Name = 649283-B21_Cx-Dx
Description = HP IB QDR/EN 10Gb 2P 544FLR-QSFP Adpr

[PS_INFO]
Name = 649282-B21_MS_Cx-Dx
Description = HP IB FDR/EN 10/40Gb 2P 544FLR-QSFP Adpr
I extracted the .ini (flint -d xxx dc > hp.ini) from the HP firmware and put it in the stock mellanox firmware with the name and description fields taken from MXC354A-FCB_A2-A5.ini. I just used the mlxburn command from the 10 and 40gb networking on the cheap thread to build it.

The 'netlink: invalid argument' error is a bug in ethtool that I'll report there. The KAIAM XQX2502 doesn't have page 3. There's comments in the ethtool code about there being no way to know if page3 exists without trying, the pretty formatter knows to skip it, but the eeprom reading code treats a missing page 3 as a fatal error and bails before displaying it. Making it a soft-failure and setting the page3 pointer to null gives me:
Code: 
    Identifier                                : 0x0d (QSFP+)
    Extended identifier                       : 0x80
    Extended identifier description           : 2.5W max. Power consumption
    Extended identifier description           : No CDR in TX, No CDR in RX
    Extended identifier description           : High Power Class (> 3.5 W) not enabled
    Power set                                 : Off
    Power override                            : On
    Connector                                 : 0x07 (LC)
    Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 40G Ethernet: 40G Base-LR4
    Encoding                                  : 0x00 (unspecified)
    BR, Nominal                               : 10300Mbps
    Rate identifier                           : 0x00
    Length (SMF,km)                           : 2km
    Length (OM3 50um)                         : 0m
    Length (OM2 50um)                         : 0m
    Length (OM1 62.5um)                       : 0m
    Length (Copper or Active cable)           : 0m
    Transmitter technology                    : 0x40 (1310 nm DFB)
    Laser wavelength                          : 1310.000nm
    Laser wavelength tolerance                : 6.500nm
    Vendor name                               : KAIAM CORP
    Vendor OUI                                : 14:ed:e4
    Vendor PN                                 : XQX2502
    Vendor rev                                : 1A
    Vendor SN                                 : KD60208104
    Date code                                 : 16020800
    Revision Compliance                       : SFF-8636 Rev 1.5
    Rx loss of signal                         : None
    Tx loss of signal                         : None
    Module temperature                        : 56.20 degrees C / 133.15 degrees F
    Module voltage                            : 3.3107 V
    Alarm/warning flags implemented           : No
    Laser tx bias current (Channel 1)         : 51.420 mA
    Laser tx bias current (Channel 2)         : 53.596 mA
    Laser tx bias current (Channel 3)         : 45.104 mA
    Laser tx bias current (Channel 4)         : 51.372 mA
    Transmit avg optical power (Channel 1)    : 1.2115 mW / 0.83 dBm
    Transmit avg optical power (Channel 2)    : 0.8790 mW / -0.56 dBm
    Transmit avg optical power (Channel 3)    : 1.2384 mW / 0.93 dBm
    Transmit avg optical power (Channel 4)    : 1.2063 mW / 0.81 dBm
    Rcvr signal avg optical power(Channel 1)  : 0.6477 mW / -1.89 dBm
    Rcvr signal avg optical power(Channel 2)  : 0.8806 mW / -0.55 dBm
    Rcvr signal avg optical power(Channel 3)  : 0.7671 mW / -1.15 dBm
    Rcvr signal avg optical power(Channel 4)  : 0.6694 mW / -1.74 dBm
That's spicy hot. Testbench doesn't have much airflow, real server does at least. Link is up and traffic is flowing.

Now that it's identified as an ethtool bug, I put them back to HP firmware and it's refusing to turn on TX when the link is brought up:
Code: 
        Transmit avg optical power (Channel 1)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 2)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.0000 mW / -inf dBm
    Rcvr signal avg optical power(Channel 1)  : 0.5985 mW / -2.23 dBm
    Rcvr signal avg optical power(Channel 2)  : 0.8011 mW / -0.96 dBm
    Rcvr signal avg optical power(Channel 3)  : 0.6857 mW / -1.64 dBm
    Rcvr signal avg optical power(Channel 4)  : 0.6239 mW / -2.05 dBm
That could be due to the power restriction in the HP firmware, but the board itself works with stock firmware now.
 
Last edited:
  • Like
Reactions: blunden
D

DaveLTX

Active Member
Dec 5, 2021
185
49
28
dmerillat said:
The 649283 is the 10g card, it has to be cross-flashed to a 649282 for 40g.

Code: 
[PS_INFO]
Name = 649283-B21_Cx-Dx
Description = HP IB QDR/EN 10Gb 2P 544FLR-QSFP Adpr

[PS_INFO]
Name = 649282-B21_MS_Cx-Dx
Description = HP IB FDR/EN 10/40Gb 2P 544FLR-QSFP Adpr
I extracted the .ini (flint -d xxx dc > hp.ini) from the HP firmware and put it in the stock mellanox firmware with the name and description fields taken from MXC354A-FCB_A2-A5.ini. I just used the mlxburn command from the 10 and 40gb networking on the cheap thread to build it.

The 'netlink: invalid argument' error is a bug in ethtool that I'll report there. The KAIAM XQX2502 doesn't have page 3. There's comments in the ethtool code about there being no way to know if page3 exists without trying, the pretty formatter knows to skip it, but the eeprom reading code treats a missing page 3 as a fatal error and bails before displaying it. Making it a soft-failure and setting the page3 pointer to null gives me:
Code: 
    Identifier                                : 0x0d (QSFP+)
    Extended identifier                       : 0x80
    Extended identifier description           : 2.5W max. Power consumption
    Extended identifier description           : No CDR in TX, No CDR in RX
    Extended identifier description           : High Power Class (> 3.5 W) not enabled
    Power set                                 : Off
    Power override                            : On
    Connector                                 : 0x07 (LC)
    Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 40G Ethernet: 40G Base-LR4
    Encoding                                  : 0x00 (unspecified)
    BR, Nominal                               : 10300Mbps
    Rate identifier                           : 0x00
    Length (SMF,km)                           : 2km
    Length (OM3 50um)                         : 0m
    Length (OM2 50um)                         : 0m
    Length (OM1 62.5um)                       : 0m
    Length (Copper or Active cable)           : 0m
    Transmitter technology                    : 0x40 (1310 nm DFB)
    Laser wavelength                          : 1310.000nm
    Laser wavelength tolerance                : 6.500nm
    Vendor name                               : KAIAM CORP
    Vendor OUI                                : 14:ed:e4
    Vendor PN                                 : XQX2502
    Vendor rev                                : 1A
    Vendor SN                                 : KD60208104
    Date code                                 : 16020800
    Revision Compliance                       : SFF-8636 Rev 1.5
    Rx loss of signal                         : None
    Tx loss of signal                         : None
    Module temperature                        : 56.20 degrees C / 133.15 degrees F
    Module voltage                            : 3.3107 V
    Alarm/warning flags implemented           : No
    Laser tx bias current (Channel 1)         : 51.420 mA
    Laser tx bias current (Channel 2)         : 53.596 mA
    Laser tx bias current (Channel 3)         : 45.104 mA
    Laser tx bias current (Channel 4)         : 51.372 mA
    Transmit avg optical power (Channel 1)    : 1.2115 mW / 0.83 dBm
    Transmit avg optical power (Channel 2)    : 0.8790 mW / -0.56 dBm
    Transmit avg optical power (Channel 3)    : 1.2384 mW / 0.93 dBm
    Transmit avg optical power (Channel 4)    : 1.2063 mW / 0.81 dBm
    Rcvr signal avg optical power(Channel 1)  : 0.6477 mW / -1.89 dBm
    Rcvr signal avg optical power(Channel 2)  : 0.8806 mW / -0.55 dBm
    Rcvr signal avg optical power(Channel 3)  : 0.7671 mW / -1.15 dBm
    Rcvr signal avg optical power(Channel 4)  : 0.6694 mW / -1.74 dBm
That's spicy hot. Testbench doesn't have much airflow, real server does at least. Link is up and traffic is flowing.

Now that it's identified as an ethtool bug, I put them back to HP firmware and it's refusing to turn on TX when the link is brought up:
Code: 
        Transmit avg optical power (Channel 1)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 2)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.0000 mW / -inf dBm
    Rcvr signal avg optical power(Channel 1)  : 0.5985 mW / -2.23 dBm
    Rcvr signal avg optical power(Channel 2)  : 0.8011 mW / -0.96 dBm
    Rcvr signal avg optical power(Channel 3)  : 0.6857 mW / -1.64 dBm
    Rcvr signal avg optical power(Channel 4)  : 0.6239 mW / -2.05 dBm
That could be due to the power restriction in the HP firmware, but the board itself works with stock firmware now.
Click to expand...
My bad, I mixed up 282 and 283, I've got 282. It basically only worked on one port and not very well, so I went back to HP firmware
 
D

dmerillat

New Member
Jan 8, 2022
12
3
3
DaveLTX said:
My bad, I mixed up 282 and 283, I've got 282. It basically only worked on one port and not very well, so I went back to HP firmware
Click to expand...
Your assessment is still correct however: the 282/283 isn't worth it outside using it with a DAC or specific HP-branded gear. Franken-firmware is just not stable enough outside testing so another ebay MCX it is.

The HP firmware works great with the DAC to my ICX6610, so I'll keep that in the server. Just can't run the 40G-LR4 module in my workstation on the other end of my house reliably.
 
M

MLP305

New Member
Sep 9, 2024
4
0
1
Hope someone can help me (also piggy backing of this thread) , im new here and been going through the forums for help. I just got 4 HP 544+FLR-QSFP (part number 764285-B21) rev 5. I saw through this thread and several others that many patrons were able to get this to stock firmware and use it on different systems rather than HP.

I have been attempting to get these configured, none of my system is picking the cards up. I have read from different post that for some it was just plug and play on other system for them to be able to configure it the PCIe picked it up right away. I understand originally these cards were meant for HP.
Is there a way I can have a Dell system pick these up? currently using LGA 2011-v3 boards not sure if the board is too old for it, the firmware are up to date. Also tried it on a X99X-Killer board still nothing. Windows 10 22H2 OS and Ubuntu 22.04LTS.

Are there any specific settings I can test or check on the systems to get them to be pickup so I can flash them with stock firmware?
 
L

LodeRunner

Active Member
Apr 27, 2019
553
235
43
MLP305 said:
Hope someone can help me (also piggy backing of this thread) , im new here and been going through the forums for help. I just got 4 HP 544+FLR-QSFP (part number 764285-B21) rev 5. I saw through this thread and several others that many patrons were able to get this to stock firmware and use it on different systems rather than HP.

I have been attempting to get these configured, none of my system is picking the cards up. I have read from different post that for some it was just plug and play on other system for them to be able to configure it the PCIe picked it up right away. I understand originally these cards were meant for HP.
Is there a way I can have a Dell system pick these up? currently using LGA 2011-v3 boards not sure if the board is too old for it, the firmware are up to date. Also tried it on a X99X-Killer board still nothing. Windows 10 22H2 OS and Ubuntu 22.04LTS.

Are there any specific settings I can test or check on the systems to get them to be pickup so I can flash them with stock firmware?
Click to expand...
I have 6 of those cards (not sure on the rev number, rev 0 maybe?) in Dell servers and they were detected immediately with no issue, then I flashed them with the generic firmware. They still show the HP part number, but have the PSID of standard Mellanox cards (though I can't get my XQX2502 KAIAM transceivers working with them, but that's another thing that I may fix by just buying non OEM-branded cards).
1725915313457.png
 
M

MLP305

New Member
Sep 9, 2024
4
0
1
LodeRunner, Thank you for responding quickly, Im not sure why its not, showing on PCIe, once the system comes up an orange from the card lit up for a micro second, indicating that power is on.

After that, nothing on bios. Which Dell server board or model are you using. Since all of mine are LGA-2011-v3.
 
L

LodeRunner

Active Member
Apr 27, 2019
553
235
43
I have two Dell R720XD servers. So in Linux, lspci/dmesg/etc don't see them? IIRC, the disaster recovery mode has been discussed somewhere in this thread, if not, it's somewhere in the forum. My memory of the process is that you short a particular jumper on the card and then should be able to flash it but have to use some overrides to do so since the device ID's and such will be wrong?

Edit: this thread had some discussion of IBM cards and the firmware not present jumper: https://forums.servethehome.com/ind...connectx-3-en-with-stock-firmware.8689/page-2

From this thread, by the ever helpful fohdeesha: https://forums.servethehome.com/ind...al-port-qsfp-adapter.20525/page-3#post-198961
 
M

MLP305

New Member
Sep 9, 2024
4
0
1
Alright, I went and installed Ubuntu 16.04 and did the updates, and installed the MLNX_OFED 4.9-7.1.0.0. I believe the system does see the device but still having issues. lspci, dmidecode. This time im testing it on a HP server, still getting pretty much nowhere. On the Dell servers I get that Tainted error, and on HP pretty right I boot I get Failed to configure Mellanox Device and when I look at the logs its what I get below.



Sep 26 06:23:15 mlconfig2 systemd[1]: Starting openibd - configure Mellanox devices...
Sep 26 06:23:17 mlconfig2 openibd[873]: [71B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [69B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [71B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [69B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [71B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [69B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [63B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [69B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [71B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [70B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [49B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [69B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [68B blob data]
Sep 26 06:23:17 mlconfig2 openibd[873]: [70B blob data]
Sep 26 06:23:17 mlconfig2 systemd[1]: openibd.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Sep 26 06:23:17 mlconfig2 systemd[1]: Failed to start openibd - configure Mellanox devices.
Sep 26 06:23:17 mlconfig2 systemd[1]: openibd.service: Unit entered failed state.
Sep 26 06:23:17 mlconfig2 systemd[1]: openibd.service: Failed with result 'exit-code'.


Anyone have any idea, all 4 cards are the same and responding the same way.
 

Attachments

G

gavin888

New Member
Nov 17, 2024
1
0
1
Firebat said:
Can anyone provide original VPD information of a HP/HPE 649281-B21 card? I'm trying to crossflash an IBM model to HPE but I don't have the correct vpd information from an original HPE card.
Click to expand...
I just want to crossflash OEM card to channel ，just crosslash firmware is ok，but VPD not change。can you help to change OEM CX7 VPD ？thanks
 
You must log in or register to reply here.
Top Bottom