Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Robert Townley

Robert Townley

New Member
Dec 23, 2014
29
5
3
Luzer said:
To piggyback off of this. I got a Mellanox CX354A ConnectX-3 QDR QSFP+ 40+10 Dual Port Server HBA 050-0050-02 but is only showing one port in ESXi. What's the best way to flash it to current and enable the second port?

Thanks.
Click to expand...
Can you still get to the Linux cli of vmware?
Bash: 
lsmod | egrep -i '(MLX4)'
modinfo mlx4_core | egrep port_type_array
parm:           port_type_array:Array of port types: HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet (array of int)
cat /sys/module/mlx4_core/parameters/port_type_array
2,2        # The card should  behave as two Ethernet ports.
             # If this output is '1,2' or '2,1' the other port is set to infiniband in the firmware of the nic or somewhere in vmware.  
             # If '0,1' or '1,0' then one port is set to infiniband and the 
             # If '0,0' the Linux kernel module will use what the card firmware says.
 
S

seanneko

New Member
Feb 1, 2022
17
5
3
Has anyone got any experience with getting an HP 544+FLR-QSFP (part number 764285-B21) to work?

I've successfully flashed regular ConnectX-3 cards with MCX354A-FCBT firmware and got them working, but this HP card is an MCX354A-FCCT (a ConnectX-3 Pro).

It's not possible to flash it with non-Pro firmware but I have flashed it with the correct Pro firmware. I've tried both the HP firmware (2.42.5700) as well as the Mellanox firmware (2.42.5000). I also tried MCX342A firmware (Ethernet only ConnectX-3 Pro) just out of curiosity.

Using mlxconfig I've set:

LINK_TYPE = ETH
PHY_TYPE = XFI
XFI_MODE = 40G

The flash seems to be successful and I can query the card and see both network interfaces when I do an ifconfig in Linux. If I insert an SFP, I can query the SFP using ethtool -m. There's nothing in dmesg to indicate any issues.

But no matter what I do, I can never get the link to come up. I've tried:

- Connecting it to a switch
- Connecting port 1 on the card to port 2
- Using an LC loopback connector (goes out the TX and back in the RX on the same SFP)

It makes no difference what sort of SFP I used. I've tried LR4-Lite, Cisco SR4, and even a genuine HP branded active optical cable. With the SR4, which uses visible light, I can't see any light coming out of it. So the card doesn't seem to be telling the SFP to turn its lasers on.

All of these SFPs and cables are tested working on a regular ConnectX-3 card so it's nothing to do with them. It's only on this HP ConnectX-3 Pro that there's this problem.

Does anyone have any ideas?
 
P

prdtabim

Active Member
Jan 29, 2022
171
66
28
seanneko said:
Has anyone got any experience with getting an HP 544+FLR-QSFP (part number 764285-B21) to work?

I've successfully flashed regular ConnectX-3 cards with MCX354A-FCBT firmware and got them working, but this HP card is an MCX354A-FCCT (a ConnectX-3 Pro).

It's not possible to flash it with non-Pro firmware but I have flashed it with the correct Pro firmware. I've tried both the HP firmware (2.42.5700) as well as the Mellanox firmware (2.42.5000). I also tried MCX342A firmware (Ethernet only ConnectX-3 Pro) just out of curiosity.

Using mlxconfig I've set:

LINK_TYPE = ETH
PHY_TYPE = XFI
XFI_MODE = 40G

The flash seems to be successful and I can query the card and see both network interfaces when I do an ifconfig in Linux. If I insert an SFP, I can query the SFP using ethtool -m. There's nothing in dmesg to indicate any issues.

But no matter what I do, I can never get the link to come up. I've tried:

- Connecting it to a switch
- Connecting port 1 on the card to port 2
- Using an LC loopback connector (goes out the TX and back in the RX on the same SFP)

It makes no difference what sort of SFP I used. I've tried LR4-Lite, Cisco SR4, and even a genuine HP branded active optical cable. With the SR4, which uses visible light, I can't see any light coming out of it. So the card doesn't seem to be telling the SFP to turn its lasers on.

All of these SFPs and cables are tested working on a regular ConnectX-3 card so it's nothing to do with them. It's only on this HP ConnectX-3 Pro that there's this problem.

Does anyone have any ideas?
Click to expand...
I use 6 of this card 764285-B21.
Code: 
Example config from one of my cards
mlxconfig q

Device #1:
----------

Device type:    ConnectX3Pro   
Device:         0000:33:00.0   

Configurations:                              Next Boot
         SRIOV_EN                            True(1)        
         NUM_OF_VFS                          8              
         WOL_MAGIC_EN_P2                     True(1)        
         LINK_TYPE_P1                        ETH(2)         
         PHY_TYPE_P1                         XFI(2)         
         XFI_MODE_P1                         _10G(0)        
         FORCE_MODE_P1                       False(0)       
         LINK_TYPE_P2                        ETH(2)         
         PHY_TYPE_P2                         XFI(2)         
         XFI_MODE_P2                         _10G(0)        
         FORCE_MODE_P2                       False(0)       
         LOG_BAR_SIZE                        5              
         BOOT_PKEY_P1                        0              
         BOOT_PKEY_P2                        0              
         BOOT_OPTION_ROM_EN_P1               True(1)        
         BOOT_VLAN_EN_P1                     False(0)       
         BOOT_RETRY_CNT_P1                   0              
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)         
         BOOT_VLAN_P1                        1              
         BOOT_OPTION_ROM_EN_P2               True(1)        
         BOOT_VLAN_EN_P2                     False(0)       
         BOOT_RETRY_CNT_P2                   0              
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)         
         BOOT_VLAN_P2                        1              
         IP_VER_P1                           IPv4(0)        
         IP_VER_P2                           IPv4(0)        
         CQ_TIMESTAMP                        True(1)        
         STEER_FORCE_VLAN                    False(0)       

flint -d /dev/mst/mt4103_pci_cr0  q full                                                                
Image type:            FS2
FW Version:            2.42.5700
FW Release Date:       8.4.2020
MIC Version:           2.0.0
Config Sectors:        2
PRS Name:              cx3pro_hp_alom_2p_fdr.prs
Product Version:       02.42.57.00
Rom Info:              version_id=8025 type=CLP 
                       type=UEFI version=14.11.49 cpu=AMD64
                       type=PXE version=3.4.754
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 
MACs:                 
VSD:                   
PSID:                  HP_1380110017
Important settings:
LINK_TYPE_P1 ETH(2) <--- dont use VPI. It's buggy at negotiate
PHY_TYPE_P1 XFI(2) <--- default mode that works for me
LOG_BAR_SIZE 5 <--- shouldn't be too high
NUM_OF_VFS 8 <--- shouldn't be too high

This parameter is usefull only to force a speed using an unknown/incompatible cable
XFI_MODE_P1/2 _10G(0) <--- only works if FORCE_MODE_P1/2 = True and mode =XFI

I hope that this will help.
 
  • Like
Reactions: konata and nedimzukic2
S

seanneko

New Member
Feb 1, 2022
17
5
3
prdtabim said:
I use 6 of this card 764285-B21.
Code: 
Example config from one of my cards
mlxconfig q

Device #1:
----------

Device type:    ConnectX3Pro  
Device:         0000:33:00.0  

Configurations:                              Next Boot
         SRIOV_EN                            True(1)       
         NUM_OF_VFS                          8             
         WOL_MAGIC_EN_P2                     True(1)       
         LINK_TYPE_P1                        ETH(2)        
         PHY_TYPE_P1                         XFI(2)        
         XFI_MODE_P1                         _10G(0)       
         FORCE_MODE_P1                       False(0)      
         LINK_TYPE_P2                        ETH(2)        
         PHY_TYPE_P2                         XFI(2)        
         XFI_MODE_P2                         _10G(0)       
         FORCE_MODE_P2                       False(0)      
         LOG_BAR_SIZE                        5             
         BOOT_PKEY_P1                        0             
         BOOT_PKEY_P2                        0             
         BOOT_OPTION_ROM_EN_P1               True(1)       
         BOOT_VLAN_EN_P1                     False(0)      
         BOOT_RETRY_CNT_P1                   0             
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)        
         BOOT_VLAN_P1                        1             
         BOOT_OPTION_ROM_EN_P2               True(1)       
         BOOT_VLAN_EN_P2                     False(0)      
         BOOT_RETRY_CNT_P2                   0             
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)        
         BOOT_VLAN_P2                        1             
         IP_VER_P1                           IPv4(0)       
         IP_VER_P2                           IPv4(0)       
         CQ_TIMESTAMP                        True(1)       
         STEER_FORCE_VLAN                    False(0)      

flint -d /dev/mst/mt4103_pci_cr0  q full                                                               
Image type:            FS2
FW Version:            2.42.5700
FW Release Date:       8.4.2020
MIC Version:           2.0.0
Config Sectors:        2
PRS Name:              cx3pro_hp_alom_2p_fdr.prs
Product Version:       02.42.57.00
Rom Info:              version_id=8025 type=CLP
                       type=UEFI version=14.11.49 cpu=AMD64
                       type=PXE version=3.4.754
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                
MACs:                
VSD:                  
PSID:                  HP_1380110017
Important settings:
LINK_TYPE_P1 ETH(2) <--- dont use VPI. It's buggy at negotiate
PHY_TYPE_P1 XFI(2) <--- default mode that works for me
LOG_BAR_SIZE 5 <--- shouldn't be too high
NUM_OF_VFS 8 <--- shouldn't be too high

This parameter is usefull only to force a speed using an unknown/incompatible cable
XFI_MODE_P1/2 _10G(0) <--- only works if FORCE_MODE_P1/2 = True and mode =XFI

I hope that this will help.
Click to expand...
Thanks. I've reflashed it again and triple checked all the settings. It all matches yours exactly but still isn't working. I have two cards (which were bought from different suppliers) and they're both doing the same thing so I don't think it's faulty hardware.

I did some more testing with different SFPs. I was wrong in saying that with an SR4 I couldn't see any light. It was actually an SR-BD I was using which I believe doesn't work because it's a 3.5W high power SFP. With a *REAL* SR4, I can see red light coming from it. I don't have a suitable cable to test if the link will come up though.

The SFPs that I'm trying to use are KAIAM LR4-Lite which I successfully use in regular ConnectX-3 cards.

ethtool -m output:

Code: 
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x80
        Extended identifier description           : 2.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-LR4
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 2km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 0m
        Transmitter technology                    : 0x40 (1310 nm DFB)
        Laser wavelength                          : 1310.000nm
        Laser wavelength tolerance                : 6.500nm
        Vendor name                               : KAIAM CORP
        Vendor OUI                                : 14:ed:e4
        Vendor PN                                 : XQX2502
        Vendor rev                                : 1A
        Vendor SN                                 : KD61111423
        Date code                                 : 16111100
        Revision Compliance                       : SFF-8636 Rev 1.5
        Module temperature                        : 43.30 degrees C / 109.93 degrees F
        Module voltage                            : 3.3411 V
        Alarm/warning flags implemented           : No
        Laser tx bias current (Channel 1)         : 24.946 mA
        Laser tx bias current (Channel 2)         : 22.970 mA
        Laser tx bias current (Channel 3)         : 26.922 mA
        Laser tx bias current (Channel 4)         : 20.500 mA
        Transmit avg optical power (Channel 1)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 2)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.0000 mW / -inf dBm
        Rcvr signal avg optical power(Channel 1)  : 0.8641 mW / -0.63 dBm
        Rcvr signal avg optical power(Channel 2)  : 0.9069 mW / -0.42 dBm
        Rcvr signal avg optical power(Channel 3)  : 0.7119 mW / -1.48 dBm
        Rcvr signal avg optical power(Channel 4)  : 0.7204 mW / -1.42 dBm
Under rcvr signal it seems to indicate that it's receiving light from the SFP on the switch. The values of around ~0.9 mW are the same as what I see on my working ConnectX-3 cards. The Pro just doesn't want to turn on the laser and transmit.
 
P

prdtabim

Active Member
Jan 29, 2022
171
66
28
seanneko said:
Thanks. I've reflashed it again and triple checked all the settings. It all matches yours exactly but still isn't working. I have two cards (which were bought from different suppliers) and they're both doing the same thing so I don't think it's faulty hardware.

I did some more testing with different SFPs. I was wrong in saying that with an SR4 I couldn't see any light. It was actually an SR-BD I was using which I believe doesn't work because it's a 3.5W high power SFP. With a *REAL* SR4, I can see red light coming from it. I don't have a suitable cable to test if the link will come up though.

The SFPs that I'm trying to use are KAIAM LR4-Lite which I successfully use in regular ConnectX-3 cards.

ethtool -m output:

Code: 
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x80
        Extended identifier description           : 2.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-LR4
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 2km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 0m
        Transmitter technology                    : 0x40 (1310 nm DFB)
        Laser wavelength                          : 1310.000nm
        Laser wavelength tolerance                : 6.500nm
        Vendor name                               : KAIAM CORP
        Vendor OUI                                : 14:ed:e4
        Vendor PN                                 : XQX2502
        Vendor rev                                : 1A
        Vendor SN                                 : KD61111423
        Date code                                 : 16111100
        Revision Compliance                       : SFF-8636 Rev 1.5
        Module temperature                        : 43.30 degrees C / 109.93 degrees F
        Module voltage                            : 3.3411 V
        Alarm/warning flags implemented           : No
        Laser tx bias current (Channel 1)         : 24.946 mA
        Laser tx bias current (Channel 2)         : 22.970 mA
        Laser tx bias current (Channel 3)         : 26.922 mA
        Laser tx bias current (Channel 4)         : 20.500 mA
        Transmit avg optical power (Channel 1)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 2)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.0000 mW / -inf dBm
        Rcvr signal avg optical power(Channel 1)  : 0.8641 mW / -0.63 dBm
        Rcvr signal avg optical power(Channel 2)  : 0.9069 mW / -0.42 dBm
        Rcvr signal avg optical power(Channel 3)  : 0.7119 mW / -1.48 dBm
        Rcvr signal avg optical power(Channel 4)  : 0.7204 mW / -1.42 dBm
Under rcvr signal it seems to indicate that it's receiving light from the SFP on the switch. The values of around ~0.9 mW are the same as what I see on my working ConnectX-3 cards. The Pro just doesn't want to turn on the laser and transmit.
Click to expand...
Did you tested using a DAC or an AOC cable ?
If a message show about unsupported cable, you can force the link using the option FORCE_MODE_P1/2 = true . The speed will be that on XFI_MODE_P1/2 .
 
  • Like
Reactions: Robert Townley
F

Firebat

New Member
Jan 12, 2022
12
2
3
I want to chime in here with a few questions. I've bought two IBM 00D9552, they are shown as following:

Code: 
mlxvpd -d /dev/mst/mt4099_pci_cr0

  VPD-KEYWORD    DESCRIPTION             VALUE
  -----------    -----------             -----
Read Only Section:

  PN             Part Number             00D9551
  EC             Revision                N32186D
  V2             N/A                     00D9552
  SN             Serial Number           
  V3             N/A                     
  MN             Manufacture ID          A5
  VA             N/A                     IBM:MODL=MCX354A-FCBT:CNLY=22:PKTY=9:PHTY=2:MN=IBM:CSKU=V2:UUID=V3
  V0             Misc Info               Mellanox ConnectX3 FDR IB / 40GE Adapter for IBM System x
  RV             Checksum Complement     0x5
  IDTAG          Board Id                Mellanox CX3 VPI Adapter
 
flint -d /dev/mst/mt4099_pci_cr0 q full
Image type:            FS2
FW Version:            2.30.3000
MIC Version:           1.5.0
Rom Info:              type=UEFI version=11.2.38 proto=ETH
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                   
MACs:                                           
VSD:
PSID:                  IBM1090111019


lspci -vvvs 08:00.0 | grep -A 15 'Vital Product Data'
        Capabilities: [48] Vital Product Data
                Product Name: Mellanox CX3 VPI Adapter
                Read-only fields:
                        [PN] Part number: 00D9551
                        [EC] Engineering changes: N32186D
                        [V2] Vendor specific: 00D9552
                        [SN] Serial number:
                        [V3] Vendor specific:
                        [MN] Manufacture ID: A5
                        [VA] Vendor specific: IBM:MODL=MCX354A-FCBT:CNLY=22:PKTY=9:PHTY=2:MN=IBM:CSKU=V2:UUID=V3
                        [V0] Vendor specific: Mellanox ConnectX3 FDR IB / 40GE Adapter for IBM System x
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [V1] Vendor specific: N/A
                        [YA] Asset tag: N/A
                        [RW] Read-write area: 212 byte(s) free
                        
mlxfwmanager
Querying Mellanox devices firmware ...

Device #2:
----------

  Device Type:      ConnectX3
  Part Number:      00D9551_Ax
  Description:      Mellanox ConnectX-3 VPI Dual-port QSFP FDR14/40GbE HCA
  PSID:             IBM1090111019
  PCI Device Name:  /dev/mst/mt4099_pci_cr0
  Port1 GUID:       
  Port2 GUID:       
  Versions:         Current        Available
     FW             2.30.3000      2.30.3000
     UEFI           11.2.0038      11.2.0038

  Status:           Up to date
As one can see, crossflashing to MCX354A-FCBT generic should be possible. BUT the PSID shown is the PSID of a MCX354A-FCCT as far as I can tell. I THINK these are ConnectX-3 Pro cards with standard Firmware. Revision is A5, at least that's what the sticker tells me. But crossflashing with ConnectX-3 Pro firmware doesn't work.

Code: 
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin -allow_psid_change burn

    Current FW version on flash:  2.30.3000
    New FW version:               2.42.5000


    You are about to replace current PSID on flash - "IBM1090111019" with a different PSID - "MT_1090111019".
    Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only
So now I want to hear what you would try next. Would be great to get a real ConnectX-3 Pro out of it. :)
 
BeTeP

BeTeP

Well-Known Member
Mar 23, 2019
657
433
63
PSID does not mean much. Its similarity with "MT_1090111019" does not mean anything. Always go by the "Device ID". And "4099" means it's a regular non-pro CX3.
 
  • Like
Reactions: Firebat
F

Firebat

New Member
Jan 12, 2022
12
2
3
Thanks a lot for the clarification, both of you :) Will try to get the ConnectX-3 standard fw on the card tomorrow.

€dit: Worked like a charm with standard ConnectX-3 firmware.
 
Last edited:
F

Firebat

New Member
Jan 12, 2022
12
2
3
Can anyone provide original VPD information of a HP/HPE 649281-B21 card? I'm trying to crossflash an IBM model to HPE but I don't have the correct vpd information from an original HPE card.
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
Robert Townley said:
Can you still get to the Linux cli of vmware?
Bash: 
lsmod | egrep -i '(MLX4)'
modinfo mlx4_core | egrep port_type_array
parm:           port_type_array:Array of port types: HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet (array of int)
cat /sys/module/mlx4_core/parameters/port_type_array
2,2        # The card should  behave as two Ethernet ports.
             # If this output is '1,2' or '2,1' the other port is set to infiniband in the firmware of the nic or somewhere in vmware. 
             # If '0,1' or '1,0' then one port is set to infiniband and the
             # If '0,0' the Linux kernel module will use what the card firmware says.
Click to expand...
I'll try this and let you know.
 
E

Ergonomic4553

New Member
May 2, 2023
4
0
1
Well, here is little old me with a flashed CX-3 dual port QSFP+ not having any luck getting the ports to show up. A picture says more then a thousand words:
2023-05-02 22_50_11-TrueNAS - 192.168.5.117 and 3 more pages - Personal - Microsoft​ Edge.png
Yet it seems that the device is set up correctly and shows up correctly:
2023-05-02 22_50_04-TrueNAS - 192.168.5.117 and 3 more pages - Personal - Microsoft​ Edge.png

Yet when trying to set the link type...
1683060835805.png

Any clue on what's happening here?
 
P

prdtabim

Active Member
Jan 29, 2022
171
66
28
Ergonomic4553 said:
Well, here is little old me with a flashed CX-3 dual port QSFP+ not having any luck getting the ports to show up. A picture says more then a thousand words:
View attachment 28749
Yet it seems that the device is set up correctly and shows up correctly:
View attachment 28750

Yet when trying to set the link type...
View attachment 28751

Any clue on what's happening here?
Click to expand...
I guess that your card need a configuration reset
Code: 
mlxconfig -d /dev/mst/mt4099_pciconf0 reset
I already update a card with a very old firmware and the configs are missing parts. After the config reset all the configs show as expected.
Look here for more details Examples of mlxconfig Usage - MFT v4.17.0 - NVIDIA Networking Docs
 
G

gethro2k

New Member
Jan 22, 2023
1
0
1
seanneko said:
Has anyone got any experience with getting an HP 544+FLR-QSFP (part number 764285-B21) to work?
Click to expand...
Sorry to barge in, but did you ever get this working? Which model ProLiant do you have?

I have 2 x DL380P Gen8 and two of these cards, when I put them in, both DL's run for about three minutes then the servers shut down with a "Critical Temperature Warning."

Watching the temps in iLO does not show any temps too high. I believe the CTW's are bogus.
 
S

seanneko

New Member
Feb 1, 2022
17
5
3
gethro2k said:
Sorry to barge in, but did you ever get this working? Which model ProLiant do you have?

I have 2 x DL380P Gen8 and two of these cards, when I put them in, both DL's run for about three minutes then the servers shut down with a "Critical Temperature Warning."

Watching the temps in iLO does not show any temps too high. I believe the CTW's are bogus.
Click to expand...
No, unfortunately I didn't. I tried on a few different servers (DL380p, DL560, DL580) and couldn't get it to work. I think from memory I got some SFPs to work, but not the ones I wanted to use. I spent a good number of hours trying different things, and in the end I gave up and got some regular non-HP PCIe cards instead.
 
Robert Townley

Robert Townley

New Member
Dec 23, 2014
29
5
3
seanneko said:
No, unfortunately I didn't. I tried on a few different servers (DL380p, DL560, DL580) and couldn't get it to work. I think from memory I got some SFPs to work, but not the ones I wanted to use. I spent a good number of hours trying different things, and in the end I gave up and got some regular non-HP PCIe cards instead.
Click to expand...
@gethro2k did you try flashing it with firmware from HPE? When i put a HP branded Mellanox into a Dell, the Dell Firmware update flashed it with the latest "Dell" firmware and worked fine. So yes, a HPE branded card running Dell Firmware. Not sure if it was the lifecycle controller firmware update or one of the Dell firmware ISO images. Probably the latest ISO download.
 
Robert Townley

Robert Townley

New Member
Dec 23, 2014
29
5
3
seanneko said:
Thanks. I've reflashed it again and triple checked all the settings. It all matches yours exactly but still isn't working. I have two cards (which were bought from different suppliers) and they're both doing the same thing so I don't think it's faulty hardware.

I did some more testing with different SFPs. I was wrong in saying that with an SR4 I couldn't see any light. It was actually an SR-BD I was using which I believe doesn't work because it's a 3.5W high power SFP. With a *REAL* SR4, I can see red light coming from it. I don't have a suitable cable to test if the link will come up though.

The SFPs that I'm trying to use are KAIAM LR4-Lite which I successfully use in regular ConnectX-3 cards.

ethtool -m output:

Code: 
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x80
        Extended identifier description           : 2.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-LR4
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 2km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 0m
        Transmitter technology                    : 0x40 (1310 nm DFB)
        Laser wavelength                          : 1310.000nm
        Laser wavelength tolerance                : 6.500nm
        Vendor name                               : KAIAM CORP
        Vendor OUI                                : 14:ed:e4
        Vendor PN                                 : XQX2502
        Vendor rev                                : 1A
        Vendor SN                                 : KD61111423
        Date code                                 : 16111100
        Revision Compliance                       : SFF-8636 Rev 1.5
        Module temperature                        : 43.30 degrees C / 109.93 degrees F
        Module voltage                            : 3.3411 V
        Alarm/warning flags implemented           : No
        Laser tx bias current (Channel 1)         : 24.946 mA
        Laser tx bias current (Channel 2)         : 22.970 mA
        Laser tx bias current (Channel 3)         : 26.922 mA
        Laser tx bias current (Channel 4)         : 20.500 mA
        Transmit avg optical power (Channel 1)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 2)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.0000 mW / -inf dBm
        Rcvr signal avg optical power(Channel 1)  : 0.8641 mW / -0.63 dBm
        Rcvr signal avg optical power(Channel 2)  : 0.9069 mW / -0.42 dBm
        Rcvr signal avg optical power(Channel 3)  : 0.7119 mW / -1.48 dBm
        Rcvr signal avg optical power(Channel 4)  : 0.7204 mW / -1.42 dBm
Under rcvr signal it seems to indicate that it's receiving light from the SFP on the switch. The values of around ~0.9 mW are the same as what I see on my working ConnectX-3 cards. The Pro just doesn't want to turn on the laser and transmit.
Click to expand...
I am a noob, but looks to me that the QSFP+ is a LongRange SingleModeFiber transmiter that requires as much 3.5Watts, but 2.5Watts is the max power consumption because the High Power Class (> 3.5W) is not enabled? Post the same output for the working cards, please?
Extended identifier description : 2.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
 
Robert Townley

Robert Townley

New Member
Dec 23, 2014
29
5
3
Firebat said:
Can anyone provide original VPD information of a HP/HPE 649281-B21 card? I'm trying to crossflash an IBM model to HPE but I don't have the correct vpd information from an original HPE card.
Click to expand...
Wonder if you have access to the HPE firmware updates? HPE might flash the IBM card so it is branded HPE. Dell did this to three HPE Mellanox cards.
 
Robert Townley

Robert Townley

New Member
Dec 23, 2014
29
5
3
Znuff said:
Hi,

I've followed the instructions and successfully flashed some:

ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6

to generic firmware.

And for most of it, the cards seem to work fine.

I can get around >30Gbit/s with a single iperf3 between 2 servers with these cards, which seems great.

But once I actually put them to real work, they'll cap at 17Gbit/s:

View attachment 22863

At this point, iperf3 will also return poor speeds (sub 10Mbit), so it's clear that it has reached it's capped.

Tranceivers are Arista:

Code: 
Identifier                                : 0x0d (QSFP+)
Extended identifier                       : 0xc0
Extended identifier description           : 3.5W max. Power consumption
Extended identifier description           : No CDR in TX, No CDR in RX
Extended identifier description           : High Power Class (> 3.5 W) not enabled
Connector                                 : 0x07 (LC)
Transceiver codes                         : 0x02 0x00 0x00 0x00 0x10 0x10 0x01 0xd5
Transceiver type                          : 40G Ethernet: 40G Base-LR4
Transceiver type                          : FC: long distance (L)
Transceiver type                          : FC: Longwave laser (LL)
Transceiver type                          : FC: Single Mode (SM)
Transceiver type                          : FC: 1200 MBytes/sec
Transceiver type                          : FC: 800 MBytes/sec
Transceiver type                          : FC: 400 MBytes/sec
Transceiver type                          : FC: 200 MBytes/sec
Transceiver type                          : FC: 100 MBytes/sec
Encoding                                  : 0x03 (NRZ)
BR, Nominal                               : 10300Mbps
Rate identifier                           : 0x00
Length (SMF,km)                           : 10km
Length (OM3 50um)                         : 0m
Length (OM2 50um)                         : 0m
Length (OM1 62.5um)                       : 0m
Length (Copper or Active cable)           : 0m
Transmitter technology                    : 0x40 (1310 nm DFB)
Laser wavelength                          : 1310.000nm
Laser wavelength tolerance                : 10.000nm
Vendor name                               : Arista Networks
Vendor OUI                                : 64:9d:99
Vendor PN                                 : QSFP-LR4-40G
Vendor rev                                : 1A
Vendor SN                                 : F2020105366
Revision Compliance                       : SFF-8636 Rev 2.5/2.6/2.7
Module temperature                        : 39.28 degrees C / 102.70 degrees F
Module voltage                            : 3.1662 V
Alarm/warning flags implemented           : Yes
Laser tx bias current (Channel 1)         : 38.464 mA
Laser tx bias current (Channel 2)         : 40.896 mA
Laser tx bias current (Channel 3)         : 37.952 mA
Laser tx bias current (Channel 4)         : 36.864 mA
Transmit avg optical power (Channel 1)    : 0.8780 mW / -0.57 dBm
Transmit avg optical power (Channel 2)    : 0.8186 mW / -0.87 dBm
Transmit avg optical power (Channel 3)    : 1.0237 mW / 0.10 dBm
Transmit avg optical power (Channel 4)    : 0.8855 mW / -0.53 dBm
Rcvr signal avg optical power(Channel 1)  : 0.6321 mW / -1.99 dBm
Rcvr signal avg optical power(Channel 2)  : 1.0105 mW / 0.05 dBm
Rcvr signal avg optical power(Channel 3)  : 0.9729 mW / -0.12 dBm
Rcvr signal avg optical power(Channel 4)  : 0.9047 mW / -0.43 dBm
And they seem to be in fine shape.

Has anyone encountered a similar issue? I'm puzzled of what it could be.
Click to expand...
Wonder if the fiber optic cable is not long enough because that Arista QSFP-LR4-40G is meant for Long Range multi kilometer distances.
 
You must log in or register to reply here.
Top Bottom