Help with CX456A ( HP OEM)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nthu9280

Well-Known Member
Feb 3, 2016
1,628
498
83
San Antonio, TX
I have a CX456A (HP OEM) flashed to Mellonox stock FW. It's in a PCIe 3 x16 slot. lshw shows 56 Gb/s on each port. Is that normal?


Code:
# lspci -vv | grep -i mellanox
lspci: Unable to load libkmod resources: error -12
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
    Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
04:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
    Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]


# lshw -c network
  *-network:0               
       description: Ethernet interface
       product: MT27700 Family [ConnectX-4]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:04:00.0
       logical name: ens4f0np0
       version: 00
       serial: 04:09:73:dc:12:44
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd 56000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.3-1.0.0 firmware=12.28.2006 (MT_2190110032) latency=0 link=no multicast=yes
       resources: iomemory:33f0-33ef irq:24 memory:33ffc000000-33ffdffffff
  *-network:1
       description: Ethernet interface
       product: MT27700 Family [ConnectX-4]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:04:00.1
       logical name: ens4f1np1
       version: 00
       serial: 04:09:73:dc:12:45
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd 56000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.3-1.0.0 firmware=12.28.2006 (MT_2190110032) latency=0 link=no multicast=yes
       resources: iomemory:33f0-33ef irq:46 memory:33ffa000000-33ffbffffff
  *-network
       description: Ethernet interface
       product: Ethernet Connection I217-LM
       vendor: Intel Corporation
       physical id: 19
       bus info: pci@0000:00:19.0
       logical name: enp0s25
       version: 05
       serial: 50:9a:4c:52:55:06
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=3.2.6-k duplex=full firmware=0.13-4 ip=10.16.32.174 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:40 memory:f7f00000-f7f1ffff memory:f7f32000-f7f32fff ioport:f000(size=32)


# ibdev2netdev -v
0000:04:00.0 mlx5_0 (MT4115 - CN781305DRV0) HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter fw 12.28.2006 port 1 (DOWN  ) ==> ens4f0np0 (Down)
0000:04:00.1 mlx5_1 (MT4115 - CN781305DRV0) HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter fw 12.28.2006 port 1 (DOWN  ) ==> ens4f1np1 (Down)
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
It looks like you flashed it with MCX456A-FCAT (40/56GBE only) instead of MCX456A-ECAT (supports up to 100GBE).
 

nthu9280

Well-Known Member
Feb 3, 2016
1,628
498
83
San Antonio, TX
@i386 Just checked the history..

Code:
flint -d /dev/mst/mt4115_pciconf0 -i fw-ConnectX4-rel-12_28_2006-MCX456A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin --allow_psid_change burn

and also:

# mlxfwmanager -d /dev/mst/mt4115_pciconf0 --online
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4
  Part Number:      MCX456A-ECA_Ax
  Description:      ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; ROHS R6
  PSID:             MT_2190110032
  PCI Device Name:  /dev/mst/mt4115_pciconf0
  Base MAC:         040973dc1244
  Versions:         Current        Available     
     FW             12.28.2006     12.28.2006    
     PXE            3.6.0102       3.6.0102      
     UEFI           14.21.0017     14.21.0017    

  Status:           Up to date
 

jpmomo

Active Member
Aug 12, 2018
531
192
43
looks like you need to switch from IB to Ethernet using the mellanox tools mlxcfg etc.
 

nthu9280

Well-Known Member
Feb 3, 2016
1,628
498
83
San Antonio, TX
Assuming you are referring to the below.. If so, I did that already immediately after flashing the mlnx fw.

Code:
mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


mlxconfig -d /dev/mst/mt4115_pciconf0 query

Device #1:
----------

Device type:    ConnectX4       
Name:           MCX456A-ECA_Ax 
Description:    ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; ROHS R6
Device:         /dev/mst/mt4115_pciconf0

         LINK_TYPE_P1                        ETH(2)         
         LINK_TYPE_P2                        ETH(2)
 

nthu9280

Well-Known Member
Feb 3, 2016
1,628
498
83
San Antonio, TX
@Rand__ Any other things I can check?

mlxconfig shows that this is a 100GBe.
lshw / ubuntu identify this as 56GBe. Using the latest mlx5 driver and latest FW.
Plugged in to PCIe 3.0 x16 slot.

I only have one dual port card so can't really test other than a loop back.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
have you forced the link to 100G ?

or whats on the other side? what cable are you using?

Edit - Ah loopback - so cable maybe
 

jpmomo

Active Member
Aug 12, 2018
531
192
43
you can try the mxlink cmd if you think it might be the cables. I had an issue with a 200GE mellanox nic that would not sync up at the 200G rate with some generic qsfp56 cables. I need to use the cables directly from mellanox before they would show the correct 200G rate capability. Never really had an issue with the 100G cards even with generic DAC cables. The latest vmware also supports the 200G cards now (they use to only support the 200G cards at 100Gbps rate.)
 

jpmomo

Active Member
Aug 12, 2018
531
192
43
[root@localhost:/opt/mellanox/bin] ./mlxlink -d mt4123_pciconf0 -m

Operational Info
----------------
State : Active
Physical state : LinkUp
Speed : 40G
Width : 4x
FEC : No FEC
Loopback Mode : No Loopback
Auto Negotiation : ON

Supported Info
--------------
Enabled Link Speed (Ext.) : 0x000017f2 (200G,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G)
Supported Cable Speed (Ext.) : 0x00000032 (40G,10G,1G)
 
  • Like
Reactions: fohdeesha