Crossflashed CX354A - Can only get 40G not 56G

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
Hi,

I bought some Oracle labeled CX354A's from ebay (Oracle 7046442) .
Flashed them with the FCBT firmware, everything is working out nice so far.

But I just can't get them to connect with 56G, or even enable the extended link speed.

Device Info:
Code:
Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  /dev/mst/mt4099_pci_cr1
  Port1 GUID:       0010e00001759be1
  Port2 GUID:       0010e00001759be2
  Versions:         Current        Available     
     FW             2.42.5000      N/A           

  Status:           No matching image found

Device #2:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  /dev/mst/mt4099_pci_cr0
  Port1 GUID:       0010e00001789fa1
  Port2 GUID:       0010e00001789fa2
  Versions:         Current        Available     
     FW             2.42.5000      N/A           

  Status:           No matching image found
Showing up as

43:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]


ibstatus:
Code:
Infiniband device 'mlx4_0' port 1 status:
        default gid:     fe80:0000:0000:0000:0010:e000:0178:9fa1
        base lid:        0x1
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      InfiniBand

Infiniband device 'mlx4_0' port 2 status:
        default gid:     fe80:0000:0000:0000:0010:e000:0178:9fa2
        base lid:        0x0
        sm lid:          0x0
        state:           1: DOWN
        phys state:      2: Polling
        rate:            10 Gb/sec (4X)
        link_layer:      InfiniBand

Infiniband device 'mlx4_1' port 1 status:
        default gid:     fe80:0000:0000:0000:0010:e000:0175:9be1
        base lid:        0x2
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      InfiniBand

Infiniband device 'mlx4_1' port 2 status:
        default gid:     fe80:0000:0000:0000:0010:e000:0175:9be2
        base lid:        0x0
        sm lid:          0x0
        state:           1: DOWN
        phys state:      2: Polling
        rate:            10 Gb/sec (4X)
        link_layer:      InfiniBand
ibstat:
Code:
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x0010e00001789fa0
        System image GUID: 0x0010e00001789fa3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251486a
                Port GUID: 0x0010e00001789fa1
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x0010e00001789fa2
                Link layer: InfiniBand
CA 'mlx4_1'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x0010e00001759be0
        System image GUID: 0x0010e00001759be3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 2
                LMC: 0
                SM lid: 1
                Capability mask: 0x02514868
                Port GUID: 0x0010e00001759be1
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x0010e00001759be2
                Link layer: InfiniBand
I tried to put them into FDR mode via ibportstate, but no whatter what I do, it doesn't work.

Code:
CA/RT PortInfo:
# Port info: Lid 1 port 1
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................1
SMLid:...........................1
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................10.0 Gbps
LinkSpeedExtSupported:...........14.0625 Gbps
LinkSpeedExtEnabled:.............0
LinkSpeedExtActive:..............No Extended Speed
Mkey:............................<not displayed>
MkeyLeasePeriod:.................0
ProtectBits:.....................0
# MLNX ext Port info: Lid 1 port 1
StateChangeEnable:...............0x00
LinkSpeedSupported:..............0x01
LinkSpeedEnabled:................0x00
LinkSpeedActive:.................0x00

Are the card not capable of FDR IB, or am I just using the wrong commands?
 

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
I'm using a 0.5m QSFP+ DAC from FS, sold as 40G.
Are the connectors keyed with specific connection speeds?

I will receive a 56G and 100G cable together with a genuine CX354A-FCBT later...


EDIT: Switched them to Ethernet mode, that works with 40G (which must me FDR, right?)
Code:
ethtool enp67s0
Settings for enp67s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  40000baseCR4/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
 

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
Alright - Progress.
56G indeed requires a 56G cable. And ONLY 56G cables work.

40G FS Cable -> QDR, 40G
100G Mellanox Cable -> FDR10, but still 40G.
56G Mellanox Cable -> FDR, 56G

Meh.
 

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
Nope, standard passive QSFP.

The 56G cable that works is a MC2207128-003-V-A3
The 100G cable is a MCP1600-C003

100G Cable:
Code:
CA 'mlx4_1'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x0010e00001759be0
        System image GUID: 0x0010e00001759be3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40 (FDR10)
                Base lid: 2
                LMC: 0
                SM lid: 1
                Capability mask: 0x02514868
                Port GUID: 0x0010e00001759be1
                Link layer: InfiniBand
56G Cable:
Code:
CA 'mlx4_1'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x0010e00001759be0
        System image GUID: 0x0010e00001759be3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 56
                Base lid: 2
                LMC: 0
                SM lid: 1
                Capability mask: 0x02514868
                Port GUID: 0x0010e00001759be1
                Link layer: InfiniBand
I suppose that it's just a programming thing, i.e. the Transceiver EEPROM needs to contain special Infiniband flags or otherwise the card itself refuses to use FDR 64b/66b encoding.
 

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
I'm currently in contact FS and we're figuring out whether it is possible to reprogram 100G QSFP28 cables to be accepted as 56G FDR cables by the Mellanox cards...
 

NablaSquaredG

Active Member
Aug 17, 2020
160
56
28
FS was like "Yes", then "Maybe", then "Yes" and finally "No".
According to them it is not possible to reprogram FS QSFP28 cables to Mellanox FDR cables.

For all those looking for a solution, you could
a) Buy used original Mellanox cables. I bought 5 for about 29€ each (2m), that's okay. I don't wanted to buy the 1m cables. 1.5m could have been enough, I'll see when I set up the rack
b) There are used Finisar 56G Transceivers on ebay that might work. Caveat: They need MPO-12 / 8 (4 TX, 4 RX, 4 unused)
c) Buy one of the prohibitively expensive programming devices (e.g. SFPTotal or dimiks) and try to reflash 100G QSFP28 to 56G FDR
d) Skip 56G and use EDR, that might indeed work with generic 100G QSFP28 cables.
 

thetoad

Member
Feb 10, 2021
48
14
8
hey, sorry to resurect this thread, are there instructions anywhere to follow for the flashing operation you did?