Cross-flashed Mellanox CX354A-QCBT to FCBT only run at QDR (32Gb/s)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

WadeW

New Member
Oct 10, 2021
4
0
1
Hi everyone,

I cross flashed a Oracle OEM 7046442 (354A-QCBT) to CX354A-FCBT with the following command. It is connected to a FDR10 switch, which then gos to a bunch of CX354A-FCBT, either native or flashed from QCBT.

/usr/bin/flint -d /dev/mst/mt4099_pciconf0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin --allow_psid_change burn

However, ib_read_bw gives only 31.54Gb/s, ibstat also reads QDR speed only. I switched the cables already, nothing had changed. Any advice is appreciated.


# mlxfwmanager -u -d 01:00.0 --query
Querying Mellanox devices firmware ...

Device #1:
----------

Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: 01:00.0
Port1 GUID: 0010e000010ad491
Port2 GUID: 0010e000010ad492
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752

Status: Up to date

#ib_read_bw -a -F 172.168.168.1 -d mlx4_0 --report_gbits
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1f QPN 0x0218 PSN 0x875a34 OUT 0x10 RKey 0x010100 VAddr 0x007fa49ce5e000
remote address: LID 0x10 QPN 0x29f6b PSN 0x2432a6 OUT 0x10 RKey 0x20411237 VAddr 0x002b29cd000000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.124028 0.077789 4.861819
4 1000 0.25 0.21 6.715182
8 1000 0.47 0.44 6.819802
16 1000 0.99 0.78 6.130253
32 1000 1.98 1.58 6.167505
64 1000 3.85 3.16 6.163159
128 1000 7.72 6.32 6.170413
256 1000 14.24 11.57 5.650549
512 1000 23.94 20.35 4.968865
1024 1000 29.59 28.09 3.428613
2048 1000 30.82 30.80 1.879988
4096 1000 31.39 31.38 0.957492
8192 1000 31.46 31.46 0.479968
16384 1000 31.50 31.50 0.240301
32768 1000 31.52 31.52 0.120232
65536 1000 31.53 31.53 0.060135
131072 1000 31.53 31.53 0.030073
262144 1000 31.54 31.54 0.015037
524288 1000 31.54 31.54 0.007519
1048576 1000 31.54 31.54 0.003760
2097152 1000 31.54 31.54 0.001880
4194304 1000 31.54 31.54 0.000940
8388608 1000 31.54 31.54 0.000470

# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0010e000010ad490
System image GUID: 0x0010e000010ad493
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 31
LMC: 0
SM lid: 5
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad491
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad492
Link layer: InfiniBand
 

necr

Active Member
Dec 27, 2017
151
47
28
124
Hey WadeW,
What does iblinkinfo show? Have you ever seen FDR10 on the switch working? Connecting back-to-back (HCA to HCA), do you get a better rate?
 

WadeW

New Member
Oct 10, 2021
4
0
1
Hey WadeW,
What does iblinkinfo show? Have you ever seen FDR10 on the switch working? Connecting back-to-back (HCA to HCA), do you get a better rate?
Hey necr,

Thank you for your reply. I tried both switches available, a Mellanox M4001T (FDR10) and a 6025 (FDR), other cards have seen FDR10/FDR when connected to the switches.

I haven't tried to connect back-to-back yet and will do so later in the day. However, I did try a second port on the card and that one is good for FDR10/FDR. Very bizarre...


# iblinkinfo|grep node0
41 24[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 42 2[ ] "node0 HCA-1" ( )
1 6[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 31 1[ ] "node0 HCA-1" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
CA: node0 HCA-1:
0x0010e000010ad491 31 1[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 1 6[ ] "SwitchX - Mellanox Technologies" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
0x0010e000010ad492 42 2[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 41 24[ ] "SwitchX - Mellanox Technologies" ( )