Hi everyone,
I cross flashed a Oracle OEM 7046442 (354A-QCBT) to CX354A-FCBT with the following command. It is connected to a FDR10 switch, which then gos to a bunch of CX354A-FCBT, either native or flashed from QCBT.
/usr/bin/flint -d /dev/mst/mt4099_pciconf0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin --allow_psid_change burn
However, ib_read_bw gives only 31.54Gb/s, ibstat also reads QDR speed only. I switched the cables already, nothing had changed. Any advice is appreciated.
# mlxfwmanager -u -d 01:00.0 --query
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: 01:00.0
Port1 GUID: 0010e000010ad491
Port2 GUID: 0010e000010ad492
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752
Status: Up to date
#ib_read_bw -a -F 172.168.168.1 -d mlx4_0 --report_gbits
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1f QPN 0x0218 PSN 0x875a34 OUT 0x10 RKey 0x010100 VAddr 0x007fa49ce5e000
remote address: LID 0x10 QPN 0x29f6b PSN 0x2432a6 OUT 0x10 RKey 0x20411237 VAddr 0x002b29cd000000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.124028 0.077789 4.861819
4 1000 0.25 0.21 6.715182
8 1000 0.47 0.44 6.819802
16 1000 0.99 0.78 6.130253
32 1000 1.98 1.58 6.167505
64 1000 3.85 3.16 6.163159
128 1000 7.72 6.32 6.170413
256 1000 14.24 11.57 5.650549
512 1000 23.94 20.35 4.968865
1024 1000 29.59 28.09 3.428613
2048 1000 30.82 30.80 1.879988
4096 1000 31.39 31.38 0.957492
8192 1000 31.46 31.46 0.479968
16384 1000 31.50 31.50 0.240301
32768 1000 31.52 31.52 0.120232
65536 1000 31.53 31.53 0.060135
131072 1000 31.53 31.53 0.030073
262144 1000 31.54 31.54 0.015037
524288 1000 31.54 31.54 0.007519
1048576 1000 31.54 31.54 0.003760
2097152 1000 31.54 31.54 0.001880
4194304 1000 31.54 31.54 0.000940
8388608 1000 31.54 31.54 0.000470
# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0010e000010ad490
System image GUID: 0x0010e000010ad493
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 31
LMC: 0
SM lid: 5
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad491
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad492
Link layer: InfiniBand
I cross flashed a Oracle OEM 7046442 (354A-QCBT) to CX354A-FCBT with the following command. It is connected to a FDR10 switch, which then gos to a bunch of CX354A-FCBT, either native or flashed from QCBT.
/usr/bin/flint -d /dev/mst/mt4099_pciconf0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin --allow_psid_change burn
However, ib_read_bw gives only 31.54Gb/s, ibstat also reads QDR speed only. I switched the cables already, nothing had changed. Any advice is appreciated.
# mlxfwmanager -u -d 01:00.0 --query
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: 01:00.0
Port1 GUID: 0010e000010ad491
Port2 GUID: 0010e000010ad492
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752
Status: Up to date
#ib_read_bw -a -F 172.168.168.1 -d mlx4_0 --report_gbits
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1f QPN 0x0218 PSN 0x875a34 OUT 0x10 RKey 0x010100 VAddr 0x007fa49ce5e000
remote address: LID 0x10 QPN 0x29f6b PSN 0x2432a6 OUT 0x10 RKey 0x20411237 VAddr 0x002b29cd000000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.124028 0.077789 4.861819
4 1000 0.25 0.21 6.715182
8 1000 0.47 0.44 6.819802
16 1000 0.99 0.78 6.130253
32 1000 1.98 1.58 6.167505
64 1000 3.85 3.16 6.163159
128 1000 7.72 6.32 6.170413
256 1000 14.24 11.57 5.650549
512 1000 23.94 20.35 4.968865
1024 1000 29.59 28.09 3.428613
2048 1000 30.82 30.80 1.879988
4096 1000 31.39 31.38 0.957492
8192 1000 31.46 31.46 0.479968
16384 1000 31.50 31.50 0.240301
32768 1000 31.52 31.52 0.120232
65536 1000 31.53 31.53 0.060135
131072 1000 31.53 31.53 0.030073
262144 1000 31.54 31.54 0.015037
524288 1000 31.54 31.54 0.007519
1048576 1000 31.54 31.54 0.003760
2097152 1000 31.54 31.54 0.001880
4194304 1000 31.54 31.54 0.000940
8388608 1000 31.54 31.54 0.000470
# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0010e000010ad490
System image GUID: 0x0010e000010ad493
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 31
LMC: 0
SM lid: 5
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad491
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x0010e000010ad492
Link layer: InfiniBand