Hi,
Like many first time posters I have been a long time lurker and have used the forum for a source of knowledge but I have run into a problem that I can't google myself out of. Thanks to everyone on here who contributes to the knowledge base.
I was looking to crank up the speed on my home network and decided to setup an Infiniband backbone after a ton of reading. I settled on MHQH29B-XTR cards and a 3036 switch. Everything has been working great so far during my bench testing and the problems I had were solved by going through this forum.
I needed some longer cables to get from the workstations to the switch so I settled on 6 active copper cables New HP 12M 4X QDR Active Quad SFP QSFP InfiniBand Copper Cable 570270-b23 but for the life of me I can't get them to connect at full speed, the almost always connect at 8 gbps instead of 32 gbps. I had two servers directly connected at first and then I put the switch between them to see if that helped and it did not improve anything.
Did I purchase the wrong cables?
I ran some diagnostics in an attempt to figure something out but all they tell me is that they are connecting at SDR instead of QDR. Please see below for the results from iblinkinfo, ibnetdiscover and ibdiagnet --get_cable_info.
Just in case I was not clear, with all the other cables I have they connect at 32 gbps.
Thanks for taking the time to read this post and for whatever help you are able to provide.
Seamus
iblinkinfo
CA: WIN-5GOTVNRHEQ4:
0x0002c903000b9501 1 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 2 1[ ] "WIN-7BUFQ8IUO1H" ( Could be 10.0 Gbps)
CA: WIN-7BUFQ8IUO1H:
0x0002c903000d7e89 2 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 1 1[ ] "WIN-5GOTVNRHEQ4" ( Could be 10.0 Gbps)
ibnetdiscover
#
# Topology file: generated on Sat Sep 24 00:53:10 2016
#
# Initiated from node 0002c903000d7e88 port 0002c903000d7e89
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000b9503
caguid=0x2c903000b9500
Ca 2 "H-0002c903000b9500" # "WIN-5GOTVNRHEQ4"
[1](2c903000b9501) "H-0002c903000d7e88"[1] (2c903000d7e89) # lid 1 lmc 0 "WIN-7BUFQ8IUO1H" lid 2 4xSDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000d7e8b
caguid=0x2c903000d7e88
Ca 1 "H-0002c903000d7e88" # "WIN-7BUFQ8IUO1H"
[1](2c903000d7e89) "H-0002c903000b9500"[1] (2c903000b9501) # lid 2 lmc 0 "WIN-5GOTVNRHEQ4" lid 1 4xSDR
Running: ibdiagnet --get_cable_info
----------
Load Plugins from:
C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ibdiagnet_plugins
(You can specify more paths to be looked in with "IBDIAGNET_PLUGINS_PATH" env variable)
Plugin Name Result Comment
libibdiagnet_cable_diag_plugin Succeeded Plugin loaded
---------------------------------------------
Discovery
-I- Fabric Discover finished successfully
-I- Discovering ... 2 nodes (0 Switches & 2 CA-s) discovered.
-I- Discovery finished successfully
-I- Duplicated GUIDs detection finished successfully
-I- Duplicated Node Description detection finished successfully
-I- Switch Info retrieving finished successfully
---------------------------------------------
Lids Check
-I- Lids Check finished successfully
---------------------------------------------
Links Check
-I- Links Check finished successfully
---------------------------------------------
Subnet Manager
-W- SM Info retrieving finished with errors
-W- S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4)/P1 - No response for MAD SMPSMInfoMad
-E- Subnet Manager Check finished with errors
-E- Not found master subnet manager in fabric
---------------------------------------------
Port Counters
-I- Ports counters retrieving finished successfully
-I- Going to sleep for 1 seconds until next counters sample
-I- Ports counters retrieving (second time) finished successfully
-I- Ports counters value Check finished successfully
-I- Ports counters Difference Check (during run) finished successfully
---------------------------------------------
Nodes Information
-I- Nodes Info retrieving finished successfully
-E- FW Check finished with errors
-E- S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4) - The firmware of this device returned invalid general info data
-E- S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H) - The firmware of this device returned invalid general info data
---------------------------------------------
Speed / Width checks
-I- Link Speed Check (Compare to supported link speed)
-E- Links Speed Check finished with errors
-E- Link: S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4)/P1<-->S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H)/P1 - Unexpected actual link speed 2.5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
-I- Link Width Check (Compare to supported link width)
-I- Links Width Check finished successfully
---------------------------------------------
Partition Keys
-I- Partition Keys retrieving finished successfully
-I- Partition Keys finished successfully
---------------------------------------------
Alias GUIDs
-I- Alias GUIDs retrieving finished successfully
-I- Alias GUIDs finished successfully
---------------------------------------------
Cable Diagnostic (Plugin)
-W- Cable Info retrieving finished with errors
-W- S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H)/P1 - No response for MAD SMPCableInfo
---------------------------------------------
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 1 1
-I- Port Counters 0 0
-I- Nodes Information 0 2
-I- Speed / Width checks 0 1
-I- Partition Keys 0 0
-I- Alias GUIDs 0 0
-I- Cable Diagnostic (Plugin) 1 0
-I- You can find detailed errors/warnings in: \tmp\ibdiagnet2.log
-I- ibdiagnet database file : \tmp\ibdiagnet2.db_csv
-I- LST file : \tmp\ibdiagnet2.lst
-I- Subnet Manager file : \tmp\ibdiagnet2.sm
-I- Ports Counters file : \tmp\ibdiagnet2.pm
-I- Nodes Information file : \tmp\ibdiagnet2.nodes_info
-I- Partition keys file : \tmp\ibdiagnet2.pkey
-I- Alias guids file : \tmp\ibdiagnet2.aguid
-I- Cables Information file : \tmp\ibdiagnet2.cables
Like many first time posters I have been a long time lurker and have used the forum for a source of knowledge but I have run into a problem that I can't google myself out of. Thanks to everyone on here who contributes to the knowledge base.
I was looking to crank up the speed on my home network and decided to setup an Infiniband backbone after a ton of reading. I settled on MHQH29B-XTR cards and a 3036 switch. Everything has been working great so far during my bench testing and the problems I had were solved by going through this forum.
I needed some longer cables to get from the workstations to the switch so I settled on 6 active copper cables New HP 12M 4X QDR Active Quad SFP QSFP InfiniBand Copper Cable 570270-b23 but for the life of me I can't get them to connect at full speed, the almost always connect at 8 gbps instead of 32 gbps. I had two servers directly connected at first and then I put the switch between them to see if that helped and it did not improve anything.
Did I purchase the wrong cables?
I ran some diagnostics in an attempt to figure something out but all they tell me is that they are connecting at SDR instead of QDR. Please see below for the results from iblinkinfo, ibnetdiscover and ibdiagnet --get_cable_info.
Just in case I was not clear, with all the other cables I have they connect at 32 gbps.
Thanks for taking the time to read this post and for whatever help you are able to provide.
Seamus
iblinkinfo
CA: WIN-5GOTVNRHEQ4:
0x0002c903000b9501 1 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 2 1[ ] "WIN-7BUFQ8IUO1H" ( Could be 10.0 Gbps)
CA: WIN-7BUFQ8IUO1H:
0x0002c903000d7e89 2 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 1 1[ ] "WIN-5GOTVNRHEQ4" ( Could be 10.0 Gbps)
ibnetdiscover
#
# Topology file: generated on Sat Sep 24 00:53:10 2016
#
# Initiated from node 0002c903000d7e88 port 0002c903000d7e89
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000b9503
caguid=0x2c903000b9500
Ca 2 "H-0002c903000b9500" # "WIN-5GOTVNRHEQ4"
[1](2c903000b9501) "H-0002c903000d7e88"[1] (2c903000d7e89) # lid 1 lmc 0 "WIN-7BUFQ8IUO1H" lid 2 4xSDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000d7e8b
caguid=0x2c903000d7e88
Ca 1 "H-0002c903000d7e88" # "WIN-7BUFQ8IUO1H"
[1](2c903000d7e89) "H-0002c903000b9500"[1] (2c903000b9501) # lid 2 lmc 0 "WIN-5GOTVNRHEQ4" lid 1 4xSDR
Running: ibdiagnet --get_cable_info
----------
Load Plugins from:
C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ibdiagnet_plugins
(You can specify more paths to be looked in with "IBDIAGNET_PLUGINS_PATH" env variable)
Plugin Name Result Comment
libibdiagnet_cable_diag_plugin Succeeded Plugin loaded
---------------------------------------------
Discovery
-I- Fabric Discover finished successfully
-I- Discovering ... 2 nodes (0 Switches & 2 CA-s) discovered.
-I- Discovery finished successfully
-I- Duplicated GUIDs detection finished successfully
-I- Duplicated Node Description detection finished successfully
-I- Switch Info retrieving finished successfully
---------------------------------------------
Lids Check
-I- Lids Check finished successfully
---------------------------------------------
Links Check
-I- Links Check finished successfully
---------------------------------------------
Subnet Manager
-W- SM Info retrieving finished with errors
-W- S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4)/P1 - No response for MAD SMPSMInfoMad
-E- Subnet Manager Check finished with errors
-E- Not found master subnet manager in fabric
---------------------------------------------
Port Counters
-I- Ports counters retrieving finished successfully
-I- Going to sleep for 1 seconds until next counters sample
-I- Ports counters retrieving (second time) finished successfully
-I- Ports counters value Check finished successfully
-I- Ports counters Difference Check (during run) finished successfully
---------------------------------------------
Nodes Information
-I- Nodes Info retrieving finished successfully
-E- FW Check finished with errors
-E- S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4) - The firmware of this device returned invalid general info data
-E- S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H) - The firmware of this device returned invalid general info data
---------------------------------------------
Speed / Width checks
-I- Link Speed Check (Compare to supported link speed)
-E- Links Speed Check finished with errors
-E- Link: S0002c903000b9503/N0002c903000b9500(WIN-5GOTVNRHEQ4)/P1<-->S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H)/P1 - Unexpected actual link speed 2.5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
-I- Link Width Check (Compare to supported link width)
-I- Links Width Check finished successfully
---------------------------------------------
Partition Keys
-I- Partition Keys retrieving finished successfully
-I- Partition Keys finished successfully
---------------------------------------------
Alias GUIDs
-I- Alias GUIDs retrieving finished successfully
-I- Alias GUIDs finished successfully
---------------------------------------------
Cable Diagnostic (Plugin)
-W- Cable Info retrieving finished with errors
-W- S0002c903000d7e8b/N0002c903000d7e88(WIN-7BUFQ8IUO1H)/P1 - No response for MAD SMPCableInfo
---------------------------------------------
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 1 1
-I- Port Counters 0 0
-I- Nodes Information 0 2
-I- Speed / Width checks 0 1
-I- Partition Keys 0 0
-I- Alias GUIDs 0 0
-I- Cable Diagnostic (Plugin) 1 0
-I- You can find detailed errors/warnings in: \tmp\ibdiagnet2.log
-I- ibdiagnet database file : \tmp\ibdiagnet2.db_csv
-I- LST file : \tmp\ibdiagnet2.lst
-I- Subnet Manager file : \tmp\ibdiagnet2.sm
-I- Ports Counters file : \tmp\ibdiagnet2.pm
-I- Nodes Information file : \tmp\ibdiagnet2.nodes_info
-I- Partition keys file : \tmp\ibdiagnet2.pkey
-I- Alias guids file : \tmp\ibdiagnet2.aguid
-I- Cables Information file : \tmp\ibdiagnet2.cables