Twinax Infiniband Cables - Speed Issue

Spartus

Active Member
Mar 28, 2012
313
119
43
Toronto, Canada
So my cluster uses QDR Infiniband (Mellanox Connectx-2 cards). To date I had bought all 2 m passive twinax cables. These got me consistently 3300+ MB/s and around a 1.2 to 1.3 microsecond latency

latency = 1.3766 microseconds
bytes bandwidth MB/s
----- --------------
8 5.8113
1024 380.1697
4096 910.6501
16384 1899.7975
65536 2622.4429
262144 3052.9990
1048576 3309.7010
4194304 3214.4822
MPI Test has successfully completed!

My rack started to overflow so I got a second one, and was going to space them out so I bought 8 x active 10m cables. Still QDR rated cables, I checked.

Now the speed is only about 950 MB/s and latency is up to 1.6 to 1.7 ms. Latency I an understand, but that speed is only 29% of what I was getting before, and now down into 10gbit territory. I checked 2 different nodes with 2 different cables, it seems consistent... the active cables are much slower.

So, question to you all. Do you see a difference between active and passive copper cables? Is this normal? did I get a bad batch? Would fibre cable fix this?

I could probably get by with 5m passive cables, but I would likely splurge for 10-15m fibre if I wasn't worried about throwing more good money after bad. I don't think I'll go for QSFP+ transceivers, but just the fully integrated options since I believe they are much cheaper. Any advice?
 

i386

Well-Known Member
Mar 18, 2016
3,521
1,210
113
33
Germany
If your run "ibdiagnet --get_cable_info" and open the ibdiagnet2.log what does it say the speed of the cable?

Example of a ibdiagnet2.log file
Code:
-------------------------------------------------------

Port=2 Lid=0x0001 GUID=0x0002c90300efe952 Port Name=gen-l-vrt-005/U1/P2

-------------------------------------------------------

Vendor: Mellanox

OUI: 0x2c9

PN: MC2207130-001

SN: MT1323VS05817

Rev: A3

Length: 1 m

Type: Copper cable- unequalized

SupportedSpeed: SDR/DDR/QDR/FDR
 

Spartus

Active Member
Mar 28, 2012
313
119
43
Toronto, Canada
I really wish i could run that command, but my ibdiagnet is too old it seems. I'll setup a test machine soon to do this unless you can give another way.

I have serial access to the switch. It confirms the sticker on the cable is accurate, but it also confirms that they are connecting at 4xSDR speed.

Other info that may be relevant.
My 2m cables are a mix of mellanox, which the switch claims are qsfp+, and Gore, which are only qsfp, but both are QDR speed.
The 10m cables are amphenol, listed as QSFP on the switch console, and QDR speed online.
HP Infiniband 4X QDR Active QSFP Copper Cable Specs

any way to check more info able cable on voltaire switch. Any ability for force port speed and test it?

Thanks a bunch. The advice was helpful, but out of reach currently.
 

Mark Wilde

New Member
Aug 21, 2016
18
5
3
36
The difference between active and passive copper cables is the length and signal strength.

Passive twinax cables is being used between the server and the Top of Rack (ToR) switch within 5m. The upside in this configuration is that the passive twinax cabling connection is much cheaper than the cost of an optical link. The downside is that you are limited in distance and there's also some cable interoperability issue you’ll need to deal with.

When the distance exceeds 5m, it is highly recommended to use active cables to ensure signal is transferred all the way through. The downside is that they are more expensive and use more power. The upside is that you don't have to worry about distance (up to 300 meters) and, perhaps more importantly, you don't have to worry about which vendor's cable you use and the signal is improved and gives peace of mind by creating a trustworthy connection.

For more detailed info, please see this article.