Testing Results: Mellanox ConnectX-3 and achievable link speeds

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
Hey,

It is known that Mellanox hardware is generally not picky when it comes to transceivers or cables - however there are some subtle issues when it comes to the maximum link speed (56G FDR).

Below some test results to make sure you don't buy wrong cables:

NameLink Speed InfinibandLink Speed Ethernet
Generic 40G QSFP+ cable (FS.com)40 Gb/sec (4X QDR)40 Gb/sec (4X QDR)
MC2207130-002 (Genuine 2m Mellanox FDR cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MC2201728-003 (Genuine 3m Mellanox FDR cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MCP1600-C003 (Genuine 3m Mellanox 100GBe cable)40 Gb/sec (4X FDR10)56 Gb/sec (4X FDR)
MCP1600-E003 (Genuine 3m Mellanox IB EDR (100Gb) cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MFA1A00-E020: Genuine 20m Mellanox IB EDR active optical cable56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
Intel 948435: Generic 100G QSFP28 cable (possibly Dell 0G0WYG)40 Gb/sec (4X QDR)56 Gb/sec (4X FDR)

I've highlighted some noteworthy results.
So, to sum things um:
  • Genuine Mellanox FDR cables behave as expected (Support FDR 56G for both Ethernet / IB)
  • QSFP+ 40G cables will generally only give you 40G
  • Mellanox is PICKY when it comes to IB speeds! Only specifically marked IB cables give you will speed with IB
    • Non-Mellanox cables seem to only give you 40G QDR!
    • While the card recognises that the 100G Mellanox Ethernet cable (C003) supports FDR (which is required for 100GBe) and uses it (to reduce overhead, which already gives you a nice speed increase), it refuses to increase the link speed to 14G instead of 10G, because it is not a specifically marked IB cable
  • Mellanox IB cables also give you full speed with Ethernet, but not vice-versa!
  • Mellanox is not picky when it comes to Ethernet speeds

All tests were done using Mellanox ConnectX-3 (CX354A).
Big thanks to @Rand__ for lending me the last three cables from the list!
 
Last edited: