Testing Results: Mellanox ConnectX-3 and achievable link speeds

NablaSquaredG

Active Member
Aug 17, 2020
225
92
28
Hey,

It is known that Mellanox hardware is generally not picky when it comes to transceivers or cables - however there are some subtle issues when it comes to the maximum link speed (56G FDR).

Below some test results to make sure you don't buy wrong cables:

NameLink Speed InfinibandLink Speed Ethernet
Generic 40G QSFP+ cable (FS.com)40 Gb/sec (4X QDR)40 Gb/sec (4X QDR)
MC2207130-002 (Genuine 2m Mellanox FDR cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MC2201728-003 (Genuine 3m Mellanox FDR cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MCP1600-C003 (Genuine 3m Mellanox 100GBe cable)40 Gb/sec (4X FDR10)56 Gb/sec (4X FDR)
MCP1600-E003 (Genuine 3m Mellanox IB EDR (100Gb) cable)56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
MFA1A00-E020: Genuine 20m Mellanox IB EDR active optical cable56 Gb/sec (4X FDR)56 Gb/sec (4X FDR)
Intel 948435: Generic 100G QSFP28 cable (possibly Dell 0G0WYG)40 Gb/sec (4X QDR)56 Gb/sec (4X FDR)

I've highlighted some noteworthy results.
So, to sum things um:
  • Genuine Mellanox FDR cables behave as expected (Support FDR 56G for both Ethernet / IB)
  • QSFP+ 40G cables will generally only give you 40G
  • Mellanox is PICKY when it comes to IB speeds! Only specifically marked IB cables give you will speed with IB
    • Non-Mellanox cables seem to only give you 40G QDR!
    • While the card recognises that the 100G Mellanox Ethernet cable (C003) supports FDR (which is required for 100GBe) and uses it (to reduce overhead, which already gives you a nice speed increase), it refuses to increase the link speed to 14G instead of 10G, because it is not a specifically marked IB cable
  • Mellanox IB cables also give you full speed with Ethernet, but not vice-versa!
  • Mellanox is not picky when it comes to Ethernet speeds

All tests were done using Mellanox ConnectX-3 (CX354A).
Big thanks to @Rand__ for lending me the last three cables from the list!
 
Last edited: