So my cluster uses QDR Infiniband (Mellanox Connectx-2 cards). To date I had bought all 2 m passive twinax cables. These got me consistently 3300+ MB/s and around a 1.2 to 1.3 microsecond latency
latency = 1.3766 microseconds
bytes bandwidth MB/s
----- --------------
8 5.8113
1024 380.1697
4096 910.6501
16384 1899.7975
65536 2622.4429
262144 3052.9990
1048576 3309.7010
4194304 3214.4822
MPI Test has successfully completed!
My rack started to overflow so I got a second one, and was going to space them out so I bought 8 x active 10m cables. Still QDR rated cables, I checked.
Now the speed is only about 950 MB/s and latency is up to 1.6 to 1.7 ms. Latency I an understand, but that speed is only 29% of what I was getting before, and now down into 10gbit territory. I checked 2 different nodes with 2 different cables, it seems consistent... the active cables are much slower.
So, question to you all. Do you see a difference between active and passive copper cables? Is this normal? did I get a bad batch? Would fibre cable fix this?
I could probably get by with 5m passive cables, but I would likely splurge for 10-15m fibre if I wasn't worried about throwing more good money after bad. I don't think I'll go for QSFP+ transceivers, but just the fully integrated options since I believe they are much cheaper. Any advice?
latency = 1.3766 microseconds
bytes bandwidth MB/s
----- --------------
8 5.8113
1024 380.1697
4096 910.6501
16384 1899.7975
65536 2622.4429
262144 3052.9990
1048576 3309.7010
4194304 3214.4822
MPI Test has successfully completed!
My rack started to overflow so I got a second one, and was going to space them out so I bought 8 x active 10m cables. Still QDR rated cables, I checked.
Now the speed is only about 950 MB/s and latency is up to 1.6 to 1.7 ms. Latency I an understand, but that speed is only 29% of what I was getting before, and now down into 10gbit territory. I checked 2 different nodes with 2 different cables, it seems consistent... the active cables are much slower.
So, question to you all. Do you see a difference between active and passive copper cables? Is this normal? did I get a bad batch? Would fibre cable fix this?
I could probably get by with 5m passive cables, but I would likely splurge for 10-15m fibre if I wasn't worried about throwing more good money after bad. I don't think I'll go for QSFP+ transceivers, but just the fully integrated options since I believe they are much cheaper. Any advice?