I can, but I don't have another cable to test with on hand. I ordered new lamps referenced above and fiber cable last night, so I'll try that once they arrive.
[SOLVED]Slow speeds between two Connectx-2 machines
- Thread starter rubylaser
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I just replaced the lamps and cable with OM3 cable as suggested above by @Quasduco and I'm getting 8+Gbps now via iperf between the two machines, and 300-400 MB/s over SMB to my fileserver. Now, I'm a happy camper. Thanks to everyone for their help 
Glad to see my name next to "things worked out", and not "that no good lousy so and so"!I just replaced the lamps and cable with OM3 cable as suggested above by @Quasduco and I'm getting 8+Gbps now via iperf between the two machines, and 300-400 MB/s over SMB to my fileserver. Now, I'm a happy camper. Thanks to everyone for their help
Awesome!I just replaced the lamps and cable with OM3 cable as suggested above by @Quasduco and I'm getting 8+Gbps now via iperf between the two machines, and 300-400 MB/s over SMB to my fileserver. Now, I'm a happy camper. Thanks to everyone for their help
Which cables were you using before and if you recall which seller? Might help me, and someone else out in the future
I used this cable originally. It shipped fast and was well packaged, it just didn't work for me.Awesome!
Which cables were you using before and if you recall which seller? Might help me, and someone else out in the future
BELKIN 10M SFP+10GBASE DirectAttachPassive Twinaxial Cable from hypermicrosystems
Sure, I used these transceivers
Avago AFBR-703ASDZ-E2 10GB SFP+ 10 GB -SR -SW EMC 019-078-04 w/dust caps
And this 10 meter OM3 cable
10 Meter/33 Feet Duplex Multimode LC to LC Fiber Patch Cable - OM3 10GB Aqua
So bumping an old post.
Seems just about all google searchs about ConnectX-2 slow speeds point back to STH, there are a number of posts about this here.
I have been battling this problem for a long time now with no real effects to get me to 10G speeds. I picked up a ConnectX-3 Pro the other day to give it a tray (thanks Patrick for info on the SFP+ adapter).
System is like this.
DS1618+ with ConnectX-3 (this came with the unit)
Workstation - ConnectX-3 Pro
Switch D-Link DXS-1210-12TC
Mellanox drivers 5.50 (I have tried earlier versions also)
I am lucky to get 2G speeds, even if I direct connect workstation to NAS.
A year ago or so I picked up a pair of ConnectX-2 single port cards that came with cables which are CISCO SFP-H10GB-CU2M.
So after all this testing and trying again when I got the ConnectX-3 Pro card I came across this thread.
Is it possible that these NIC's just don't like the cables I have ?
Is it worth while to grab a set of these ?
Cisco SFP+ Copper Twinax Cable - Twinaxial 5 m for Catalyst 2960, 2960-24, 2960-48, 2960G-24, 2960G-48, 2960S-24, 2960S-48 | eBay
Or maybe some settings on my Workstation need to be changed, motherboard is a Z10PE-D16 WS, OS Win 10 Pro.
The card is in the first slot which is a PCIe Gen3 x8 and SR-IOV is enabled in the BIOS.
All LEDs on the NIC's and Switch show green (10G).
Seems just about all google searchs about ConnectX-2 slow speeds point back to STH, there are a number of posts about this here.
I have been battling this problem for a long time now with no real effects to get me to 10G speeds. I picked up a ConnectX-3 Pro the other day to give it a tray (thanks Patrick for info on the SFP+ adapter).
System is like this.
DS1618+ with ConnectX-3 (this came with the unit)
Workstation - ConnectX-3 Pro
Switch D-Link DXS-1210-12TC
Mellanox drivers 5.50 (I have tried earlier versions also)
I am lucky to get 2G speeds, even if I direct connect workstation to NAS.
A year ago or so I picked up a pair of ConnectX-2 single port cards that came with cables which are CISCO SFP-H10GB-CU2M.
So after all this testing and trying again when I got the ConnectX-3 Pro card I came across this thread.
Is it possible that these NIC's just don't like the cables I have ?
Is it worth while to grab a set of these ?
Cisco SFP+ Copper Twinax Cable - Twinaxial 5 m for Catalyst 2960, 2960-24, 2960-48, 2960G-24, 2960G-48, 2960S-24, 2960S-48 | eBay
Or maybe some settings on my Workstation need to be changed, motherboard is a Z10PE-D16 WS, OS Win 10 Pro.
The card is in the first slot which is a PCIe Gen3 x8 and SR-IOV is enabled in the BIOS.
All LEDs on the NIC's and Switch show green (10G).
With actual workloads? Or iperf?
On windows I never really got good results in iperf on the windows machines, but had no problems to get ~ 9gbit/s transfers between workstations with iodrive2s.
Those numbers look about right for a single stream iperf test from windows. I think I got something close to that before I switched to using 8 parallel streams, in parallel I got very close to full line speed.
Last edited:
Also, with jumbo frames I lost a significant amount of bandwidth on a direct connection between two cards, I think I dropped from 39Gb/s to like 14Gb/s with jumbo frames on. If you're using 9k MTU test with 8 stream iperf and test 1500 MTU too.
Last edited:
You need to use -P 8 on the iperf client command line. That's it, don't mess with nic RSS settings.
Yup, that should do it. I don't remember if you need to run the server with -P 8 too, I think I had to for one of the iperfs (iperf or iperf3) but I can't recall which.
I didn't need to specify window size, I just let it rip with the defaults.
There is no need to run multithreaded iperf tests. ConnectX-2 and ConnectX-3 is capable of maxing at 9.8Gbps single stream. Just make sure your PC is up to the task at it takes some CPU power. I had an issue with my old PC with Core i7 2600k OC-ed to 4.3GHz where just opening up MPC-HC (video player) made the test drop to 4.8Gbps for RX test. Do make sure you have recent firmware on them. I didn't get full speed on my ConnectX-3 until it was upgraded.
Here's iperf3 with 8 streams (2 threads according to top):
Here's iperf3 non-parallel:
The CPU on the receiving linux machine bottlenecked the 8 stream transfer to ~35Gb/s, that's the maximum I was able to get out of that point to point link. 8 streams gets the job done without needing ludicrous single thread performance to do it.
The server is a G4400 Pentium, pushing 35Gb/s.
Last edited: