[SOLVED]Slow speeds between two Connectx-2 machines

Discussion in 'Networking' started by rubylaser, Mar 1, 2016.

  1. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    I can, but I don't have another cable to test with on hand. I ordered new lamps referenced above and fiber cable last night, so I'll try that once they arrive.
     
    #41
  2. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    I just replaced the lamps and cable with OM3 cable as suggested above by @Quasduco and I'm getting 8+Gbps now via iperf between the two machines, and 300-400 MB/s over SMB to my fileserver. Now, I'm a happy camper. Thanks to everyone for their help :)
     
    #42
    gigatexal and Jon Massey like this.
  3. Quasduco

    Quasduco Active Member

    Joined:
    Nov 16, 2015
    Messages:
    125
    Likes Received:
    46
    Glad to see my name next to "things worked out", and not "that no good lousy so and so"!
     
    #43
    ethera and Jannis Jacobsen like this.
  4. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,888
    Likes Received:
    1,511
    Awesome!

    Which cables were you using before and if you recall which seller? Might help me, and someone else out in the future :) :)
     
    #44
  5. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    I used this cable originally. It shipped fast and was well packaged, it just didn't work for me.

    BELKIN 10M SFP+10GBASE DirectAttachPassive Twinaxial Cable from hypermicrosystems
     
    #45
    T_Minus likes this.
  6. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    550
    Likes Received:
    20
    Hi Rubylaser,
    May I know exactly which adapters you used and the OM3 cable that came together for you?
     
    #46
  7. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    #47
    epicurean, herby and T_Minus like this.
  8. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    Thanks again for your help :)
     
    #48
  9. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    762
    Likes Received:
    236
    So bumping an old post.
    Seems just about all google searchs about ConnectX-2 slow speeds point back to STH, there are a number of posts about this here.

    I have been battling this problem for a long time now with no real effects to get me to 10G speeds. I picked up a ConnectX-3 Pro the other day to give it a tray (thanks Patrick for info on the SFP+ adapter).

    System is like this.
    DS1618+ with ConnectX-3 (this came with the unit)
    Workstation - ConnectX-3 Pro
    Switch D-Link DXS-1210-12TC
    Mellanox drivers 5.50 (I have tried earlier versions also)

    I am lucky to get 2G speeds, even if I direct connect workstation to NAS.

    A year ago or so I picked up a pair of ConnectX-2 single port cards that came with cables which are CISCO SFP-H10GB-CU2M.

    So after all this testing and trying again when I got the ConnectX-3 Pro card I came across this thread.

    Is it possible that these NIC's just don't like the cables I have ?
    Is it worth while to grab a set of these ?
    Cisco SFP+ Copper Twinax Cable - Twinaxial 5 m for Catalyst 2960, 2960-24, 2960-48, 2960G-24, 2960G-48, 2960S-24, 2960S-48 | eBay

    Or maybe some settings on my Workstation need to be changed, motherboard is a Z10PE-D16 WS, OS Win 10 Pro.
    The card is in the first slot which is a PCIe Gen3 x8 and SR-IOV is enabled in the BIOS.
    All LEDs on the NIC's and Switch show green (10G).

    iperf results new.JPG
     
    #49
  10. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,715
    Likes Received:
    425
    With actual workloads? Or iperf?

    On windows I never really got good results in iperf on the windows machines, but had no problems to get ~ 9gbit/s transfers between workstations with iodrive2s.
     
    #50
  11. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Those numbers look about right for a single stream iperf test from windows. I think I got something close to that before I switched to using 8 parallel streams, in parallel I got very close to full line speed.
     
    #51
    Last edited: Sep 15, 2018
  12. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    762
    Likes Received:
    236
    Workloads are nothing special. Something like a DVD ISO copy from the NAS to workstation takes like a minute, under 10G it should be like 10 seconds.

    Let me look at how many threads are being used.
     
    #52
  13. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    762
    Likes Received:
    236
    Max number of RSS Processors is set to 8
    RSS Base Processor Number is set to 8

    I think those are the right ones to change ?

    This showed no difference is speed. I will restart when I can and see if that fixes it.
     
    #53
  14. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Also, with jumbo frames I lost a significant amount of bandwidth on a direct connection between two cards, I think I dropped from 39Gb/s to like 14Gb/s with jumbo frames on. If you're using 9k MTU test with 8 stream iperf and test 1500 MTU too.
     
    #54
    Last edited: Sep 15, 2018
  15. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    You need to use -P 8 on the iperf client command line. That's it, don't mess with nic RSS settings.
     
    #55
  16. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    762
    Likes Received:
    236
    I used this
    iperf3.exe -w 1024k -P 8 -i 2 -t 30 -c 10.0.0.198
     
    #56
  17. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Yup, that should do it. I don't remember if you need to run the server with -P 8 too, I think I had to for one of the iperfs (iperf or iperf3) but I can't recall which.

    I didn't need to specify window size, I just let it rip with the defaults.
     
    #57
  18. saivert

    saivert Member

    Joined:
    Nov 2, 2015
    Messages:
    118
    Likes Received:
    12
    There is no need to run multithreaded iperf tests. ConnectX-2 and ConnectX-3 is capable of maxing at 9.8Gbps single stream. Just make sure your PC is up to the task at it takes some CPU power. I had an issue with my old PC with Core i7 2600k OC-ed to 4.3GHz where just opening up MPC-HC (video player) made the test drop to 4.8Gbps for RX test. Do make sure you have recent firmware on them. I didn't get full speed on my ConnectX-3 until it was upgraded.
     
    #58
  19. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Here's iperf3 with 8 streams (2 threads according to top):
    iperf 40gb 5m SAS.png iperf 40gb 2m SAS.png

    Here's iperf3 non-parallel:
    iperf3 CX3 netapp SAS 5m mtu 1500.png iperf3 CX3 netapp SAS 2m mtu 1500.png

    The CPU on the receiving linux machine bottlenecked the 8 stream transfer to ~35Gb/s, that's the maximum I was able to get out of that point to point link. 8 streams gets the job done without needing ludicrous single thread performance to do it.

    The server is a G4400 Pentium, pushing 35Gb/s.
     
    #59
    Last edited: Sep 15, 2018
  20. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    762
    Likes Received:
    236
    WOW very nice.

    Like I said above, my system is Z10PE-D16 WS, OS Win 10 Pro. CPU's are 2x E5-2699v3 and 256GB RAM installed.
    Although these are not fast CPU's there is a lot of cores/threads.
     
    #60
Similar Threads: [SOLVED]Slow speeds
Forum Title Date
Networking Fluctuating speeds on 40gb Mellanox Aug 4, 2019
Networking Different Speeds from opposite ends of a VPN [Diagram]? Oct 2, 2018
Networking Can't get 10Gbit speeds on Intel X550-T1 Jul 3, 2018
Networking ConnectX-3s Work with 10, 40 and 56 speeds? Oct 10, 2017
Networking Not getting 10gbe Speeds Jul 14, 2017

Share This Page