[SOLVED]Slow speeds between two Connectx-2 machines

rubylaser · Mar 16, 2016

Any other ideas? This is unusabley slow. Here are the two cards both in x8 electrical slots. One system is an e5-2670 in a Supermicro X9SRL-F with 32GB of RAM with Windows 10 64-bit and the 5.10 drivers. The other a dual e5-2670 system in an Intel S2600CP with 64GB of RAM on Ubuntu 14.04.4 64-bit. I have even tried setting the MTU to 9000 on both ends with no benefit. This is turd slow...

Code:

C:\Users\rubylaser\Downloads\iperf-3.0.11-win64>iperf3.exe -B 192.168.50.2 -c 192.168.50.3
Connecting to host 192.168.50.3, port 5201
[  4] local 192.168.50.2 port 57282 connected to 192.168.50.3 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[  4]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec
[  4]   2.00-3.00   sec  1.50 MBytes  12.6 Mbits/sec
[  4]   3.00-4.00   sec   640 KBytes  5.24 Mbits/sec
[  4]   4.00-5.00   sec  4.62 MBytes  38.7 Mbits/sec
[  4]   5.00-6.00   sec  5.62 MBytes  47.3 Mbits/sec
[  4]   6.00-7.00   sec  1.88 MBytes  15.7 Mbits/sec
[  4]   7.00-8.00   sec  6.62 MBytes  55.6 Mbits/sec
[  4]   8.00-9.00   sec  1.38 MBytes  11.5 Mbits/sec
[  4]   9.00-10.00  sec  5.38 MBytes  45.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  28.2 MBytes  23.7 Mbits/sec                  sender
[  4]   0.00-10.00  sec  28.1 MBytes  23.5 Mbits/sec                  receiver

iperf Done.

T_Minus · Mar 16, 2016

wow! @rubylaser you'd think there wouldn't be some secret to get them to work at even 50% let alone better than that terrible speed

rubylaser · Mar 17, 2016

T_Minus said:
wow! @rubylaser you'd think there wouldn't be some secret to get them to work at even 50% let alone better than that terrible speed

Yeah, this is my first experiment with 10GBe and it is a complete failure. At least the gigabit connections between these two boxes are running at full speed.

At this point, I'm not sure what else to try.

nephri · Mar 17, 2016

Do you think i'ts software or hardware ?

What is your approach to disctinct between theses 2 categories of issues ?
I think you can :
- can you try by example with 2 windows PC (if it works fine, it's not hardware but driver/linux conf) ?
- swap with another DAC cable
- check with another NIC ?

uplinks was negociated at which speed and mode ?

rubylaser · Mar 17, 2016

nephri said:
Do you think i'ts software or hardware ?

What is your approach to disctinct between theses 2 categories of issues ?
I think you can :
- can you try by example with 2 windows PC (if it works fine, it's not hardware but driver/linux conf) ?
- swap with another DAC cable
- check with another NIC ?

uplinks was negociated at which speed and mode ?

Thanks for the ideas

I guessing this is a hardware issue and I'm thinking it's the long 10m DAC passive cable, but I don't want to just throw money at the problem either.

1. Unfortunately, in the last month I have sold all of my extra hardware (two servers and donated a PC to the church), so I just have a bunch of laptops, HTPC's, and phones/tablets around the house, but no extra workstations or servers beyond these two (I guess I could take my work machine home though).

2. I'd need to buy another DAC cable to test. Although, I don't really want to spend an additional $30+ on a 10m cable if it's not going to work anyway.

3. I don't have an extra Connectx-2 card either.

4. The links are negotiated at 10GBe and 9000 MTU.

Code:

root@fileserver:~# ethtool p257p1
Settings for p257p1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes

BackupProphet · Mar 17, 2016

Have you tried Ubuntu to Ubuntu? If that is working, then you can eliminate hardware issues.

rubylaser · Mar 17, 2016

BackupProphet said:
Have you tried Ubuntu to Ubuntu? If that is working, then you can eliminate hardware issues.

I have not, but I can easily test that with the live CD on my workstation tonight. I'll let you know how it goes

marv · Mar 17, 2016

If the issue is cable, I would expect error counter on interfaces to rise.

rubylaser · Mar 17, 2016

BackupProphet said:
Have you tried Ubuntu to Ubuntu? If that is working, then you can eliminate hardware issues.

I just tried Ubuntu to Ubuntu between the two machines and was still left with terrible results

No errors in ifconfig on either device either. I don't know what to try next as I don't have duplicates of any of this hardware.

BackupProphet · Mar 17, 2016

try with iperf2

I also get horrible results with iperf3

rubylaser · Mar 17, 2016

BackupProphet said:
try with iperf2

I also get horrible results with iperf3

Thanks! I tried iperf2 and indeed I did see great speeds in one direction with iperf2. From my dual e5-2670 server acting as the server to my single e5-2670 system, I was getting 9+GB/s. In the other direction I was getting 3-4 MB/s.

Also, actual file transfers from ramdisk to ramdisk on each side was only yielding 400 - 500 KB/s, so something is obviously not working correctly.

Quasduco · Mar 17, 2016

rubylaser said:
I just tried Ubuntu to Ubuntu between the two machines and was still left with terrible results No errors in ifconfig on either device either. I don't know what to try next as I don't have duplicates of any of this hardware.

Not a small amount of retransmits (Retr) on a small 10 second test. I would go fiber, personally - I think I did mine for 10/bulb, and fiber for $2/meter for OM3. Good luck!

rubylaser · Mar 17, 2016

Quasduco said:
Not a small amount of retransmits (Retr) on a small 10 second test. I would go fiber, personally - I think I did mine for 10/bulb, and fiber for $2/meter for OM3. Good luck!

Thanks for the ideas. Would you be willing to provide some links to parts that I could order?

Stereodude · Mar 17, 2016

Have you tried a much shorter DAC?

rubylaser · Mar 17, 2016

Stereodude said:
Have you tried a much shorter DAC?

I can't do any shorter than 10m. The server is in my basement and the workstation is on the first floor. 10 meters barely reaches [emoji15]

Quasduco · Mar 17, 2016

rubylaser said:
Thanks for the ideas. Would you be willing to provide some links to parts that I could order?

Bulbs - Avago AFBR-703ASDZ-E2 10GB SFP+ 10 GB -SR -SW EMC 019-078-04 w/dust caps - bet you could offer $10 each for a few...

Fiber - 2M OM3 LC/LC Duplex Multimode 10gb Fiber Cable Aqua - I bought from them, $5 per 2m on a few...

Good luck, hope that helps!

DISCLAIMER - I am not these sellers, and I use Intel gear, so not a Connectx expert, make 100% sure they are compatible. A quick google suggested the Connectx cards are not at all fussy...

Quasduco · Mar 17, 2016

rubylaser said:
I can't do any shorter than 10m. The server is in my basement and the workstation is on the first floor. 10 meters barely reaches [emoji15]

You WOULD have to say that after I had started posting my reply...

Well, here is a good deal on 10m - 10 Meter/33 Feet Duplex Multimode LC to LC Fiber Patch Cable - OM3 10GB Aqua

20m - Cablesys LC-LC Fiber Optic Patch Cord Duplex 10 Gig Multimode OM3 Aqua Cable 20M

Kanedo · Mar 17, 2016

I have two Connect-X2 installed in two systems.
I also had poor performance until I finally realized my DAC was bad. It would only get good performance when bent a certain way.
I replaced the DAC with a new one and all problems went away. Now able to consistently hit 9.8Gbit both ways using iperf.

This is the one I got from eBay that works now.

NEW Cisco Original SFP-H10GB-CU5M 5 Meter Twinax 10GB Cable Used Bulk Lot Tested

Jerry Renwick · Mar 18, 2016

rubylaser said:
Well something went horribly wrong. I just setup my second 2670 system and put the other card in a x8 slot and the iperf speeds between my hosts are now about 3.5 MB/s (terrible). This has really been a failed experiment so far

oh, sorry to hear that.

Stereodude · Mar 18, 2016

rubylaser said:
I can't do any shorter than 10m. The server is in my basement and the workstation is on the first floor. 10 meters barely reaches [emoji15]

You can't move one of them close to the other for the sake of testing/debugging?

[SOLVED]Slow speeds between two Connectx-2 machines

Active Member

Build. Break. Fix. Repeat

Active Member

Active Member

Active Member

Well-Known Member

Active Member

Active Member

Active Member

Well-Known Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

New Member

Active Member

Active Member