Poor 10gb Performance - Mellanox X-2 + UBNT US‑16‑XG

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
waith

waith

Member
Aug 3, 2015
44
4
8
44
I was wondering if anyone had experience tuning the Mellanox cards for 10gb performance. I seem to be having an issue where sending is very slow, but receiving is considerably better. I am using the 5.35.12978.0 drivers on Win10 Client and 2016 Server. Firmware has been updated to 2.9.1200 on both devices. Send and receive buffers maxed to 4096. Jumbo frames not enabled.

Sending

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 68.4 MBytes 573 Mbits/sec
[ 4] 1.00-2.00 sec 136 MBytes 1.14 Gbits/sec
[ 4] 2.00-3.00 sec 218 MBytes 1.83 Gbits/sec
[ 4] 3.00-4.00 sec 226 MBytes 1.89 Gbits/sec
[ 4] 4.00-5.00 sec 202 MBytes 1.69 Gbits/sec
[ 4] 5.00-6.00 sec 164 MBytes 1.38 Gbits/sec
[ 4] 6.00-7.00 sec 199 MBytes 1.67 Gbits/sec
[ 4] 7.00-8.00 sec 227 MBytes 1.90 Gbits/sec
[ 4] 8.00-9.00 sec 231 MBytes 1.94 Gbits/sec
[ 4] 9.00-10.00 sec 232 MBytes 1.95 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.86 GBytes 1.60 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.86 GBytes 1.60 Gbits/sec receiver

Receiving

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 680 MBytes 5.70 Gbits/sec
[ 4] 1.00-2.00 sec 789 MBytes 6.62 Gbits/sec
[ 4] 2.00-3.00 sec 788 MBytes 6.61 Gbits/sec
[ 4] 3.00-4.00 sec 792 MBytes 6.64 Gbits/sec
[ 4] 4.00-5.00 sec 792 MBytes 6.64 Gbits/sec
[ 4] 5.00-6.00 sec 791 MBytes 6.64 Gbits/sec
[ 4] 6.00-7.00 sec 800 MBytes 6.71 Gbits/sec
[ 4] 7.00-8.00 sec 798 MBytes 6.69 Gbits/sec
[ 4] 8.00-9.00 sec 800 MBytes 6.71 Gbits/sec
[ 4] 9.00-10.00 sec 804 MBytes 6.75 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.65 GBytes 6.57 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.65 GBytes 6.57 Gbits/sec receiver
 
P

PigLover

Moderator
Jan 26, 2011
3,186
1,545
113
Need a lot more info about the configuration in order to be helpful. A few questions to help you debug this:

- What are you connecting the Mellanox card to the switch with? DAC (how long)? Fiber (what SFP+)?
- What's on the other side of this test (is the machine on the other end using Mellanox cards too)?
- You don't have Jumbo frames enabled - which is OK - but have you tested with them
--- Note: Jumbo frames would normally give "the last 5%" kind of improvement - but you need 5x improvement before that's even interesting.
- What MB are you using? Is the CX2 card plugged into an x8 slot? Is it really x8 (lots of MBs have "x4 in x8 physical" slots)?
 
waith

waith

Member
Aug 3, 2015
44
4
8
44
What are you connecting the Mellanox card to the switch with? DAC (how long)? Fiber (what SFP+)?
10GTek SPF+ (for Mellanox and UBNT) and 10m OM3 runs to each machine.

Amazon.com: 10Gtek for Mellanox MFM1T02A-SR, 10Gb/s SFP+ Transceiver module, MMF, 850nm, 300-meter: Computers & Accessories

Amazon.com: 10Gtek for Ubiquiti Transceiver, 10Gb SFP+ module 10GBase-SR, MMF, 850nm, 300-meter: Home Improvement

What's on the other side of this test (is the machine on the other end using Mellanox cards too)?
Mellanox ConnectX-2 in both machines

You don't have Jumbo frames enabled - which is OK - but have you tested with them
I haven't tested with them yet. Right now I'm less concerned about saturating the link than I am about having consistent send/receive speeds

What MB are you using? Is the CX2 card plugged into an x8 slot? Is it really x8 (lots of MBs have "x4 in x8 physical" slots)?
Server is Asrock ep2c602-4l/d16 - Dual 2670v1
Client is ASUS Maximus Gene VI - Core i7-4770K
 
D

djflow195

Member
Jan 1, 2016
42
9
8
What iperf options are you using for the test? Try setting the tcp window size to 1MB and test again. What is the CPU on each side (client/server)? Reverse the iperf role and try again (client/server).
 
wildchild

wildchild

Active Member
Feb 4, 2014
389
57
28
Running a rather similar setup, and running fine mellanox > mellanox , and mellanox > intel x520-da2.

Could you post the fiber port output ?
These switches are rather picky about spf and dacs.

What happens when you run iperf with a direct connection ?
 
P

PigLover

Moderator
Jan 26, 2011
3,186
1,545
113
Have you checked stats on the host or the switch to see if either side is reporting errors? Lots of retransmissions? You should at least be able to narrow it down to a specific link or even direction on that link.
 
waith

waith

Member
Aug 3, 2015
44
4
8
44
Increased window size (this is also with jumbo frames enabled)


Send

iperf3.exe -c 192.168.1.3 -w 1M
Connecting to host 192.168.1.3, port 5201
[ 4] local 192.168.1.26 port 58938 connected to 192.168.1.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 482 MBytes 4.04 Gbits/sec
[ 4] 1.00-2.00 sec 437 MBytes 3.67 Gbits/sec
[ 4] 2.00-3.00 sec 424 MBytes 3.55 Gbits/sec
[ 4] 3.00-4.00 sec 434 MBytes 3.64 Gbits/sec
[ 4] 4.00-5.00 sec 434 MBytes 3.64 Gbits/sec
[ 4] 5.00-6.00 sec 421 MBytes 3.53 Gbits/sec
[ 4] 6.00-7.00 sec 421 MBytes 3.53 Gbits/sec
[ 4] 7.00-8.00 sec 434 MBytes 3.64 Gbits/sec
[ 4] 8.00-9.00 sec 432 MBytes 3.63 Gbits/sec
[ 4] 9.00-10.00 sec 434 MBytes 3.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.25 GBytes 3.65 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.25 GBytes 3.65 Gbits/sec receiver


Reverse Send

iperf3.exe -c 192.168.1.3 -w 1M -R
Connecting to host 192.168.1.3, port 5201
Reverse mode, remote host 192.168.1.3 is sending
[ 4] local 192.168.1.26 port 58951 connected to 192.168.1.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 681 MBytes 5.71 Gbits/sec
[ 4] 1.00-2.00 sec 676 MBytes 5.67 Gbits/sec
[ 4] 2.00-3.00 sec 695 MBytes 5.83 Gbits/sec
[ 4] 3.00-4.00 sec 693 MBytes 5.81 Gbits/sec
[ 4] 4.00-5.00 sec 683 MBytes 5.73 Gbits/sec
[ 4] 5.00-6.00 sec 687 MBytes 5.77 Gbits/sec
[ 4] 6.00-7.00 sec 695 MBytes 5.83 Gbits/sec
[ 4] 7.00-8.00 sec 681 MBytes 5.72 Gbits/sec
[ 4] 8.00-9.00 sec 691 MBytes 5.79 Gbits/sec
[ 4] 9.00-10.00 sec 683 MBytes 5.73 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 6.70 GBytes 5.76 Gbits/sec sender
[ 4] 0.00-10.00 sec 6.70 GBytes 5.76 Gbits/sec receiver
 
Last edited:
waith

waith

Member
Aug 3, 2015
44
4
8
44
It seems the 2016 server seems pissed off about a few things. Hyper-V being one of them.

Failed to allocate virtual function for NIC 7FC2D3D1-C075-41F3-9DAC-289F09CC112B--A17DF296-0015-426E-8D29-306B51B2078F (Friendly Name: Network Adapter), status = The device is not in a valid state to perform this request..

The File Transfer (SMB) performance may be affected as Network Direct functionality is not supported in ConnectX-2 firmware version.
 
waith

waith

Member
Aug 3, 2015
44
4
8
44
wildchild said:
Running a rather similar setup, and running fine mellanox > mellanox , and mellanox > intel x520-da2.

Could you post the fiber port output ?
These switches are rather picky about spf and dacs.

What happens when you run iperf with a direct connection ?
Click to expand...
Fairly new at non-copper ethernet. How would I post the fiber port output? Haven't tested a direct connection yet. Probably won't get a chance until this evening.
 
wildchild

wildchild

Active Member
Feb 4, 2014
389
57
28
Go to the console of your switch ( through controller), then telnet localhost, then issue :

show fiber-ports optics-info all
 
waith

waith

Member
Aug 3, 2015
44
4
8
44
(UBNT) #show fiber-ports optics-info all

Link Link Nominal
Length Length Bit
50um 62.5um Rate
Port Vendor Name [m] [m] Serial Number Part Number [Mbps] Rev Compliance
-------- ---------------- --- ---- ---------------- ---------------- ----- ---- ----------------
0/1 FINISAR CORP. 8 3 WTSSRH20979 FTLX8572D3BCL 10300 A 10GBase-SR
0/2 FINISAR CORP. 8 3 WTSSRH20996 FTLX8572D3BCL 10300 A 10GBase-SR
0/3 FINISAR CORP. 8 3 WTSSRH20923 FTLX8572D3BCL 10300 A 10GBase-SR
0/4 FINISAR CORP. 8 3 WTSSRH20865 FTLX8572D3BCL 10300 A 10GBase-SR
0/5 FINISAR CORP. 8 3 WTSSRH20929 FTLX8572D3BCL 10300 A 10GBase-SR
 
wildchild

wildchild

Active Member
Feb 4, 2014
389
57
28
Well seems you have finisar gbics, should be fine.
What do the port stats tell you, drops or port errors ?
 
P

PigLover

Moderator
Jan 26, 2011
3,186
1,545
113
Try running it w/out Hyper-V involved (from the Windows host OS, without having the NIC configured on any Hyper-V v-switch). I think you might be limited by transfers across the v-switch.
 
wildchild

wildchild

Active Member
Feb 4, 2014
389
57
28
I must confess , i'm using these cards on esx and omnios, so i could very well be hyper-v specific.
If direct connection doesnt improve things , maybe run a live linux (knoppix or something similar) on both machines and iperf again.
That should give conclusive evidence if your problem is hw related or software ( windows or hyper-v) related.
On other thing, it's rather important to make sure you're running the last firmware in conjuction with the latest drivers
 
waith

waith

Member
Aug 3, 2015
44
4
8
44
Back at work until 4PM Central, I'll grab port stats this evening. Thanks for the help so far guys.
 
  • Like
Reactions: wildchild
waith

waith

Member
Aug 3, 2015
44
4
8
44
Disabling virtual machine queues had some interesting results. Still only getting 1/2 speed on the receive.


Capture1.PNG Capture2.PNG
 
You must log in or register to reply here.
Top Bottom