Mellanox 40gbe windows configuration???

[aron]

I am trying to connect a client windows 10 computer to a windows server 2016, with mellanox MCX314A-BCCT cards but getting bad speeds. Single stream Iperf around 5 gbit/sec, and up to around 12 gbit/sec maximum using 8 threads.

I am trying to find some step by step instruction on how to optimize, but cant find anything. Anyone who can point on where to find this or what can be wrong?

Things I tried:
- Latest firmware
- Cards running in proper PCIe slot, from vstat: uplink={BUS=PCI_E Gen3, SPEED=8.0 Gbps, WIDTH=x8, CAPS=8.0*x8}
- run auto tuning for the driver
- tried to tweak all sorts of advanced parameters in the driver.
- different cables

What else should I try?

What speed to expect on windows computers?

 

[fohdeesha]

what's the CPU? have you monitored cpu usage during the iperf tests? My guess is it's maxed out, it's not easy pushing 10gbps+

 

[aron]

what's the CPU? have you monitored cpu usage during the iperf tests? My guess is it's maxed out, it's not easy pushing 10gbps+

the client is running a 1950X and the server an i7-4790k

saw processor usage on client was like 5% when running iperf, but that is of course the full CPU. Will see what % the thread that seems to the the iperf job is having.

 

[fohdeesha]

those procs should definitely do more than 5gbit single core/12gbit multistream, are the cards directly connected? I can get 10gbps maxed out single stream iperf with default 1500mtu on my 4790k

 

[aron]

checked CPU usage. Single thread, most used CPU core like 30%. When doing 16 threads, the CPU start to see some work spread over a couple of cores.

When i do netstat -t i see

TCP 192.168.8.4:5201 DESKTOP-K8B34AS:50729 TIME_WAIT InHost
TCP 192.168.8.4:5201 DESKTOP-K8B34AS:50734 TIME_WAIT InHost
TCP 192.168.8.4:51270 DESKTOP-K8B34AS:microsoft-ds ESTABLISHED InHost

could this be a hint that some offloading doenst work properly?

 

[aron]

when i do Get-SmbClientNetworkInterface i get below.

Server is RDMA capable, but not client. Is that correct?

on client I get:

Interface Index RSS Capable RDMA Capable Speed IpAddresses Friendly Name
--------------- ----------- ------------ ----- ----------- -------------
9 True False 10 Gbps {fe80::5177:69a3:3990:e14b, 192.168.2.5} Ethernet 3
2 True False 40 Gbps {fe80::cd1f:1105:17d9:9d93, 192.168.8.5} Ethernet 5
11 False False 0 bps {fe80::646b:3a79:b691:a115} Ethernet 6
16 True False 40 Gbps {fe80::6811:d336:1c54:3b7, 192.168.11.5} Ethernet 4
12 True False 1 Gbps {fe80::41d5:8fc:9693:51d1, 192.168.0.134} Ethernet
6 False False 3 Mbps {fe80::b990:38ea:b1c8:5306} Bluetooth Network Connection


on server I get

Interface Index RSS Capable RDMA Capable Speed IpAddresses Friendly Name
--------------- ----------- ------------ ----- ----------- -------------
20 True True 40 Gbps {fe80::5c8:2abe:2592:d8a4, 192.168.8.4} Ethernet 5
25 True True 10 Gbps {fe80::10be:7fc3:691c:110b, 169.254.17.11} Ethernet 4
23 False False 100 Kbps {fe80::5efe:192.168.0.177} isatap.localdomain
24 False False 100 Kbps {fe80::5efe:192.168.0.4} isatap.{C828471E-49B5-4401-B93E-9030EC64D67E}
17 False False 100 Kbps {fe80::5efe:192.168.8.4} isatap.{CD975744-8ACA-4046-BBFB-E108BCFC32C9}
18 False False 100 Kbps {fe80::5efe:169.254.17.11} isatap.{E2492550-A48D-494B-995C-053490C740DF}
5 False False 100 Kbps {fe80::ffff:ffff:fffe} Teredo Tunneling Pseudo-Interface
19 True False 1 Gbps {fe80::a5ea:c59a:eba8:c627, 192.168.0.4} Ethernet
7 True False 1 Gbps {fe80::ed4a:4ea3:fded:ca4c, 192.168.0.177} vEthernet (Intel(R) Ethernet Connection (2) I218-LM - Virtual Switch)

 

[aron]

when I do ibv_stat i see below

is Real rate: 32.00 (QDR) correct?

CA 'ibv_device1'
CA type:
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 0x0
Node GUID: 0xec0d9a0300ea4390
System image GUID: 0xec0d9a0300ea4390
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Real rate: 32.00 (QDR)
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x80500000
Port GUID: 0xee0d9afffeea4390
Link layer: Ethernet
Transport: RoCE v2.0
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Real rate: 32.00 (QDR)
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x80500000
Port GUID: 0xee0d9afffeea4391
Link layer: Ethernet
Transport: RoCE v2.0

 

[aron]

when doing Get-SmbMultiChannelConnection on client, say client is not RDMA capable


PS C:\Windows\system32> Get-SmbMultiChannelConnection
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client
RDMA
Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ ---------
192.168.8.4 True 192.168.8.5 192.168.8.4 2 20 True False

 

[fossxplorer]

aron, did you go through Mellanox tuning guide? Most importantly, did you go through IRQ affinity setup?
And the RDMA/RoCE capability is not usd by iperf2/3.

 

[aron]

aron, did you go through Mellanox tuning guide? Most importantly, did you go through IRQ affinity setup?
And the RDMA/RoCE capability is not usd by iperf2/3.

hummm.... not sure... but isn't that part of what happens in the performance tuning in the adapter properties page? Or how do i do this on windows?

 

[aron]

now tried nd_send_bw and looks like some very good Gb/s numbers.... but what is actually being measured here?



PS C:\Windows\system32> nd_send_bw -a -C 192.168.8.4
#qp #bytes #iterations MR [Mmps] Gb/s CPU Util.
0 1 100000 2.483 0.02 100.00
0 2 100000 2.633 0.04 100.00
0 4 100000 2.621 0.08 100.00
0 8 100000 2.816 0.18 100.00
0 16 100000 2.805 0.36 100.00
0 32 100000 2.875 0.74 100.00
0 64 100000 2.621 1.34 100.00
0 128 100000 2.564 2.63 100.00
0 256 100000 2.710 5.55 100.00
0 512 100000 2.534 10.38 79.07
0 1024 100000 2.193 17.96 100.00
0 2048 100000 1.707 27.96 100.00
0 4096 100000 1.174 38.46 91.66
0 8192 100000 0.587 38.45 100.00
0 16384 100000 0.293 38.46 100.00
0 32768 100000 0.147 38.46 100.00
0 65536 100000 0.073 38.46 100.00
0 131072 100000 0.037 38.46 100.00
0 262144 100000 0.018 38.46 100.00
0 524288 100000 0.009 38.46 100.00
0 1048576 100000 0.005 38.46 100.00
0 2097152 100000 0.002 38.46 100.00

 

[fossxplorer]

Sorry, i've only done this is Linux, so nonidea how this is done in Windows.

 

[aron]

in the server i have an Asus Z97-WS motherboard. It uses a PLX chip to get more PCIe slot/lanes. Could that affect performance?

 

[fossxplorer]

ib_send_bw?
That is using RDMA with kernel bypass, zero copy etc etc

 

[aron]

now I tried NTttcp.exe -s -m 64,*,192.168.8.4 -t 60 -ns

and got below output, which should indicate 1640 MB/sec.


Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
98429.500000 60.001 8731.576 1640.464
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
26247.429 9.202 1574872.000
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
55201.147 0.986 810669.956 0.067
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
11820411 3266678 117 0 14.575