I recently got two MNPA19-XTR Mellanox-ConnectX2 cards and installed them into my servers. I followed another thread on here and they are both running fw=2.10.0720, the vstat is here:
My question is how to properly test speed, because it doesn't seem to be transferring at near 10gb/s
I was unsure of the best way to test so I followed this, using ntttpcd.exe and got the following:
The Throughput(MB/s) in the ####Totals: #### section shows 1129.622, which is close to the post which was using ntttcp to test, but the highest recorded speed in the upper section is 23408 KB/s, which is much lower than I expected; unless I am misunderstanding the test..
I also tested via iperf and here are the results:
Is there another, better way to test the speed between the two cards? I have seen a few things mentioning ethernet vs IB, wondering if I need to change the card type? I couldn't find a lot of information on this.
INFO:
2 SuperMicro servers, running Windows Server 2016
Both servers are using the MNPA19-XTR Mellanox-ConnectX2 cards
These two servers are connected via a Ubiquiti ES-16-XG 10gb/s switch.
Thank you in advance for your input!
Code:
hca_idx=0
uplink={BUS=PCI_E Gen2, SPEED=5.0 Gbps, WIDTH=x8, CAPS=5.0*x8}
MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=42, ALL_MASKED=N}
vendor_id=0x02c9
vendor_part_id=26448
hw_ver=0xb0
fw_ver=2.10.0720
PSID=MT_0F60110010
node_guid=0002:c903:0056:a256
num_phys_ports=1
port=1
port_guid=0202:c9ff:fe56:a256
port_state=PORT_ACTIVE (4)
link_speed=NA
link_width=NA
rate=10.00 Gbps
port_phys_state=LINK_UP (5)
active_speed=10.00 Gbps
sm_lid=0x0000
port_lid=0x0000
port_lmc=0x0
transport=RoCE v1.25
max_mtu=2048 (4)
active_mtu=1024 (3)
GID[0]=0000:0000:0000:0000:0000:ffff:c0a8:011e
GID[1]=fe80:0000:0000:0000:489c:9283:78cb:162e
I was unsure of the best way to test so I followed this, using ntttpcd.exe and got the following:
Code:
C:\Program Files>NTttcp.exe -s -m 28,*,192.168.1.30 -l 512k -a 2 -t 5
Copyright Version 5.33
Network activity progressing...
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 4.909 6883.683 524288.000
1 5.004 52693.845 524288.000
2 4.988 38287.089 524288.000
3 5.004 52796.163 524288.000
4 5.003 16783.530 524288.000
5 5.004 38266.986 524288.000
6 5.004 38164.668 524288.000
7 5.004 52796.163 524288.000
8 4.988 28227.747 524288.000
9 5.004 57093.525 524288.000
10 5.004 57400.480 524288.000
11 5.004 38266.986 524288.000
12 5.004 25272.582 524288.000
13 4.956 9814.366 524288.000
14 5.004 52796.163 524288.000
15 5.004 38266.986 524288.000
16 4.988 11085.806 524288.000
17 5.004 42768.985 524288.000
18 4.941 8497.065 524288.000
19 5.020 6731.474 524288.000
20 5.004 42871.303 524288.000
21 5.004 234308.553 524288.000
22 5.004 51261.391 524288.000
23 4.988 21658.380 524288.000
24 5.004 42768.985 524288.000
25 5.019 38152.620 524288.000
26 5.004 42768.985 524288.000
27 4.988 10367.281 524288.000
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
5651.500000 5.003 1456.532 1129.622
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
2259.244 1.366 11303.000
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
56292.824 0.765 73672.796 0.584
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
4068588 215310 4 0 6.741
I also tested via iperf and here are the results:
Code:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.30, port 49810
[ 5] local 192.168.1.10 port 5201 connected to 192.168.1.30 port 49817
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 390 MBytes 3.27 Gbits/sec
[ 5] 1.00-2.00 sec 415 MBytes 3.48 Gbits/sec
[ 5] 2.00-3.00 sec 406 MBytes 3.41 Gbits/sec
[ 5] 3.00-4.00 sec 447 MBytes 3.75 Gbits/sec
[ 5] 4.00-5.00 sec 418 MBytes 3.51 Gbits/sec
[ 5] 5.00-6.00 sec 396 MBytes 3.32 Gbits/sec
[ 5] 6.00-7.00 sec 390 MBytes 3.27 Gbits/sec
[ 5] 7.00-8.00 sec 390 MBytes 3.27 Gbits/sec
[ 5] 8.00-9.00 sec 387 MBytes 3.25 Gbits/sec
[ 5] 9.00-10.00 sec 396 MBytes 3.32 Gbits/sec
[ 5] 10.00-10.01 sec 3.02 MBytes 3.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.01 sec 3.94 GBytes 3.39 Gbits/sec receiver
INFO:
2 SuperMicro servers, running Windows Server 2016
Both servers are using the MNPA19-XTR Mellanox-ConnectX2 cards
These two servers are connected via a Ubiquiti ES-16-XG 10gb/s switch.
Thank you in advance for your input!
Last edited: