Any considerations on a multi-gigabit (1, 2.5, 5, 10) switch?

madbrain

Active Member
Jan 5, 2019
208
38
28
Did more troubleshooting today. First iperf below is USB NIC through existing 1GbE network. 2nd one is USB NIC directly connected to 10GbE adapter in the server. If I set the USB NIC to a 1GbE link, I get the same huge number of retries. So this leads me to believe it is the Ipolex SFP+ module. Which sucks as I bought it from outside the country. :(

Code:
PS E:\Downloads\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.1.50 -P 2 -R
Connecting to host 192.168.1.50, port 5201
Reverse mode, remote host 192.168.1.50 is sending
[  4] local 192.168.1.10 port 62766 connected to 192.168.1.50 port 5201
[  6] local 192.168.1.10 port 62767 connected to 192.168.1.50 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   565 MBytes   474 Mbits/sec    0             sender
[  4]   0.00-10.00  sec   564 MBytes   473 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   569 MBytes   478 Mbits/sec    0             sender
[  6]   0.00-10.00  sec   568 MBytes   477 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  1.11 GBytes   950 Mbits/sec                  receiver

iperf Done.
PS E:\Downloads\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.1.50 -P 2 -R
Connecting to host 192.168.1.50, port 5201
Reverse mode, remote host 192.168.1.50 is sending
[  4] local 192.168.1.10 port 62821 connected to 192.168.1.50 port 5201
[  6] local 192.168.1.10 port 62822 connected to 192.168.1.50 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   631 MBytes   529 Mbits/sec  20973             sender
[  4]   0.00-10.00  sec   631 MBytes   529 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   787 MBytes   660 Mbits/sec  22268             sender
[  6]   0.00-10.00  sec   787 MBytes   660 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec  43241             sender
[SUM]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver

iperf Done.
If you are using the RTL8156, I think it has problems. Drivers or firmware, I don't know.

Is it the Cable Creation model you have ? Even if it worked, it is really not well made.
The USB cord is too short. When using it with a desktop, it hangs from the USB 3 ports and loses the USB connection under the weight of the heavy CAT6A STP cable I used ... It needs a short USB extension cord. Sigh. Or maybe if I find an old PCIe USB 3.0 controller card in a drawer, put it at the bottom slot, that'll solve the problem ...

Anyway, the RTL8156 has problems even with my Trendnet 7080-ES NBASE-T switch. See http://blog.madbrain.com/2020/06/realtek-rtl-8156-usb-25-gbps-ethernet.html . Seems like the chip you are using also has incompatibilities with your SFP+.
 

lowfat

Active Member
Nov 25, 2016
130
90
28
38
@lowfat - any updates? can you describe your environment in more detail ? Windows 10? Win 201x server? Linux?
Win 10 2004 on my desktop. Ubuntu Server 19.10 on server. Both it and the networking gear are in the same room.





If you are using the RTL8156, I think it has problems. Drivers or firmware, I don't know.

Is it the Cable Creation model you have ? Even if it worked, it is really not well made.
The USB cord is too short. When using it with a desktop, it hangs from the USB 3 ports and loses the USB connection under the weight of the heavy CAT6A STP cable I used ... It needs a short USB extension cord. Sigh. Or maybe if I find an old PCIe USB 3.0 controller card in a drawer, put it at the bottom slot, that'll solve the problem ...

Anyway, the RTL8156 has problems even with my Trendnet 7080-ES NBASE-T switch. See http://blog.madbrain.com/2020/06/realtek-rtl-8156-usb-25-gbps-ethernet.html . Seems like the chip you are using also has incompatibilities with your SFP+.
It is a Cable Matters USB-C NIC. Cable is about 1ft. It was long enough. But I at first I noticed it would stop working after waking the computer. It needed to be unplugged and reconnected. But that issue seemed to stop.

Even w/ the onboard Intel NIC and going through the Ipolex transceiver it was getting thousands of retries in iperf receiver mode. Switch was showing RX MAC errors for SPF module too.

I resolved my issue by not using the Ipolex receiver or the 2.5GbE USB adapter :eek:. I installed a smaller heatsink for my GPU and it freed up an x8 slot. So I threw a ConnectX-2 in it, connected w/ a DAC cable to the Mitrotik switch. Now iperf does 10GbE, without any retries.

Code:
PS D:\Apps\iperf3> .\iperf3.exe -c 192.168.1.50 -R
Connecting to host 192.168.1.50, port 5201
Reverse mode, remote host 192.168.1.50 is sending
[  4] local 192.168.1.10 port 57739 connected to 192.168.1.50 port 5201
...
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver
Besides the increased noise on the desktop from the smaller GPU heatsink, the only issue now is that
WOL doesn't work and it doesn't look like its something easily enabled.
 

madbrain

Active Member
Jan 5, 2019
208
38
28
Win 10 2004 on my desktop. Ubuntu Server 19.10 on server. Both it and the networking gear are in the same room.

It is a Cable Matters USB-C NIC. Cable is about 1ft. It was long enough. But I at first I noticed it would stop working after waking the computer. It needed to be unplugged and reconnected. But that issue seemed to stop.

Even w/ the onboard Intel NIC and going through the Ipolex transceiver it was getting thousands of retries in iperf receiver mode. Switch was showing RX MAC errors for SPF module too.

I resolved my issue by not using the Ipolex receiver or the 2.5GbE USB adapter :eek:. I installed a smaller heatsink for my GPU and it freed up an x8 slot. So I threw a ConnectX-2 in it, connected w/ a DAC cable to the Mitrotik switch. Now iperf does 10GbE, without any retries.

Code:
PS D:\Apps\iperf3> .\iperf3.exe -c 192.168.1.50 -R
Connecting to host 192.168.1.50, port 5201
Reverse mode, remote host 192.168.1.50 is sending
[  4] local 192.168.1.10 port 57739 connected to 192.168.1.50 port 5201
...
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver
Besides the increased noise on the desktop from the smaller GPU heatsink, the only issue now is that
WOL doesn't work and it doesn't look like its something easily enabled.
Thanks for the data. Can't help you with any of the transceiver issues I'm afraid. I only use NBASE-T equipment. I guess that's one less thing to debug.

Don't recall if they were any retries with the 2.5 Gbps NIC - I will check again. And I will also see if Ic an get WOL working with it or not.

Assume you are running jumbo frames in the above test to get 9.89 Gbits/s with a single TCP stream. I can never get quite that high with single stream on my NBASE-T gear. However, I do usually reach that with multiple TCP streams in at least one direction - rarely both. Guess there is still some tuning to do in the Aquantia drivers and/or firmware.

BTW, -N flag (disable Nagle) is helpful in iperf3 tests, especially as a sender (without -R).
 
Last edited:
  • Like
Reactions: lowfat

hmw

Active Member
Apr 29, 2019
337
110
43
Win 10 2004 on my desktop. Ubuntu Server 19.10 on server. Both it and the networking gear are in the same room.
I have Windows 10 2004 on the CableMatters 2.5GbE, MacOS 10.15 on Sonnet 10GbE & Asustor 2.5GbE. And there's an ESXi server connected to the switch via a Mellanox and SFP 10GbE at the moment

The Windows and Mac machines are connected via Cat5E to the SFP ports of the switch - and the IPOLEX 10Gbase-T transceivers.

When I test iperf3 between the Windows and Mac machines with the 2.5GbE USB adapters - I get 2.39 gbit/s and no retries

When I test either windows or mac machine with a guest running on the ESXi server - and the win/mac machines have 2.5GbE USB adapters - I get 2.4 gbit/s in one direction and 1.5 gbit/sec in the other with some retries

When I test the mac machine with the Sonnet 10GbE and a guest on the ESXi server - I get 9.49 gbit/s and no retries

I would say this isnt a hardware fault with the IPOLEX transceiver but maybe it's reporting 10G on one side and 2.5G on the other and causes modern OSes to 'auto-tune' to the wrong settings

Will do more testing and post results ...
 
  • Like
Reactions: lowfat

lowfat

Active Member
Nov 25, 2016
130
90
28
38
I would say this isnt a hardware fault with the IPOLEX transceiver but maybe it's reporting 10G on one side and 2.5G on the other and causes modern OSes to 'auto-tune' to the wrong settings

Will do more testing and post results ...
I had thought the same thing and it sort of makes sense. On the Mitrotik switch, it reports it as a 10G link, even though Windows shows the 2.5GbE adapter w/ a 2.5G link. Sending data from the 2.5GB NIC works fine w/o errors. But when sending from the server, it shows a 10GbE link, so its sending too much data? I don't know enough theory behind networks tho so I'm straight guessing. AFAICT there is no way to force a 2.5GB speed in SwOS either to test this further.
 
Last edited:

pod

New Member
Mar 31, 2020
15
7
3
I hate thought the same thing and it sort of makes sense. On the Mitrotik switch, it reports it as a 10G link, even though Windows shows the 2.5GbE adapter w/ a 2.5G link. Sending data from the 2.5GB NIC works fine w/o errors. But when sending from the server, it shows a 10GbE link, so its sending too much data? I don't know enough theory behind networks tho so I'm straight guessing. AFAICT there is no way to force a 2.5GB speed in SwOS either to test this further.
Not possible to turn off auto-negotiation?
 

pod

New Member
Mar 31, 2020
15
7
3
Routeros does allow link speed advertisement change, so maybe see what happens? Seems a fairly severe Mikrotik Swos overlook.
 

madbrain

Active Member
Jan 5, 2019
208
38
28
Besides the increased noise on the desktop from the smaller GPU heatsink, the only issue now is that
WOL doesn't work and it doesn't look like its something easily enabled.
If we are still talking about the Realtek 2.5gbps, I tried with my Cable Matters USB-A model. WOL is easily enabled in the Windows driver settings. But it never wakes up, unfortunately. Definitely a bug with the driver NIC.

As for performance, here is how it's behaving today, talking to a 10Gbps Linux server ...


C:\Users\Julien Pierre\Desktop\iperf3>iperf3 -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.236 port 17927 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 1.00-2.00 sec 271 MBytes 2.27 Gbits/sec
[ 4] 2.00-3.00 sec 269 MBytes 2.26 Gbits/sec
[ 4] 3.00-4.00 sec 282 MBytes 2.36 Gbits/sec
[ 4] 4.00-5.00 sec 264 MBytes 2.22 Gbits/sec
[ 4] 5.00-6.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 6.00-7.00 sec 267 MBytes 2.24 Gbits/sec
[ 4] 7.00-8.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 8.00-9.00 sec 282 MBytes 2.36 Gbits/sec
[ 4] 9.00-10.00 sec 282 MBytes 2.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec sender
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver

iperf Done.

C:\Users\Julien Pierre\Desktop\iperf3>iperf3 -N -c server10g -R
Connecting to host server10g, port 5201
Reverse mode, remote host server10g is sending
[ 4] local 192.168.1.236 port 17929 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 5.31 MBytes 44.5 Mbits/sec
[ 4] 1.00-2.01 sec 6.53 MBytes 54.1 Mbits/sec
[ 4] 2.01-3.01 sec 13.6 MBytes 115 Mbits/sec
[ 4] 3.01-4.00 sec 10.7 MBytes 90.6 Mbits/sec
[ 4] 4.00-5.00 sec 6.29 MBytes 52.8 Mbits/sec
[ 4] 5.00-6.00 sec 9.19 MBytes 77.1 Mbits/sec
[ 4] 6.00-7.00 sec 6.45 MBytes 54.1 Mbits/sec
[ 4] 7.00-8.00 sec 5.84 MBytes 48.9 Mbits/sec
[ 4] 8.00-9.01 sec 5.93 MBytes 49.2 Mbits/sec
[ 4] 9.01-10.00 sec 8.35 MBytes 71.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 78.5 MBytes 65.9 Mbits/sec 2076 sender
[ 4] 0.00-10.00 sec 78.4 MBytes 65.7 Mbits/sec receiver

iperf Done.

Really atrocious performance.

Internet speedtest is hilarious, too.

realtek-2.5g.png

From the same Windows 10 PC, disabling the Realtek NIC, and re-enabling the PCIe Aquantia NIC :
aquantia10g.png
Strangely, the Speedtest native app chose servers in different cities, even though it was going through the exact switch and router at home. I'm actually in San Jose and not SF. Not that it matters.
 

lowfat

Active Member
Nov 25, 2016
130
90
28
38
Assume you are running jumbo frames in the above test to get 9.89 Gbits/s with a single TCP stream. I can never get quite that high with single stream on my NBASE-T gear. However, I do usually reach that with multiple TCP streams in at least one direction - rarely both. Guess there is still some tuning to do in the Aquantia drivers and/or firmware.
Missed this. Yes I am using jumbo frames. Had no issues so I continued to use it.
Routeros does allow link speed advertisement change, so maybe see what happens? Seems a fairly severe Mikrotik Swos overlook.
Tried RouterOS and it does allow you to select 2.5G as advertised. But AFAICT it does nothing. Even if I only advertise 2.5G, RouterOS shows the module @ 10G. However it does allow you to disable auto negotiate and force 10G. I lost connection when I tried it.

If we are still talking about the Realtek 2.5gbps, I tried with my Cable Matters USB-A model. WOL is easily enabled in the Windows driver settings. But it never wakes up, unfortunately. Definitely a bug with the driver NIC.

As for performance, here is how it's behaving today, talking to a 10Gbps Linux server ...


C:\Users\Julien Pierre\Desktop\iperf3>iperf3 -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.236 port 17927 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 1.00-2.00 sec 271 MBytes 2.27 Gbits/sec
[ 4] 2.00-3.00 sec 269 MBytes 2.26 Gbits/sec
[ 4] 3.00-4.00 sec 282 MBytes 2.36 Gbits/sec
[ 4] 4.00-5.00 sec 264 MBytes 2.22 Gbits/sec
[ 4] 5.00-6.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 6.00-7.00 sec 267 MBytes 2.24 Gbits/sec
[ 4] 7.00-8.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 8.00-9.00 sec 282 MBytes 2.36 Gbits/sec
[ 4] 9.00-10.00 sec 282 MBytes 2.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec sender
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver

iperf Done.

C:\Users\Julien Pierre\Desktop\iperf3>iperf3 -N -c server10g -R
Connecting to host server10g, port 5201
Reverse mode, remote host server10g is sending
[ 4] local 192.168.1.236 port 17929 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 5.31 MBytes 44.5 Mbits/sec
[ 4] 1.00-2.01 sec 6.53 MBytes 54.1 Mbits/sec
[ 4] 2.01-3.01 sec 13.6 MBytes 115 Mbits/sec
[ 4] 3.01-4.00 sec 10.7 MBytes 90.6 Mbits/sec
[ 4] 4.00-5.00 sec 6.29 MBytes 52.8 Mbits/sec
[ 4] 5.00-6.00 sec 9.19 MBytes 77.1 Mbits/sec
[ 4] 6.00-7.00 sec 6.45 MBytes 54.1 Mbits/sec
[ 4] 7.00-8.00 sec 5.84 MBytes 48.9 Mbits/sec
[ 4] 8.00-9.01 sec 5.93 MBytes 49.2 Mbits/sec
[ 4] 9.01-10.00 sec 8.35 MBytes 71.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 78.5 MBytes 65.9 Mbits/sec 2076 sender
[ 4] 0.00-10.00 sec 78.4 MBytes 65.7 Mbits/sec receiver

iperf Done.

Really atrocious performance.

Internet speedtest is hilarious, too.


From the same Windows 10 PC, disabling the Realtek NIC, and re-enabling the PCIe Aquantia NIC :
Strangely, the Speedtest native app chose servers in different cities, even though it was going through the exact switch and router at home. I'm actually in San Jose and not SF. Not that it matters.
WOL works fine on the Cable Matters 2.5G USB-C adapter I have. Were you ever able to get WOL working on other adapters? I was referring to the Mellanox ConnectX-2.

So your iperf looks pretty similar to mine. Good performance as sender, performance all over the place for receiver.
 

madbrain

Active Member
Jan 5, 2019
208
38
28
Missed this. Yes I am using jumbo frames. Had no issues so I continued to use it.
I had issues with the combination of IPv6 and jumbo frames. The problem is that my ISP's cable modem won't support jumbo frames. Having jumbo frames only on the LAN created problems. The Path-MTU protocol failed. Some tests, especially Test your IPv6., failed due to mismatched MTU between LAN and WAN. Jumbo frames are also thorny with Wifi, if some machines use Ethernet to connect to a Wifi bridge, as jumbo frames are dropped by Wifi APs generally. I recently gave up on trying to make peace between jumbo frames and IPv6. I disabled jumbo frames on my LAN this week. In the best case, they give throughput from 9.49 to 9.89 Gbps in iperf3. I think those last 400 Mbps on the LAN aren't really worth the trouble. For WAN they make no difference since cable modem doesn't support jumbo frames. And my Ubiquiti USG router doesn't support them well at all either.

Tried RouterOS and it does allow you to select 2.5G as advertised. But AFAICT it does nothing. Even if I only advertise 2.5G, RouterOS shows the module @ 10G. However it does allow you to disable auto negotiate and force 10G. I lost connection when I tried it.
I see.

WOL works fine on the Cable Matters 2.5G USB-C adapter I have. Were you ever able to get WOL working on other adapters? I was referring to the Mellanox ConnectX-2.
I have WOL working on all 4 of my Aquantia NICs, under Windows 10 and Ubuntu. Have had it working on Realtek and Intel 1 Gbps adapters built to the motherboards before. Maybe there is some additional USB setting I need in the BIOS. But I can already wake up my machine by keyboard or mouse activity which is over USB also, so I think that should be in place. I currently have that setting disabled, though.

So your iperf looks pretty similar to mine. Good performance as sender, performance all over the place for receiver.
I have seen tons more varied performance.

I wrote a more extensive review of this adapter pinging a bunch of hosts on my LAN. It finally became public on Amazon last night.
See RTL8156 chipset has problems. Zero support from Cable Creation or Realtek

Results are all over the place on both sides for some hosts. Server10g was running Ubuntu 18.04 in those tests, and getting full throughput on one side, but not the other. Other host with the same Aquantia NIC, but running Windows 10, were getting inconsistent performance on both sides.

I'm very suspicious about the RTL8156. When first installing the NIC, Windows automatically downloaded a driver for it dated 2015. 2.5 GBASE-T did not become an IEEE standard until about a year later. That says to me the chip might have been developed before the standard was complete. Maybe it's not fully compliant with the NBASE-T standard. For my tests, I downloaded the latest Windows 10 drivers from Realtek at Realtek USB FE / GBE / 2.5G / Gaming Ethernet Family Controller Software - REALTEK . I was originally inclined to blame my Trendnet 7080-ES switch, given that the RTL8156 worked fine with one switch (Netgear GS110MX) but not the Trendnet. But since you appear to be running into issues with another NIC of the same Realtek chipset, but with different switches (in fact, SFP+ NBASE-T transceivers), that shifts my suspicion back on Realtek.

After that review posted, Amazon invited me back into the VOICE program, which I had left years ago after failing to review some of the free gear they sent me for a year. This time I'll be careful to only free request stuff I really care about. Just be clear, I paid for this Cablecreation NIC, and it's going back.

Trendnet said they would get back to me by the 24th about the issue. I'm really curious what they will say. Maybe the fix actually has to come from Realtek, though. Unfortunately, as I mentioned in my Amazon review, neither CableCreation nor Realtek responded to tech support email inquiries submitted the day after I received the NIC and found the problem.

I held off purchasing an Odroid H2+ recently because it has two Realtek 2.5 Gbps NICs, albeit PCIe RTL8125 models. It would have to come from Korea with expensive shipping, and I couldn't easily return it if there is a compatibility issue with the NIC also. There is a US distributor for it, Ameridroid, but there is a restocking fee, and I don't want to be paying that to be a guinea pig. Also have to buy some SODIMMs to run it, as well as PSU and case. I have a spare SATA SSD to use with it (no SD card boot with this SBC). It would have come to at least $250 with all the needed parts.

Odroid offered me to get me a free SBC from them recently after I helped edit their technical blog. But I chose the Odroid-N2+, not H2+. They just shipped it thursday. Hopefully with a case and PSU too. I already have spare SD cards to use it, so should be no additional cost to me. No 2.5 Gbps on it, though. But can add it via USB 3.0 ports in the future if Realtek gets its act together.
 
Last edited:

lowfat

Active Member
Nov 25, 2016
130
90
28
38
I still believe the issue is w/ the IPolex transceiver. It seems to only advertise 10G. Even if I use an Intel 1Gbps NIC connected to the IPolex I have issues. In RouterOS, it still shows its 10G. And I still get tons of retries and poor throughput in iperf3.

It sounds exactly like the issues the first gen Mitrotik S+RJ10 modules had.
Link speed reporting is fixed - actual link speed in the interface menu is listed. Previously 100M/1G/2.5G/5G link speeds were not reported correctly;
I'd like to buy one to try out, but unfortunately it would be $100USD+ to get one to Canada currently. Which isn't worth it for me.
 

hmw

Active Member
Apr 29, 2019
337
110
43
I still believe the issue is w/ the IPolex transceiver. It seems to only advertise 10G. Even if I use an Intel 1Gbps NIC connected to the IPolex I have issues. In RouterOS, it still shows its 10G.
I have a Wiitek 10G on order and it should be here next week, will test with that.

My test setup is below

- Node A: Win 10 (2.5GbE)
- Node B: MacOS (10GbE / 2.5GbE)

The two nodes above are connected to the switch via Cat5E and go into the Ipolex transceivers which in turn go into the switch's SFP ports

The ESXi host is connected to switch via a DAC running into a ConnectX-4 LX @ 40 GbE

- Node C: Win 2019 Server (SR-IOV)
- Node D: Ubuntu 20.04 (SR-IOV)

I also plan to test with a MacBook and a 10GbE / SFP - so this takes the Ipolex out of the equation ...
test_setup.png
 
Last edited:

hmw

Active Member
Apr 29, 2019
337
110
43
@lowfat - here's some preliminary results.


Some changes to the above diagram: Node A and B are MacBook Pro's: one with a TB3 10GbE adapter and the other with aTB3 cable to a ConnectX-3 card. A small wrinkle is that MacOS drivers for standard ConnectX-3 cards don't exist. Hence I don't have Node A to Node B for MacOS

Where I do not have the Ipolex in the path, I've noted that down.

Receiver
SenderNode B
(MacOS & 10GbE)
Node B
(Linux & 10GbE)
no Ipolex
Node C
(Win2019 & 40GbE)
Node D
(Linux &40 GbE)
Node A
(MacOS & 10GbE)
Node B
(MacOS & 10GbE)
Node B
(Linux & 10GbE)
no Ipolex
9.44 Gbit/sec
No retries
9.40 Gbit/sec
2 retries
7.51 Gbit/sec
No retries
Node C
(Win2019 & 40GbE)
8.04 Gbits/sec
No retries
5.08 Gbits/sec
No retries
Node D
(Linux &40 GbE)
9.39 Gbit/sec
21105 retries
6.32 Gbits/sec
6607 retries
Node A
(MacOS & 10GbE)
9.40 Gbit/sec
No retries
9.24 Gbits/sec
No Retries
9.40 Gbits/sec
No Retries
 
  • Like
Reactions: lowfat

hmw

Active Member
Apr 29, 2019
337
110
43
I'm not using jumbo frames or parallel streams, in order to optimize the lowest common denominator first

Things to note is
  • Linux to Linux always seems to be good.
  • Sending from Mac to Linux is good
  • Sending TO the Mac from either Linux or Windows is SO much slower
  • Sending from the Mac to Windows - is surprisingly okay

The other experiment I did was send to & from two nodes - MacOS and Win10 Ent - connected with the USB 2.5 GbE (and Ipolex): I get 2.34 Gbit/sec in either direction and no retries.

Also - one has to be VERY careful with blindly typing in values for network parameters and expecting miraculous results. One has to understand what the parameters do and whether they would help in the particular scenario. e.g. increasing the MSS on MacOS from 512 to 1448, will mean Node C -> Node A speed goes from 5.08 to 5.93 Gbit/sec. That's a 16% increase. Nothing to laugh at, but clearly well below the 9.40 gbit/sec speed in the reverse direction
 

lowfat

Active Member
Nov 25, 2016
130
90
28
38
Results are very odd.

Node C Win 2019 + 40GbE -> Node B Linux 10GbE = no retries. But yet Node D Linux 40GbE -> Node B Linux 10GbE has 22k. But Win2019 has lower throughput. Exact same hardware, just different sending OS.

Can you set the 40GbE to 10GbE to see if all the retires stops?
 

hmw

Active Member
Apr 29, 2019
337
110
43
Results are very odd.
I couldn't agree more! It would seem the results are more to do with OS TCP/IP implementation stack than anything else. I made sure to use iPerf 3.1.3 in all tests

Can you set the 40GbE to 10GbE to see if all the retires stops?
Tried to do that but can't. The server had a ConnectX-4 LX card that's attached via a QSFP+ DAC to an ICX6610 switch. The switch absolutely refuses to enable the link when the card is set to anything other than 40 Gbit/s
 
  • Sad
Reactions: lowfat

madbrain

Active Member
Jan 5, 2019
208
38
28
BTW, for those looking at the 2.5 Gbps NICs . I'm not sure Realtek is the only problem here. I forced one of my Aquantia NICs to negotiate down to 2.5 Gbps and 5 Gbps. Got line rate when sending multiple streams. But very low rate with receive - and near zero with single stream receive, just like with the Realtek 8156. So, it looks like the Trendnet TEG-7080ES switch really doesn't work properly with either 2.5 or 5 Gbps speed, even though it is advertised as such. All is fine at 1 Gbps or 10 Gbps. Seems to me these sorts of issues are going to kill deployments of 2.5 Gbase-T in the home if not resolved by vendors.
 

hmw

Active Member
Apr 29, 2019
337
110
43
BTW, for those looking at the 2.5 Gbps NICs . I'm not sure Realtek is the only problem here. I forced one of my Aquantia NICs to negotiate down to 2.5 Gbps and 5 Gbps. Got line rate when sending multiple streams. But very low rate with receive - and near zero with single stream receive, just like with the Realtek 8156. So, it looks like the Trendnet TEG-7080ES switch really doesn't work properly with either 2.5 or 5 Gbps speed, even though it is advertised as such. All is fine at 1 Gbps or 10 Gbps. Seems to me these sorts of issues are going to kill deployments of 2.5 Gbase-T in the home if not resolved by vendors.
I'm not so sure Realtek is the problem at all. I just tested two machines

Machine A: Intel NUC 8i7 BEH, connected to a Pluggable USB 2.5
Machine B: MacBook Pro 2017 connected to an Asustor USB 2.5

And I tested with two switches: ICX 6610 and Netgear MS510TX

Code:
MBP:$ ./iperf3 -c machine_a
Connecting to host machine_a, port 5201
[  4] local machine_b port 63491 connected to machine_a port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec               
[  4]   1.00-2.00   sec   283 MBytes  2.37 Gbits/sec               
[  4]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   3.00-4.00   sec   283 MBytes  2.37 Gbits/sec               
[  4]   4.00-5.00   sec   283 MBytes  2.37 Gbits/sec               
[  4]   5.00-6.00   sec   283 MBytes  2.37 Gbits/sec               
[  4]   6.00-7.00   sec   283 MBytes  2.37 Gbits/sec               
[  4]   7.00-8.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   8.00-9.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   9.00-10.00  sec   283 MBytes  2.37 Gbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.76 GBytes  2.37 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.76 GBytes  2.37 Gbits/sec                  receiver
And in reverse

Code:
MBP:$ ./iperf3 -c machine_a -R
Connecting to host machine_a, port 5201
Reverse mode, remote host machine_a is sending
[  4] local machine_b port 63675 connected to machine_a port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   279 MBytes  2.34 Gbits/sec               
[  4]   1.00-2.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   3.00-4.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec               
[  4]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec               
[  4]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec               
[  4]   7.00-8.00   sec   282 MBytes  2.37 Gbits/sec               
[  4]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec               
[  4]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec                  receiver
This is with the ICX6610 (with Ipolex 10GbaseT transceivers). Even with the Netgear, the results are the same. I would suggest you get a proper switch and also make sure your cabling and termination is up to spec. I've also had the Asustor connected to a Windows server 24/7 for 2 months and it's been fine, never had drop outs or anything. The USB adapters do get considerably warm hence need to be cooled or provided with adequate ventilation.

btw - I am using very good quality Cat5E (and can get 10 GbE on it, with one run being 80 feet) and recently redid my patch panel termination to make sure the wires were twisted right upto the insertion points. I'm also redoing the wall jacks slowly - you'd be surprised when you look at the way some of the wiring is done. When you get to 2.5 GbE, you need just that little bit better quality.
 
Last edited:
  • Like
Reactions: lowfat