Mix 10G 2.5G, slow speed, high Retr

jena

New Member
May 30, 2020
22
5
3
Update:
I end up returned the Mikrotik CRS305 and bought NetGear MS510TX.

TLDR:
  • It looks like Mikrotik still have pause frame/flow control issue with 2.5/5G, with common RJ45-SFP+ modules (6COM, ipolex, etc..)
  • The only known RJ45-SFP+ module that works at 2.5G/5G is Supermicro AOM-AQS-107-B0C2-CX (Aquantia AQS-107-CX-1)
  • IMO, take caution with QNAP M308/408/2108 switches, buy at place that you can return. There seems to be reports on its firmware quirks and QA issue. In general, a SOHO product, not a solid build.
Questions
Q1: Seek diagnose suggestions. see my testing.

My guess is that when 10G NIC is sending packets to MikroTik switch, which is not aware of the receiving end is at 2.5G.
Thus it pumps data as fast as it can out of 10G's TX buffer and receiving 2.5G NIC RX buffer couldn't handle and drops them a lot. Thus Re-transmit is needed.

Q2.Will high Retr corrupts transmitted data (like from PC to SMB, SMB to PC)?


Setup
Router: EdgeRouter X
Switch 1 (10G): MikroTik CRS305-1G-4S+IN,

PC1 - Switch1: RJ45 - SFP+ adapter, CAT6 cable (all are quality cable and less than 25ft, I tried a few different cables, no change)
SFP+ RJ45 module that I have tried: 6COM, ipolex, GWFiber. They all claim to support 2.5G
Use three different brand of module, doesn't seems to have significant difference or solve my high Retr and slow speed

10G to Switch1: New 10G Tek DAC cable, New GWFiber 10G DAC cable. No change in terms of speed.

PC2: I borrowed a buddy's i5-7500 and B150M motherboard. Mellanox MCX-311A 10G NIC is plugged in slot 1 x16. Default driver in Linux Mint 20. (10G NIC was used in UNRAID, but I took it out and use PC2 to isolate problem.)

PC1: (ASUS B550 Intel I225V 2.5G, Rev.3, FW 1.57), TX/RX buffer = 1024, flow control enabled.
There are quite a bit report on Rev1 and 2 of I225, dropping packets. Mine is Rev3, which is "supposed" to be free of previous issue.
Driver for Windows 10 is the latest - 1.0.1.4.

Server2: Open Media Vault RTL8125 2.5G NIC, Linux 5.10.0-0.bpo.5-amd64, uses RTL8169 driver that was built in.
Use putty to SSH and test iperf.
Adapter settings: see detail at the end
I also tried to compile latest Realtek RTL8125 driver and the performance is even worse, at 110MB/s when receiving data from 10G NIC

All default, MTU 1500.

iperf3
  • host Server2 (RTL8125) to client PC1 (Intel I225V)
    • 283 MBytes/sec, no Retr displayed in output
  • host Server2 (RTL8125) to client PC2 (10G NIC)
    • 259 MBytes/sec, total Retr = ~60000
  • host PC1 (I225V) to client PC2 (10G NIC)
    • 160 MBytes/sec (interesting...), total Retr = ~30000
  • host PC1 (I225V) to client Server2 (RTL8125)
    • 221 MBytes/sec, total Retr = 28
  • host PC2 (10G NIC) to client PC1 (I225V)
    • 283 MBytes/sec, no Retr displayed in output
  • host PC2 (10G NIC) to client Server2 (RTL8125)
    • 229 MBytes/sec, Retr = 0
Code:
####### Server2 2.5G RTL8125
Settings for enp5s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
                                             10000baseT/Full
                                             2500baseT/Full
                                             5000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes


####### PC2 10G NIC
ethtool enp1s0
Settings for enp1s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
    Current message level: 0x00000014 (20)
                   link ifdown
    Link detected: yes
 
Last edited:

RTM

Well-Known Member
Jan 26, 2014
764
280
63
AFAIK Mikrotik has fixed a lot of that kind of issues in newer firmwares, have you made sure to update the firmware of the switch?

If you are using SwitchOS, you should try RouterOS as fixes are usually made available there sooner than SwitchOS.
 

jena

New Member
May 30, 2020
22
5
3
AFAIK Mikrotik has fixed a lot of that kind of issues in newer firmwares, have you made sure to update the firmware of the switch?

If you are using SwitchOS, you should try RouterOS as fixes are usually made available there sooner than SwitchOS.
I am running switchOS with latest firmware 2.13 (built at Mon Apr 26 2021).

I will try routerOS.

Update: I switch to routerOS, update to latest stable version. all default settings.
The results are the same.
Still 160MB/s when 10G is sending and 2.5G intel NIC is receiving.
 
Last edited:

unmesh

Member
Apr 17, 2017
87
26
18
62
@jena

I've tried different hardware and OSes but same basic topology which is client - 2.5G NIC (USB or PCIe) - SFP+ copper transceiver - switch - 10GDAC - server and could only get sub-2.5G performance and lots of retries like you. What did work was to borrow a QNAP QSW-M408-4C switch that had multirate RJ45 ports that supported 100/1000/2500/5000/10000 and use one of those instead of a SFP+ port with a copper transceiver

I'm not ready to buy a new 10G switch and am trying to get an Aquantia based SFP+ copper transceiver that someone reported has worked for them. Something about using Pause Frames to achieve flow control.

These are devilishly difficult to find.
 
  • Like
Reactions: jdnz

jena

New Member
May 30, 2020
22
5
3
@jena

I've tried different hardware and OSes but same basic topology which is client - 2.5G NIC (USB or PCIe) - SFP+ copper transceiver - switch - 10GDAC - server and could only get sub-2.5G performance and lots of retries like you. What did work was to borrow a QNAP QSW-M408-4C switch that had multirate RJ45 ports that supported 100/1000/2500/5000/10000 and use one of those instead of a SFP+ port with a copper transceiver

I'm not ready to buy a new 10G switch and am trying to get an Aquantia based SFP+ copper transceiver that someone reported has worked for them. Something about using Pause Frames to achieve flow control.

These are devilishly difficult to find.
Thank you for the information!

Is there an example model number of "an Aquantia based SFP+ copper transceiver" that you suggest?
Do you by chance have the source of the post about the Aquantia based transceiver?

When I bought Mikrotik CRS305, I also looked at QNAP M308 and M408 and thought Mikrotik might be more robust and better cooling.
M308/408 looks more like prosumer grade SOHO switch. Well, maybe QNAP is a better choice in mixed 2.5G situation.

I still have about a week left to return CRS305 switch and transceiver and go with QNAP.
Adding two transceiver already brings the total cost to about $220 and QNAP M408 is about $250-300 depending on the combo ports.
I am just not sure of QNAP M408 or M2108 have the same quality and reliability as Mikrotik.

Update: I will steer clear from QNAP switch products. M308/408/2108 doesn't seems to be well made and occasionally QA issues and quit working (require reboots) reported in Amazon's review. The management UI and features are amateur (in a way that doesn't seems to be well supported for years to come). Also given previous QNAP NAS severe security vulnerability issues, it does not inspire confidence in terms of QNAP firmware.

It seems that Netgear MS510TX might be a good choice. Then add a 9 port Mikrotik (all SFP+) in the future when more 10G ports are needed.

I would skip 2.5G if I could, but my ITX motherboard only have on board 2.5G and also old machine only PCIE x1 left for a 2.5G NIC.

Thanks!
 
Last edited:

DDD

New Member
Oct 8, 2013
6
1
3
Germany
Is there an example model number of "an Aquantia based SFP+ copper transceiver" that you suggest?
Do you by chance have the source of the post about the Aquantia based transceiver?
The only RJ45 - SFP+ transciever that worked for me (with a ubiquiti es 16-xg and intel i255-v & usb 5g adapter) was the Aquantia AQS-107-CX-1. I have a Supermicro OEM version ( AOM-AQS-107-B0C2-CX ) which unfortunately was very expensive, but I didn't want to install an additional switch.
Had to enable flow-control on the port.
 
Last edited:

unmesh

Member
Apr 17, 2017
87
26
18
62
...
It seems that Netgear MS510TX might be a good choice. Then add a 9 port Mikrotik (all SFP+) in the future when more 10G ports are needed.
...
It is on my short list but has only 2 10G ports and I already have two servers on 10G. MS510TXM would be better but is $500. Hence my quest for a working copper multirate SFP+ transceiver since there are plenty of used switches with SFP+ ports available for not much money.

Oh, and plenty of compalints about Mikrotik SwOS quality for their new switches on their forums.
 

coxhaus

Member
Jul 7, 2020
86
32
18
Cisco small business switches have some 2.5 gig model switches in the SG350X line and probably the new Cisco CBS switch line. I have not seen any cheap yet.
 

NateS

Active Member
Apr 19, 2021
124
63
28
Sacramento, CA, US
Trendnet also has an 8x 2.5g + 2x 10G SFP+ switch that's a bit cheaper than the Netgear offerings at 400. I don't have any experience with them myself, but I've been thinking of picking one up to add some 2.5G ports to my main 10G network.

 

jena

New Member
May 30, 2020
22
5
3
Trendnet also has an 8x 2.5g + 2x 10G SFP+ switch that's a bit cheaper than the Netgear offerings at 400. I don't have any experience with them myself, but I've been thinking of picking one up to add some 2.5G ports to my main 10G network.

Nice. 4 more 2.5G ports compare to NetGear MS510TX
 

jdnz

New Member
Apr 29, 2021
11
2
3
the engenius ecs2512 gives you 8 2.5gbe ports and 4 sfp+ ports ( so two more sfp+ ports than the trendnet).
 

NateS

Active Member
Apr 19, 2021
124
63
28
Sacramento, CA, US
That engenius will be nice if it comes down in price in the future. At 600 though, those two extra 10g SFP+ ports are 100 each, which is way too much IMO, given that I can get a brocade with 8 SFP+ ports for ~200.

I do think the 8x 2.5 + 2x 10 is a nice combination, since all the 2.5g ports would have full bandwidth to the 10G network simultaneously if both 10g ports are used as uplinks.
 

unmesh

Member
Apr 17, 2017
87
26
18
62
Mixed news to report. I got my hands on an AQS-107 based transceiver but my Juniper EX2300C switch does not recognize it whereas it did recognize the Marvell based transceivers I had tested previously :(

Looks like I will have to buy a new switch after all in which case I might as well buy one with native NBaseT support.
 

lowfat

Active Member
Nov 25, 2016
128
89
28
38
I've had the same issues. I ended up just using 10GbE or 1GbE. I have seen some 2.5G SFP (not +). Anyone know if these work?
 

unmesh

Member
Apr 17, 2017
87
26
18
62
I thought the SFP electrical side maxed out at 1G.

Since I had the SFP+ transceiver, I decided to try it in a Mellanox CX3 PCIe NIC with a Realtek based USB 2.5G NIC at the other end even though this is not something I would use in practice.

Linux on both systems; ethtool showed the Mellanox link at 10G as expected but the Realtek side would only autonegotiate 1G and the ethtool/driver (r8152 version 2.15) combination would not let me manually set it to 2.5G.

The OP's 2.5GbE NIC is PCIe and is showing a link speed of 2500.
 

lowfat

Active Member
Nov 25, 2016
128
89
28
38
I thought the SFP electrical side maxed out at 1G.
4Gbit I believe. TP Link sells a 2.5G one in China. I've seen other generic ones available.
 
  • Like
Reactions: marunjar

jena

New Member
May 30, 2020
22
5
3
Update: I end up returned the Mikrotik CRS305 and bought NetGear MS510TX.

I let most settings default on MS510TX, and set "Global Flow Control (IEEE 802.3x) Mode" Enabled.
10G can reach 1100MByte/s, 2.5G can reach 280MByte/s. And there is no issue sending and receiving between 10G and 2.5G. Very occasionally, I get Retr = less than 10. Most of time Retr = 0.

MS510TX is not without its own quirks. The user manual has no mention how to update the firmware and the firmware update guide says "refer to manual". The firmware upload page is located at "Maintenance -> HTTP File Download" (not called upload, gee...) and then choose option called "Software" (there is only two options: software and text config), which used to be labeled as "Upgrade" according to an older YouTube video. I speculate that NetGear tries to sell customer their cloud management "insight subscription" service and made firmware upgrade page hard to find.
It does have dual firmware, meaning that it will keep a old version of firmware in case the firmware upgrade goes south.
The menu just like STH review, is full of features but ancient UI. It would be much appreciated if they could add quick hint when you hover on an setting.

Anyway, it serve me well as a stop gap solution to use device that only has 2.5G and no PCIE expandability for the next five years at least.
I believe that for very long that the manufacturers will mass produce device with 2.5G as it strikes a good cost and performance balance (not for the customer) but for the their profit. For customer, if the client device has more PCIE expandability, the no braining choice is 10G SFP+ cards and all 10G SFP+ ports switches. It is not more expensive than multi-giga switch. Also SFP+ runs cooler than RJ45-SFP+ adapters.

My plan in the future is to have a dual 10G port firewall. Maybe I will build pfsense, but I am afraid of the hassle of its own quirks. Since it is the main connection to the internet. I prefer hassle free turn key solution. Unless some one convinces me otherwise, Haha.
One of the 10G port goes to my NetGear MS510, the other goes to a maybe 9 port Mikrotik SFP+ switch if they could fix the pause frame/flow control issue. Maybe a third one is a 1G or 2.5G POE switch for POE cameras and APs.

Thanks everyone.
Cheers.
 

jena

New Member
May 30, 2020
22
5
3
4Gbit I believe. TP Link sells a 2.5G one in China. I've seen other generic ones available.
Nice find.
I wonder if my surplus Cisco Catalyst Express 500G (with SFP port) can run at 2.5G speed.