Mix 10G 2.5G, slow speed, high Retr

[jena]

Update:
I end up returned the Mikrotik CRS305 and bought NetGear MS510TX.

TLDR:

• It looks like Mikrotik still have pause frame/flow control issue with 2.5/5G, with common RJ45-SFP+ modules (6COM, ipolex, etc..)
• The only known RJ45-SFP+ module that works at 2.5G/5G is Supermicro AOM-AQS-107-B0C2-CX (Aquantia AQS-107-CX-1)
• IMO, take caution with QNAP M308/408/2108 switches, buy at place that you can return. There seems to be reports on its firmware quirks and QA issue. In general, a SOHO product, not a solid build.

Questions
Q1: Seek diagnose suggestions. see my testing.

My guess is that when 10G NIC is sending packets to MikroTik switch, which is not aware of the receiving end is at 2.5G.
Thus it pumps data as fast as it can out of 10G's TX buffer and receiving 2.5G NIC RX buffer couldn't handle and drops them a lot. Thus Re-transmit is needed.

Q2.Will high Retr corrupts transmitted data (like from PC to SMB, SMB to PC)?


Setup
Router: EdgeRouter X
Switch 1 (10G): MikroTik CRS305-1G-4S+IN,

PC1 - Switch1: RJ45 - SFP+ adapter, CAT6 cable (all are quality cable and less than 25ft, I tried a few different cables, no change)
SFP+ RJ45 module that I have tried: 6COM, ipolex, GWFiber. They all claim to support 2.5G
Use three different brand of module, doesn't seems to have significant difference or solve my high Retr and slow speed

10G to Switch1: New 10G Tek DAC cable, New GWFiber 10G DAC cable. No change in terms of speed.

PC2: I borrowed a buddy's i5-7500 and B150M motherboard. Mellanox MCX-311A 10G NIC is plugged in slot 1 x16. Default driver in Linux Mint 20. (10G NIC was used in UNRAID, but I took it out and use PC2 to isolate problem.)

PC1: (ASUS B550 Intel I225V 2.5G, Rev.3, FW 1.57), TX/RX buffer = 1024, flow control enabled.
There are quite a bit report on Rev1 and 2 of I225, dropping packets. Mine is Rev3, which is "supposed" to be free of previous issue.
Driver for Windows 10 is the latest - 1.0.1.4.

Server2: Open Media Vault RTL8125 2.5G NIC, Linux 5.10.0-0.bpo.5-amd64, uses RTL8169 driver that was built in.
Use putty to SSH and test iperf.
Adapter settings: see detail at the end
I also tried to compile latest Realtek RTL8125 driver and the performance is even worse, at 110MB/s when receiving data from 10G NIC

All default, MTU 1500.

iperf3

• host Server2 (RTL8125) to client PC1 (Intel I225V)
  • 283 MBytes/sec, no Retr displayed in output
• host Server2 (RTL8125) to client PC2 (10G NIC)
  • 259 MBytes/sec, total Retr = ~60000
• host PC1 (I225V) to client PC2 (10G NIC)
  • 160 MBytes/sec (interesting...), total Retr = ~30000
• host PC1 (I225V) to client Server2 (RTL8125)
  • 221 MBytes/sec, total Retr = 28
• host PC2 (10G NIC) to client PC1 (I225V)
  • 283 MBytes/sec, no Retr displayed in output
• host PC2 (10G NIC) to client Server2 (RTL8125)
  • 229 MBytes/sec, Retr = 0

####### Server2 2.5G RTL8125
Settings for enp5s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
                                             10000baseT/Full
                                             2500baseT/Full
                                             5000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes


####### PC2 10G NIC
ethtool enp1s0
Settings for enp1s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
    Current message level: 0x00000014 (20)
                   link ifdown
    Link detected: yes

 

[RTM]

AFAIK Mikrotik has fixed a lot of that kind of issues in newer firmwares, have you made sure to update the firmware of the switch?

If you are using SwitchOS, you should try RouterOS as fixes are usually made available there sooner than SwitchOS.

 

[jena]

AFAIK Mikrotik has fixed a lot of that kind of issues in newer firmwares, have you made sure to update the firmware of the switch?

If you are using SwitchOS, you should try RouterOS as fixes are usually made available there sooner than SwitchOS.

I am running switchOS with latest firmware 2.13 (built at Mon Apr 26 2021).

I will try routerOS.

Update: I switch to routerOS, update to latest stable version. all default settings.
The results are the same.
Still 160MB/s when 10G is sending and 2.5G intel NIC is receiving.

 

[unmesh]

@jena

I've tried different hardware and OSes but same basic topology which is client - 2.5G NIC (USB or PCIe) - SFP+ copper transceiver - switch - 10GDAC - server and could only get sub-2.5G performance and lots of retries like you. What did work was to borrow a QNAP QSW-M408-4C switch that had multirate RJ45 ports that supported 100/1000/2500/5000/10000 and use one of those instead of a SFP+ port with a copper transceiver

I'm not ready to buy a new 10G switch and am trying to get an Aquantia based SFP+ copper transceiver that someone reported has worked for them. Something about using Pause Frames to achieve flow control.

These are devilishly difficult to find.

 

[jena]

@jena

I've tried different hardware and OSes but same basic topology which is client - 2.5G NIC (USB or PCIe) - SFP+ copper transceiver - switch - 10GDAC - server and could only get sub-2.5G performance and lots of retries like you. What did work was to borrow a QNAP QSW-M408-4C switch that had multirate RJ45 ports that supported 100/1000/2500/5000/10000 and use one of those instead of a SFP+ port with a copper transceiver

I'm not ready to buy a new 10G switch and am trying to get an Aquantia based SFP+ copper transceiver that someone reported has worked for them. Something about using Pause Frames to achieve flow control.

These are devilishly difficult to find.

Thank you for the information!

Is there an example model number of "an Aquantia based SFP+ copper transceiver" that you suggest?
Do you by chance have the source of the post about the Aquantia based transceiver?

When I bought Mikrotik CRS305, I also looked at QNAP M308 and M408 and thought Mikrotik might be more robust and better cooling.
M308/408 looks more like prosumer grade SOHO switch. Well, maybe QNAP is a better choice in mixed 2.5G situation.

I still have about a week left to return CRS305 switch and transceiver and go with QNAP.
Adding two transceiver already brings the total cost to about $220 and QNAP M408 is about $250-300 depending on the combo ports.
I am just not sure of QNAP M408 or M2108 have the same quality and reliability as Mikrotik.

Update: I will steer clear from QNAP switch products. M308/408/2108 doesn't seems to be well made and occasionally QA issues and quit working (require reboots) reported in Amazon's review. The management UI and features are amateur (in a way that doesn't seems to be well supported for years to come). Also given previous QNAP NAS severe security vulnerability issues, it does not inspire confidence in terms of QNAP firmware.

It seems that Netgear MS510TX might be a good choice. Then add a 9 port Mikrotik (all SFP+) in the future when more 10G ports are needed.

I would skip 2.5G if I could, but my ITX motherboard only have on board 2.5G and also old machine only PCIE x1 left for a 2.5G NIC.

Thanks!

 

[DDD]

Is there an example model number of "an Aquantia based SFP+ copper transceiver" that you suggest?
Do you by chance have the source of the post about the Aquantia based transceiver?

The only RJ45 - SFP+ transciever that worked for me (with a ubiquiti es 16-xg and intel i255-v & usb 5g adapter) was the Aquantia AQS-107-CX-1. I have a Supermicro OEM version ( AOM-AQS-107-B0C2-CX ) which unfortunately was very expensive, but I didn't want to install an additional switch.
Had to enable flow-control on the port.

 

[unmesh]

Do you by chance have the source of the post about the Aquantia based transceiver?

https://forums.servethehome.com/index.php?threads/gbase-t-transceivers-performance-with-nbase-t-marvell-88x3300-v-s-aquantia-aqs-107.30004/

 

[unmesh]

...
It seems that Netgear MS510TX might be a good choice. Then add a 9 port Mikrotik (all SFP+) in the future when more 10G ports are needed.
...

It is on my short list but has only 2 10G ports and I already have two servers on 10G. MS510TXM would be better but is $500. Hence my quest for a working copper multirate SFP+ transceiver since there are plenty of used switches with SFP+ ports available for not much money.

Oh, and plenty of compalints about Mikrotik SwOS quality for their new switches on their forums.

 

[coxhaus]

Cisco small business switches have some 2.5 gig model switches in the SG350X line and probably the new Cisco CBS switch line. I have not seen any cheap yet.

 

[NateS]

Trendnet also has an 8x 2.5g + 2x 10G SFP+ switch that's a bit cheaper than the Netgear offerings at 400. I don't have any experience with them myself, but I've been thinking of picking one up to add some 2.5G ports to my main 10G network.

Amazon.com: TRENDnet 10-Port 2.5GBASE-T Web Smart Switch, 8 x 2.5GBASE-T RJ-45 Ports, 2 x 10G SFP+ Slots, Lifetime Protection, TEG-30102WS, Black: Computers & Accessories

Amazon.com: TRENDnet 10-Port 2.5GBASE-T Web Smart Switch, 8 x 2.5GBASE-T RJ-45 Ports, 2 x 10G SFP+ Slots, Lifetime Protection, TEG-30102WS, Black: Computers & Accessories

www.amazon.com

 

[jena]

Trendnet also has an 8x 2.5g + 2x 10G SFP+ switch that's a bit cheaper than the Netgear offerings at 400. I don't have any experience with them myself, but I've been thinking of picking one up to add some 2.5G ports to my main 10G network.

Amazon.com: TRENDnet 10-Port 2.5GBASE-T Web Smart Switch, 8 x 2.5GBASE-T RJ-45 Ports, 2 x 10G SFP+ Slots, Lifetime Protection, TEG-30102WS, Black: Computers & Accessories

Amazon.com: TRENDnet 10-Port 2.5GBASE-T Web Smart Switch, 8 x 2.5GBASE-T RJ-45 Ports, 2 x 10G SFP+ Slots, Lifetime Protection, TEG-30102WS, Black: Computers & Accessories

www.amazon.com

Nice. 4 more 2.5G ports compare to NetGear MS510TX

 

[jdnz]

the engenius ecs2512 gives you 8 2.5gbe ports and 4 sfp+ ports ( so two more sfp+ ports than the trendnet).

 

[NateS]

That engenius will be nice if it comes down in price in the future. At 600 though, those two extra 10g SFP+ ports are 100 each, which is way too much IMO, given that I can get a brocade with 8 SFP+ ports for ~200.

I do think the 8x 2.5 + 2x 10 is a nice combination, since all the 2.5g ports would have full bandwidth to the 10G network simultaneously if both 10g ports are used as uplinks.

 

[unmesh]

Mixed news to report. I got my hands on an AQS-107 based transceiver but my Juniper EX2300C switch does not recognize it whereas it did recognize the Marvell based transceivers I had tested previously

Looks like I will have to buy a new switch after all in which case I might as well buy one with native NBaseT support.

 

[lowfat]

I've had the same issues. I ended up just using 10GbE or 1GbE. I have seen some 2.5G SFP (not +). Anyone know if these work?

 

[unmesh]

I thought the SFP electrical side maxed out at 1G.

Since I had the SFP+ transceiver, I decided to try it in a Mellanox CX3 PCIe NIC with a Realtek based USB 2.5G NIC at the other end even though this is not something I would use in practice.

Linux on both systems; ethtool showed the Mellanox link at 10G as expected but the Realtek side would only autonegotiate 1G and the ethtool/driver (r8152 version 2.15) combination would not let me manually set it to 2.5G.

The OP's 2.5GbE NIC is PCIe and is showing a link speed of 2500.

 

[lowfat]

I thought the SFP electrical side maxed out at 1G.

4Gbit I believe. TP Link sells a 2.5G one in China. I've seen other generic ones available.

TL-SM410U 2.5G SFP电口模块 - TP-LINK官方网站

敬请关注TP-LINK 2022春季新品发布会。TP-LINK是全球领先的网络通讯设备供应商, 产品涵盖以太网、无线局域网、宽带接入、电力线通信，在既有的传输、交换、路由等主要核心领域外，正逐步进入移动互联网终端、数字家庭、网络安全等领域。TP-LINK始终致力于为大众提供最便利的本地局域网络互联和Internet接入手段，为大众在生活、工作、娱乐上日益增长的网络使用需求，提供高品质、高性能价格比的全面设备解决方案。

www.tp-link.com.cn

 

[jena]

Update: I end up returned the Mikrotik CRS305 and bought NetGear MS510TX.

I let most settings default on MS510TX, and set "Global Flow Control (IEEE 802.3x) Mode" Enabled.
10G can reach 1100MByte/s, 2.5G can reach 280MByte/s. And there is no issue sending and receiving between 10G and 2.5G. Very occasionally, I get Retr = less than 10. Most of time Retr = 0.

MS510TX is not without its own quirks. The user manual has no mention how to update the firmware and the firmware update guide says "refer to manual". The firmware upload page is located at "Maintenance -> HTTP File Download" (not called upload, gee...) and then choose option called "Software" (there is only two options: software and text config), which used to be labeled as "Upgrade" according to an older YouTube video. I speculate that NetGear tries to sell customer their cloud management "insight subscription" service and made firmware upgrade page hard to find.
It does have dual firmware, meaning that it will keep a old version of firmware in case the firmware upgrade goes south.
The menu just like STH review, is full of features but ancient UI. It would be much appreciated if they could add quick hint when you hover on an setting.

Anyway, it serve me well as a stop gap solution to use device that only has 2.5G and no PCIE expandability for the next five years at least.
I believe that for very long that the manufacturers will mass produce device with 2.5G as it strikes a good cost and performance balance (not for the customer) but for the their profit. For customer, if the client device has more PCIE expandability, the no braining choice is 10G SFP+ cards and all 10G SFP+ ports switches. It is not more expensive than multi-giga switch. Also SFP+ runs cooler than RJ45-SFP+ adapters.

My plan in the future is to have a dual 10G port firewall. Maybe I will build pfsense, but I am afraid of the hassle of its own quirks. Since it is the main connection to the internet. I prefer hassle free turn key solution. Unless some one convinces me otherwise, Haha.
One of the 10G port goes to my NetGear MS510, the other goes to a maybe 9 port Mikrotik SFP+ switch if they could fix the pause frame/flow control issue. Maybe a third one is a 1G or 2.5G POE switch for POE cameras and APs.

Thanks everyone.
Cheers.

 

[jena]

4Gbit I believe. TP Link sells a 2.5G one in China. I've seen other generic ones available.

TL-SM410U 2.5G SFP电口模块 - TP-LINK官方网站

敬请关注TP-LINK 2022春季新品发布会。TP-LINK是全球领先的网络通讯设备供应商, 产品涵盖以太网、无线局域网、宽带接入、电力线通信，在既有的传输、交换、路由等主要核心领域外，正逐步进入移动互联网终端、数字家庭、网络安全等领域。TP-LINK始终致力于为大众提供最便利的本地局域网络互联和Internet接入手段，为大众在生活、工作、娱乐上日益增长的网络使用需求，提供高品质、高性能价格比的全面设备解决方案。

www.tp-link.com.cn

Nice find.
I wonder if my surplus Cisco Catalyst Express 500G (with SFP port) can run at 2.5G speed.

 

[dag]

Old thread but much needed clarification: the Aquantia SFP does a great job at converting a 10G-only SFP+ port into a NBase-T port, tested at 2.5G and 5G in both directions simultaneously.

*BUT*

You need to enable flow control for it to work well in both directions simultaneously. A quick wireshark dump shows that the module constantly introduces pause frames to shape the traffic, and it works well. I realize many “experts” will argue flow control is a no-no, but that’s what the trick the SFP uses to live up to its promises.

I tried a bunch of other “multigig” SFP modules, and while they perform well in 1 direction, the performance is usually abysmal in the other direction, tons of retries, and flow control does not help.

Unfortunately, most “experts” tend to test in 1 direction, which tends to mask the limitations of the SFPs. I have yet to see a single test pumping traffic in both directions simultaneously.