GBASE-T transceivers performance with NBASE-T - Marvell 88x3300 v/s Aquantia AQS-107

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
H

hmw

Active Member
Apr 29, 2019
605
244
43
I've been testing GBASE-T transceivers from Ipolex, Wiitek and Aquantia. The endpoints are connected to a variety of USB based 2.5GbE and 5GbE NICs, as well as a couple of Macs with TB3 based 10GbE NICs. The RJ45 runs are < 30m and are connected to a ICX-6610 switch. In order to get > 1GbE over copper, I've used NBase-T transceivers in the SFP ports of the ICX-6610. The other endpoints are connected via 10GbE and 40GbE via the SFP/QSFP ports on the ICX6610 and with DAC. Refer to the diagram below for how the usual setup would look like:

What I've observed is:
  • When using 10Gbase-T all the transceivers work as expected. I get 10 GbE connectivity and iPerf gets ~ 9.5 Gbit/s in both directions

  • When connected to any endpoint at 2.5G or 5G, the Aquantia transceiver can sustain 2.5G or 5G in both directions - actually 2.38 Gbit/s and 4.7 Gbit/s

  • With the Marvell based transceivers (Ipolex, Wiitek) I get the full 2.5G or 5G throughput when sending from the endpoint to any other endpoint with 10G. But the reverse shows iPerf struggling to maintain 900 mbit/s in the case of 2.5 GbE and 1.4 Gbit/s when connected as 5GbE
I've seen this with Mac, Windows and Linux, so this rules out anything OS-specific.

I've tried traffic shaping the ingress and egress on the ICX-6610 with no appreciable effects i.e. with a 5GbE connect, if I clamp the traffic at 1 Gbit/s - it works as expected, iperf will show 900 mbit/s. But if I raise that to 5 gbit/s, iperf will show the same 1.4 Gbit/s speed. (IMHO, this rules out that the cause for this is the speed difference). It's also possible to do something similar via iperf -b <bandwidth> to set the bandwidth. And it gives the same behavior. Clamping at 1 Gbit/s for the 2.5 GbE connection for example, gives 900 mbit/s in both directions. Raising that to 2.5Gbit/s (iperf -b 2500M) will give the same 2.4 gbit/s from 2.5G host to 10G host - and 900 mbit/s for 10G host to 2.5G host.

A cursory glance at the forums reveals the behavior is also present when using switches from other vendors

Aquantia AQS-107 based SFP transceivers aren't that common. Supermicro used to have OEM versions of these (and I managed to snag some on eBay) but I think this is strange behavior.

@Patrick - STH has tested many SFP transceivers - do you test for bidirectional bandwidth? have you seen behavior like this? Is this normal for the copper SFP transceivers?


1598625441827.png
 
Last edited:
B

bbqdt

Member
Sep 15, 2019
93
64
18
I think the difference is the aqs-107 is specifically designed for adapting 10g only sfp+ ports to NBASE-T. While the Marvel ones just happen to work somehow.


Aquantia SFP+ modules deliver an active rate-adaptation technology that allows end-users to connect seamlessly using existing SFP+ slots that are 10GbE fiber-optic only—offering a significant cost savings over standard fiber modules.
Click to expand...
 
H

hmw

Active Member
Apr 29, 2019
605
244
43
lowfat said:
What modules are you buying that has the aqs-107?

This says 'AOM-AQS-107-B0C2-CX Compatible', not sure what exactly that means tho.

Supermicro AOM AQS 107 B0C2 CX Compatible 10G SFP to RJ45 10GBASE T Transceiver for sale | eBay

Find great deals on eBay for Supermicro AOM AQS 107 B0C2 CX Compatible 10G SFP to RJ45 10GBASE T Transceiver. Shop with confidence.
www.ebay.ca
Click to expand...
That may or may not have the AQ-107 chip. Anecdotal evidence suggests the earlier OPTCore versions did have the Aquantia chip but later versions have the Marvell 88x3310
 
  • Sad
Reactions: lowfat
N

noduck

Member
Sep 12, 2020
41
13
8
I have used Wiitek and S+RJ10 in Mikrotik CRS305, and get ~5Gbps in both directions (Aquantia 5G PCIe - SFP+ - CR305 - MS510TX - DAC - Server). Note that CRS305 does recognize 2.5G and 5G; but Wiitek does not announce these to the switch (just 10G). Hopefully I can test with an ICX-7250 shortly.

I tried testing earlier with a 2.5G USB nic and wasn't able to get consistent results
 
N

noduck

Member
Sep 12, 2020
41
13
8
Oops, I made a mistake in my test; the connect between CRS305 and MS510TX was 5Gig, so the bottleneck was not in the S+RJ10/Wiitek module.

I redid the test (10G between CRS305 and MS510TX), and have also observed lower performance when the module is sending out traffic (both S+RJ10 and Wiitek).
 
H

hmw

Active Member
Apr 29, 2019
605
244
43
@noduck - I tested with the aquatia 10gbe nic set to 1/5/10. Anything less that 10 is a problem for the switches. You get drops in bidirectional throughout. Need to re-test with the MS510TX but honestly at this point couldn’t be bothered. Either I buy expensive 10GbE TB3 nics or I buy expensive Aquantia transceivers ☹
 
N

noduck

Member
Sep 12, 2020
41
13
8
These systems only have a single PCIe lane available (one 3.0 and one 2.0), so a 10GbE NIC is not going to work.
 
N

noduck

Member
Sep 12, 2020
41
13
8
I tested a much simplified setup, with only a 7250 switch. Two different scenarios, terminate 5G on S+RJ10 or Wiitek.
  1. Aquantia 5G -rj45- S+RJ10 - 7250 - Wiitek -rj45- Wiitek - Emulex 10G: 4.7Gbps in both directions
  2. Aquantia 5G -rj45- Wiitek - 7250 - S+RJ10 -rj45- Wiitek - Emulex 10G: 4.7Gbps vs 1.5Gbps
    I was not able to improve the second scenario through rate limiting.

This suggests to me that multigig works good enough with S+RJ10 and 7250. I cannot test 2.5G at this time.
 
N

noduck

Member
Sep 12, 2020
41
13
8
Following up on previous test results, looking at retransmits and congestion window as reported by iperf3:
  1. Wiitek: 14,000 retransmits during iperf run, window less than 30Kb
  2. S+RJ10: 450 retransmits, window >600Kb
For reference, using MS510TX: 4.7Gbps, 9,000 retransmits, window ~350Kb
 
H

hmw

Active Member
Apr 29, 2019
605
244
43
That is weird, I have NO retransmits with the Aquantia module. But I do see retransmits with the Ipolex and Wiitek ...
 
B

bbqdt

Member
Sep 15, 2019
93
64
18
Same, I don’t get retransmits with aqs-107.

Also, as a follow up, I’m finally using mine. I nabbed a Aruba ap-555 for ~300$ and it runs great at 5gbit with a aqs-107 in my icx-6610
 
Last edited:
C

clownfish

New Member
Jan 4, 2022
1
2
1
Old thread I know, but perhaps this will be useful to someone.

I think this issue is all down to the amount of buffering in the SFP. In this configuration the SFP is receiving data on a 10G interface and forwarding it to a 2.5Gbps interface. The Marvell 88x3310 I am familiar with has just 16KB of buffering and I imagine the 88x3300 commonly used in SFPs is similar. So if the data arriving on the 10G interface bursts even slightly over 2.5Gbps per second the buffer will overflow and packets will drop. If TCP is running it will back-off dramatically. TCP makes some attempts at rate-pacing but it won't be smooth enough to avoid overrunning the 16KB buffer. Even hardware rate-pacing in a switch may not be smooth enough if it uses a periodically topped up token bucket.

I'm not familiar with the AQS-107 but it seems likely SFPs based on this have a decent amount of buffering.

I can think of two ways of getting the full 2.5Gbps with the Marvell device. One is really good rate pacing on the 10G side. Some switches can do this by effectively extending the inter-packet gap. I've had success with this approach using the Marvell 88E6390. The alternative (and I've not tried this) might be pause frames. If the SFP issues pause frames when its buffer is getting full (and section 3.4.1 of the 88x3310 datasheet suggests it can) then the sending device might be able to avoid blowing the buffer. However given such a small buffer I'm by no means certain this will work.
 
  • Like
Reactions: koifish59 and Psycho_Robotico
B

blademan

New Member
Jan 7, 2022
7
2
3
Old post, but confirming that 6610 or 7250 will support Nbase-T sfp+ and link at various speeds. Spec sheet and manufacturer forum thread indicated that they wouldn't support NbaseT, only 1Gb or 10Gb.

Thanks

hmw said:
I've been testing GBASE-T transceivers from Ipolex, Wiitek and Aquantia. The endpoints are connected to a variety of USB based 2.5GbE and 5GbE NICs, as well as a couple of Macs with TB3 based 10GbE NICs. The RJ45 runs are < 30m and are connected to a ICX-6610 switch. In order to get > 1GbE over copper, I've used NBase-T transceivers in the SFP ports of the ICX-6610. The other endpoints are connected via 10GbE and 40GbE via the SFP/QSFP ports on the ICX6610 and with DAC. Refer to the diagram below for how the usual setup would look like:

What I've observed is:
  • When using 10Gbase-T all the transceivers work as expected. I get 10 GbE connectivity and iPerf gets ~ 9.5 Gbit/s in both directions

  • When connected to any endpoint at 2.5G or 5G, the Aquantia transceiver can sustain 2.5G or 5G in both directions - actually 2.38 Gbit/s and 4.7 Gbit/s

  • With the Marvell based transceivers (Ipolex, Wiitek) I get the full 2.5G or 5G throughput when sending from the endpoint to any other endpoint with 10G. But the reverse shows iPerf struggling to maintain 900 mbit/s in the case of 2.5 GbE and 1.4 Gbit/s when connected as 5GbE
I've seen this with Mac, Windows and Linux, so this rules out anything OS-specific.

I've tried traffic shaping the ingress and egress on the ICX-6610 with no appreciable effects i.e. with a 5GbE connect, if I clamp the traffic at 1 Gbit/s - it works as expected, iperf will show 900 mbit/s. But if I raise that to 5 gbit/s, iperf will show the same 1.4 Gbit/s speed. (IMHO, this rules out that the cause for this is the speed difference). It's also possible to do something similar via iperf -b <bandwidth> to set the bandwidth. And it gives the same behavior. Clamping at 1 Gbit/s for the 2.5 GbE connection for example, gives 900 mbit/s in both directions. Raising that to 2.5Gbit/s (iperf -b 2500M) will give the same 2.4 gbit/s from 2.5G host to 10G host - and 900 mbit/s for 10G host to 2.5G host.
Click to expand...
 
D

dag

Member
Apr 23, 2020
27
41
13
Old thread but much needed clarification: the Aquantia SFP does a great job at converting a 10G-only SFP+ port into a NBase-T port, tested at 2.5G and 5G in both directions simultaneously.

*BUT*

You need to enable flow control for it to work well in both directions simultaneously. A quick wireshark dump shows that the module constantly introduces pause frames to shape the traffic, and it works well. I realize many “experts” will argue flow control is a big no-no, but that’s the trick the SFP uses to live up to its promises.

I tried a bunch of other “multigig” SFP modules, and while they perform well in 1 direction, the performance is usually abysmal in the other direction, tons of retries, and flow control does not help.

Unfortunately, most “experts” tend to test in 1 direction, which tends to mask the limitations of their SFPs. I have yet to see a single test pumping traffic in both directions simultaneously.
 
  • Like
Reactions: blunden, mach3.2, koifish59 and 3 others
M

MattTheTechLV

New Member
Mar 26, 2022
12
2
1
I posted a much longer and long-winded post in the other thread about this here on the forum ( https://forums.servethehome.com/index.php?threads/mix-10g-2-5g-slow-speed-high-retr.33217/page-2 ) But I just wanted to quickly say that I just discovered those same AOM-AQS-107-B0C2-CX "Compatible" Transceivers that @lowfat Mentioned earlier in the thread back in 2020. I bought 4 of them here: Supermicro AOM-AQS-107-B0C2-CX Compatible 10GBASE-T SFP+ Transceiver | Optcore.net

And I intend to test them as soon as they arrive to see if they do indeed work as well as the Aquantia it's trying to mimic. I have high hopes, but won't be surprised if they fall short. If they do work they will be a MUCH CHEAPER option for those of us looking to use NBase-T Devices on our SFP+ Switches.

Mine should be here sometime next week, will report back once I have had a chance to test them, which I plan to do the moment they arrive.

(If anyone else has already gotten their hands on one I would love to hear your experience with them. It turns out they are the same exact ones Lowfat posted the link to from eBay, I just happened to find out that the people who make them sell them direct via their website, and they seem to have plenty of stock. So if these do end up working out, it doesn't seem like it will be so hard to get the amount we each need.)
 
Last edited:
D

dag

Member
Apr 23, 2020
27
41
13
Unfortunately the Optcores I received (and returned) were using a Marvell Alaska chip, not the Aquantia AQS107. They worked fine 1 way, but like many others, they petered out when I threw traffic at them in both directions.

Marvell bought Aquantia several years ago, but they essentially killed most of their products since then. Only Supermicro seems to have some of the old/original Aquantia stuff in stock at this point.

Let us know if you got lucky!
 
  • Like
Reactions: MattTheTechLV
M

MattTheTechLV

New Member
Mar 26, 2022
12
2
1
dag said:
Unfortunately the Optcores I received (and returned) were using a Marvell Alaska chip, not the Aquantia AQS107. They worked fine 1 way, but like many others, they petered out when I threw traffic at them in both directions.

Marvell bought Aquantia several years ago, but they essentially killed most of their products since then. Only Supermicro seems to have some of the old/original Aquantia stuff in stock at this point.

Let us know if you got lucky!
Click to expand...
Well damn, I wish I would have posted here before buying them now. But I really appreciate the information. I will let you know if I get lucky, but am far less hopeful now.

I really wish there were cheap 4 to 5 port RJ45 10Gb, Managed (Or at least able to pass VLAN Tags properly), Switches. I can't seem to find any out there around the same price of the Mikrotik CRS305. The CRS305 is perfect except for it lacking the ability to deal with NBase-T, so It has me looking for a solution to get 2.5Gb Port's that pass VLAN Tag's to 3 different Room's in my house for under a few Hundred dollars. Its a little hard to swallow $600 for 3 of the Aquantia Transceivers, but I would gladly pay that if I was getting 3 small 10Gb Switches... So irritating that this market is still no where near as filled out as the 1Gb Switch Market was when we were all first deploying 1Gb Switches everywhere in the early 2000's. Its just frustrating as slow the lower end of the market is taking to bring options to consumers.

Again though, thank you for the info @dag. I really appreciate your posts on the topic, extremely informative.
 
You must log in or register to reply here.
Top Bottom