SFP+ Dual NIC with 10G-BaseT and 5000Base-T support. Does it exist?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
Yeah, I'm pretty sure it's just a Mellanox CX342A CX3. A fellow member on STH figured out how to put this all together in a thread somewhere. He even had instructions on making a one port into a two port and cross flashing with Mellanox firmware.
Yeah, I cross-flashed 4 MCX354A-QCBT (10GbE/40GbIB) to MCX353A-FCBT (40GbE/56GbIB) and it worked great. Same card physically, just different firmware. It's super easy and pain-free.

I heard you can flash them to be the PRO model, which in hindsight I kind of wish I had gone for since they're capable of RoCE v2, but they were like twice as much, and even if they are compatible with the PRO firmware, they still have an inferior ASIC. Now that I am selling them 2-3 years later I guess I'm glad they were only $35/ea, lol.

I'm pretty juiced about going with QLogic now, though. I haven't been happy with the driver support of the CX3s in ESXi 7. Feels like they've been neglected, definitely on the buggy side.

What OS/platform are you using your CX342a on?

Edit: Apparently trying to flash non-pro to pro could brick your CX3 re: ASIC disparity.
 
Last edited:

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
Yeah, I cross-flashed 4 MCX354A-QCBT (10GbE/40GbIB) to MCX353A-FCBT (40GbE/56GbIB) and it worked great. Same card physically, just different firmware. It's super easy and pain-free.

I heard you can flash them to be the PRO model, which in hindsight I kind of wish I had gone for since they're capable of RoCE v2, but they were like twice as much, and even if they are compatible with the PRO firmware, they still have an inferior ASIC. Now that I am selling them 2-3 years later I guess I'm glad they were only $35/ea, lol.

I'm pretty juiced about going with QLogic now, though. I haven't been happy with the driver support of the CX3s in ESXi 7. Feels like they've been neglected, definitely on the buggy side.

What OS/platform are you using your CX342a on?
I'm running on OPNsense right now and have only tried the MCX311A in it for testing... it didn't work so I switched to PF sense where it did work. I learned that in OPNsense you have to enable the Mellanox drivers which I have now done, but I am still waiting for the Mellanox CX342A to arrive. It has already cleared USA customs so should be soon.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
I heard you can flash them to be the PRO model, which in hindsight I kind of wish I had gone for since they're capable of RoCE v2, but they were like twice as much, and even if they are compatible with the PRO firmware, they still have an inferior ASIC.
IIRC, attempting to flash a non Pro card up to Pro resulted in a brick. The ASIC differences are the problem (different clock rates for one). There are a couple of threads around here about flashing/cross-flashing the various COnnectX cards.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
Yes, non-Pro (CX353/354) can be flashed with firmware supporting different IB speeds, with associated PSID change, but cannot be flashed to any Pro firmware.

Keep your RDMA within a VLAN and you won't need RoCE v2.
Thanks for clarification, wouldn't want to spread misinformation

Can you elaborate on the VLAN-confined RDMA vs RoCEv2 statement? They are different things, so not sure what you're referring to.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Thanks for clarification, wouldn't want to spread misinformation

Can you elaborate on the VLAN-confined RDMA vs RoCEv2 statement? They are different things, so not sure what you're referring to.
RoCE = RDMA over Converged Ethernet; v1 is not routable, v2 is. If you don't need to route your RDMA, you basically don't need RoCEv2, though v2 does add some congestion management stuff.
 

Dave Corder

Active Member
Dec 21, 2015
291
186
43
41
Going back to the original question, are there any dual SFP+ NICs out there that play really nicely with Nbase-T transceivers at 2.5Gbps/5Gbps?

I'm currently running a Chelsio T320 10GbE SFP+ NIC in my Dell R210ii running OPNSense with a 10 Gbps DAC connecting it to my ICX6610 core switch, and an Aquantia Nbase-T module (Supermicro-branded AOM-AQS-107-B0C2-CX) connecting to my Xfinity XB7 (in bridge mode) at 2.5 Gbps.

It's working great, the only downside is that OPNSense doesn't know the link speed on its end:

1668640575106.png

It would be nice if it could display the actual link speed instead of "Unknown" just so I can easily make sure it's actually linking at 2.5 Gbps instead 1 Gbe. It's by no means critical, but if there's an easy/cheap alternative NIC I could use that would give me that, I would probably make the switch.
 
  • Like
Reactions: CIR-Engineering

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
I skimmed through the thread mostly. I'm using a T730 with an X550 to go from my 5gbps modem to my 10gbps brocade switch. I added a Noctua fan and haven't seen any issues. My internet is only 1gbps, I'm getting slightly over 1gbps using pfsense though (overprovisioning).

Yeah right now I am running the X550-T2 and I'm starting to think it may wind up working out better than the Mellanox SFP+ card that's coming to possibly replace it. I'll figure that out shortly. I see in your thread that the fan control works in your T730 BIOS. If that setting weren't broken on my T740 with newest BIOS I don't think I would have a problem with heat or need the extra fan.

I need copper to the modem and one really good thing with the X550 is that in OPNsense I can manually set the link speed to 5G. If I don't do that, the link speed auto negotiates to 2.5G and I lose 1GB of my 3.5GB WAN download speed. I'm afraid with the SFP+ copper transceiver I'll lose the ability and might get stuck at just 2.5G. I don' know of a way to manually set the transceiver's link speed to anything but 1G or 10G? With copper, you have all the options. With SFP+ it would require talking directly to the transceiver somehow.

The other thing I did was set the LAN output on the X550 to just 5GB in OPNsense instead of 10GB. This has cooled off the X550 about 8 degrees Celsius. At this point, I'm not sure I need to worry about it anymore. And maybe HP will fix the minimum fan speed setting in the BIOS in a future firmware update.
 

vcbb10

New Member
Nov 24, 2020
13
13
3
Yeah right now I am running the X550-T2 and I'm starting to think it may wind up working out better than the Mellanox SFP+ card that's coming to possibly replace it. I'll figure that out shortly. I see in your thread that the fan control works in your T730 BIOS. If that setting weren't broken on my T740 with newest BIOS I don't think I would have a problem with heat or need the extra fan.

I need copper to the modem and one really good thing with the X550 is that in OPNsense I can manually set the link speed to 5G. If I don't do that, the link speed auto negotiates to 2.5G and I lose 1GB of my 3.5GB WAN download speed. I'm afraid with the SFP+ copper transceiver I'll lose the ability and might get stuck at just 2.5G. I don' know of a way to manually set the transceiver's link speed to anything but 1G or 10G? With copper, you have all the options. With SFP+ it would require talking directly to the transceiver somehow.

The other thing I did was set the LAN output on the X550 to just 5GB in OPNsense instead of 10GB. This has cooled off the X550 about 8 degrees Celsius. At this point, I'm not sure I need to worry about it anymore. And maybe HP will fix the minimum fan speed setting in the BIOS in a future firmware update.
You might be able to use an internal USB port if the T740 has it. That's easy 5v to find and should default the fan to max speed. The Noctua also comes with an included adapter to lower the fan speed.
 

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
Going back to the original question, are there any dual SFP+ NICs out there that play really nicely with Nbase-T transceivers at 2.5Gbps/5Gbps?

I'm currently running a Chelsio T320 10GbE SFP+ NIC in my Dell R210ii running OPNSense with a 10 Gbps DAC connecting it to my ICX6610 core switch, and an Aquantia Nbase-T module (Supermicro-branded AOM-AQS-107-B0C2-CX) connecting to my Xfinity XB7 (in bridge mode) at 2.5 Gbps.

It's working great, the only downside is that OPNSense doesn't know the link speed on its end:

View attachment 25467

It would be nice if it could display the actual link speed instead of "Unknown" just so I can easily make sure it's actually linking at 2.5 Gbps instead 1 Gbe. It's by no means critical, but if there's an easy/cheap alternative NIC I could use that would give me that, I would probably make the switch.
I actually have four ipolex 10GBase-T RJ45 transceivers that I bought a couple years ago. I specifically chose the model because it said it supported 2.5G and 5G. However, due to my ignorance, I thought the switch / NIC also had to support the feature and I never even tried it until others here on STH told me it should work... and these do indeed work beautifully in both my Brocade and Mellanox NIC's.

1.jpg

2.jpg

I bought these two packs which at the time specifically said 2.5G and 5G, but no longer specify as such:

Now they have another model that costs slightly more that does specify 2.5G and 5G support. I suspect they are all the same and all support the features but can't say for sure.

Now I am on a quest for the lower 1.8 watt versions:
 

AveryFreeman

consummate homelabber
Mar 17, 2017
413
54
28
42
Near Seattle
averyfreeman.com
I need copper to the modem and one really good thing with the X550 is that in OPNsense I can manually set the link speed to 5G. If I don't do that, the link speed auto negotiates to 2.5G and I lose 1GB of my 3.5GB WAN download speed. I'm afraid with the SFP+ copper transceiver I'll lose the ability and might get stuck at just 2.5G. I don' know of a way to manually set the transceiver's link speed to anything but 1G or 10G? With copper, you have all the options. With SFP+ it would require talking directly to the transceiver somehow.
I came across this thread looking for compatible modules for my switch, and saw they were discussing some SFP+ modules autonegotiating from 10Gbps down to 5Gbps or 2.5Gbps: https://community.ui.com/questions/...r-5-gbps/3f9aa1a8-5297-48e8-8ed6-4c23059d091c

FWIW, maybe there's some modules/settings people are having luck with over there...

Edit: sorry sent this before page refresh revealed you had some mods that are working
 

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
Going back to the original question, are there any dual SFP+ NICs out there that play really nicely with Nbase-T transceivers at 2.5Gbps/5Gbps?

I'm currently running a Chelsio T320 10GbE SFP+ NIC in my Dell R210ii running OPNSense with a 10 Gbps DAC connecting it to my ICX6610 core switch, and an Aquantia Nbase-T module (Supermicro-branded AOM-AQS-107-B0C2-CX) connecting to my Xfinity XB7 (in bridge mode) at 2.5 Gbps.

It's working great, the only downside is that OPNSense doesn't know the link speed on its end:

View attachment 25467

It would be nice if it could display the actual link speed instead of "Unknown" just so I can easily make sure it's actually linking at 2.5 Gbps instead 1 Gbe. It's by no means critical, but if there's an easy/cheap alternative NIC I could use that would give me that, I would probably make the switch.
1668890007532.png

The LAN interface I set manually in OPNsense to 5000Base-T and it works brilliantly withe the ipolex plugged into the SFP+ in the Brocade.
 

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
You might be able to use an internal USB port if the T740 has it. That's easy 5v to find and should default the fan to max speed. The Noctua also comes with an included adapter to lower the fan speed.
I'm using an external USB port at the moment. However, my main problem is that I need more fresh air into the case. Putting a fan on the heatsink is just going to move really hot air around inside the case I think.

Like you, I could not easily find a 12v header that I felt confident could support the required current.

 

CIR-Engineering

I am a functional adult?
Jan 14, 2021
85
30
18
48
Chicago USA
www.cir-engineering.com
Just to note, HP came out with a new firmware 1.15. Nothing in the documentation suggested it would fix the fan speed issues, but I held out hope in vain. Evidently there was a serious security issue with version 1.14 as it's been removed from HP's web site but 1.13 is still up. Oh well.