Looking for a NIC

[Dave Corder]

Hi all,

I am looking for a dual-port PCIe NIC that supports both a 10 Gbps SFP+ DAC and a 2.5 NBase-T SFP+ module. Also, it must be supported by pfSense

I currently have a Dell R210ii as my pfSense router. I have Comcast 1000/35 cable service. I use one of the onboard 1 GbE ports as the uplink to my modem (a Netgear CM1150V) and a Chelsio S310-CR 10GbE single port NIC as the link to my ICX6610-48P core switch using a DAC.

I've heard that Comcast is starting to bump the download speed to 1200 Mbps. I don't have a problem with my download speeds as it stands, but if that speed is available to me I might as well see what it would take to take advantage of it. My (currently limited) understanding is that I'd need to switch to one of the available modems with a 2.5 Gbps NBase-T LAN port (like the Xfinity XB7 in bridge mode or the Netgear CM2050V), and then figure out how to plug everything together.

Given that my R210ii only has one PCIe slot, I'd need to swap out my NIC for something that would support an 2.5 Gbps NBase-T connection to the modem and a 10 Gbps connection to my core switch.

I do have these NICs available to me already (curse you STH deals forum...):
Supermicro AOC-STGN-I2S rev2 (Intel X520-DA2 equivalent)
Mellanox ConnectX-3 IB/EN 40Gbps QFSP card, flashed to OEM firmware (and I have a couple of spare QSFP to SFP+ adapters).
Chlsio N320E-SR Dual-port 10Gbps SFP+ adapter.

I've heard that many SFP+ switches are picky about actually working properly with NBase-T modules, but I don't know much about the NIC sid, so I'm looking for some suggestions on SFP+ NICs that are known to work well with NBase-T adapters. Any recommendations?

I suppose I could also get an Intel X520-T2 (or newer equivalent), and put a 10Gbps RJ25 module in the switch instead, but that's more money than (hopefully) just being able to use one of my existing cards and getting an SFP+ RJ45 module.

Thanks for any input!

Edit: if there's a Chelsio NIC that would work, that would be better, because then I wouldn't have to change any of the configuration on my pfSense box for my VLANs, which are all based on the cxgb0 interface... (edit2: which I just realized I also have in my spare parts drawer)

 

[Fallen Kell]

I said to hell with it on my pfsense system and installed a ConnectX-3 and connected my pfsense system to my network at 40Gb using a router-on-a-stick topography. If/when Comcast bumps up speeds I can just change whatever port I connect the cable modem in to my core switch to support whatever speed they need (using either bonded Gbe ports, 2.5Gbps transceivers in SFP+ ports, or SFP+ ports) and just set the VLAN on that port(s) to be my WAN VLAN.

 

[Dave Corder]

I said to hell with it on my pfsense system and installed a ConnectX-3 and connected my pfsense system to my network at 40Gb using a router-on-a-stick topography. If/when Comcast bumps up speeds I can just change whatever port I connect the cable modem in to my core switch to support whatever speed they need (using either bonded Gbe ports, 2.5Gbps transceivers in SFP+ ports, or SFP+ ports) and just set the VLAN on that port(s) to be my WAN VLAN.

That's not a bad idea. A tad bit more work to re-configure, but certainly within the realm of possibility (at least for me...we'll see if my wife lets the internet be down long enough to do all that, lol).

 

[Dave Corder]

That's not a bad idea. A tad bit more work to re-configure, but certainly within the realm of possibility (at least for me...we'll see if my wife lets the internet be down long enough to do all that, lol).

Upon further reflection, I'm not sure how well this would work, given my core switch is an ICX6610, which is known to not play nice with most RJ45 SFP+ modules.

Doing some more research, it appears my options are:

• Use an Aquantia AOC-107 RJ45 module in either a NIC or my switch (pricey)
• Use an Intel X550-T2 or X710-T2 card (which support NBase-T, but both are pricey) for 2.5 Gbps to the modem and 10 Gbps to a (cheaper) RJ45 SFP+ module on the switch.
• Get another switch (Mikrotik CRS305-1G-4S+IN maybe?) to use basically just as a media converter

 

[Dave Corder]

Just to follow up on this...

I ended up swapping out the single-port Chelsio 10 Gbps card in my firewall (an S310, IIRC) for a dual-port version (a "T320 10GbE Dual Port Adapter" according to pciconf) I had lying around. It was almost dumb luck that I realized I had this card to use in the first place, as it meant that it would be an easy swap (interface names wouldn't change, nor would any of my VLANs based on cxgb0).

I settled on the Aquantia AQS-107 module, as it promised full compatibility at all speeds with "10 Gbps-only" SFP+ ports, whereas it seemed all the other multi-gig RJ45 modules had some issues in this area. I purchased a AOM-AQS-107-B0C2-CX direct from SuperMicro, to ensure I got the genuine article and not some "compatible" module from an unscrupulous eBay seller or something.

I switched to the Xfinity XB7 gateway in bridge mode and the Chelsio/Aquantia combo this afternoon...just needed to physically swap the cards, and then reassign my WAN interface to cxgb1. So far so good...connection has been stable all afternoon and this evening.

pfSense shows the media as "Unknown <full-duplex>", but the XB7 shows a 2500 Mbps connection on its end. Supposedly my current 1000/35 Comcast connection is overprovisioned to around 1200/40...I'm waiting for a bit later in the evening when things are hopefully a bit quieter to try some speed tests and see what I can actually pull down.

 

[Jenfil82]

It seems to be quite fortuitous that I have the same setup as you where I just bought an R210 ii and am getting started on pfSense.

How is your connection holding up so far with your Aquantia module?

I still have to buy a 10Gb NIC and was wondering why you went with SPF+ rather than copper from the R210ii to the ICX6660, using one of the 48 RJ45 ports?

 

[Dave Corder]

It seems to be quite fortuitous that I have the same setup as you where I just bought an R210 ii and am getting started on pfSense.

How is your connection holding up so far with your Aquantia module?

I still have to buy a 10Gb NIC and was wondering why you went with SPF+ rather than copper from the R210ii to the ICX6660, using one of the 48 RJ45 ports?

The Aquantia module has been great so far...nice and stable. My connection did actually drop for a couple minutes last night for the first time (that I've noticed) since I switched to it, but I think that was a blip with the Xfinity gateway and/or Xfinity network and not my gear.

I went with the SFP+ connection to my ICX6610 just so it could be a 10 Gbps link and not be bottlenecked there to 1 Gbps on one of the 48 RJ45 ports (have 10 Gbps, might as well use it, right?).

 

[Jenfil82]

Makes sense indeed, would be stupid not to use our toys properly