Build/MiniPC/Server for pfsense build capable of SFP+ 2.5Gbps for google fiber

[themeuge]

I am currently very happy with my N6005 topton box, but google fiber is offering 2gb and potentially going to 5gb in my area soon. I heard that people have used the Unifi UDM Pro with SFP+ input to directly receive the optical signal from Google fiber, but I'm not interested in the UDM Pro and want to use Pfsense instead.

Does anyone have any knowledge or experience to help figure out how to do that short of using a server build?

Thanks in advance.

 

[NoncarbonatedClack]

It partially depends on what features you're going to use of pf, and what are your other wants/needs?
IPMI?
IDS and IPS will add a larger CPU requirement.

What features are you currently using on the topton box?

You could look at the netgate hardware for a starting point of what to buy, and go from there:
Edit: meant this as a hardware thing. like if you need x throughput with y features, check and see what hardware netgate is using (atom, xeon, etc), and then you can start building from there. Wasn't suggesting purchasing from netgate, although you could.



You would probably be fine with a somewhat modern i5/i7 and 8GB RAM.

Project Tiny/Mini/Micro would be a good place to look for a starting point as well. I did not go this route, and went with an intel nuc instead (my thread here, if it helps, there are suggestions in there.)

 

[oneplane]

I think any C3000 series used VEP or SuperMicro 3xx series will do the trick here. By default those SoCs have plenty of I/O on board with 30Gbps fdx traffic in mind. Unless you enjoy pfSense I'd not use Netgate for this.

 

[NoncarbonatedClack]

I think any C3000 series used VEP or SuperMicro 3xx series will do the trick here. By default those SoCs have plenty of I/O on board with 30Gbps fdx traffic in mind. Unless you enjoy pfSense I'd not use Netgate for this.

I should clarify (and will edit my post accordingly) that I meant to use netgate's hardware offerings as an idea of where to start with a custom build, given netgate has (hopefully lol) done the research already.

Definitely didn't mean "go buy netgate!". Even though I'm considering purchasing something from the opnsense store.

 

[oneplane]

I'd even think OpnSense's shop stuff might not be the best fit, but in both cases you are also right in that they did their homework and it does give a good idea on what cores/nics/memory to expect for packets per second and bandwidth numbers.

It is getting a bit trickier now with ARM and x86 options, it used to be "get 2 x86 cores at or above 1.8ghz" and it would be all the same as long as you stuck to similar generations. With ARM it varies more drastically, but it can also get a real good bang for the buck.

If we look at all of the stuff so far (Unifi UDM Pro, SuperMicro, Netgate, OpnSense A-series, maybe VEP14xx) it does seem to have a bunch of similarities:

- Most systems with ports that have more than 2Gbps bandwidth tend to have at least 4 cores
- Memory does't seem to be much of a concern as even base configurations with 4GB work out fine
- Medium and high-end SoCs already have the network fabric with enough bandwidth on-chip
- Pricing for board+NIC+SoC seem to be really very similar with branding, interface options and form factor driving most of the final cost

So now all we need to know is what the budget is

 

[themeuge]

It partially depends on what features you're going to use of pf, and what are your other wants/needs?
IPMI?
IDS and IPS will add a larger CPU requirement.

What features are you currently using on the topton box?

I have fairly extensive IDS/IPS filtering. I am pretty close to 8GB of RAM already with all the filters, and pulling line speed on the N6005 box uses about 70% CPU with my filter set - I upgraded from an earlier generation Atom box because I needed the extra processing power.
The Netgate options are good for business, but rather expensive for home lab, and are quite behind the curve in terms of power efficiency.

I can easily just build this with an embedded small server board which will have both IPMI and a separate SFP+ NIC, but that will likely cost close to $1000 with the case, NIC, etc, and probably use more power.

The little Aliexpress boxes are just so incredibly useful and inexpensive, that I've become spoiled for choices, and would love to see SFP+ included in something similar.

 

[oneplane]

I think the remaining options are used C3000 series boxes then, they exist with SFP+ dual 10G interfaces aplenty.
Stuff like this: https://www.etb-tech.com/dell-emc-vep1425-switch6-x-1gb-rj45-2-x-10gb-sfp-sw01802.html

 

[abq]

I think the remaining options are used C3000 series boxes then, they exist with SFP+ dual 10G interfaces aplenty.
Stuff like this: https://www.etb-tech.com/dell-emc-vep1425-switch6-x-1gb-rj45-2-x-10gb-sfp-sw01802.html

Those look like nice units (re:Dell EMC VEP1425 Switch). ...Have you seen any good 'how to' information on opening these up to *sense or other non-dell software?

 

[oneplane]

Those look like nice units (re:Dell EMC VEP1425 Switch). ...Have you seen any good 'how to' information on opening these up to *sense or other non-dell software?

I use them all over the place. They are essentially just normal SBC servers but without graphics. They have a USB port on the back that works as a serial console with normal UEFI access. It boots off of normal storage too, BSD and Linux all have drivers for everything as well, except the CPLD which doesn't mean much unless you want the power LED on the front to change color, and the LEDs on the ports to have different blinking patterns instead of default ones.

They also have USB ports which they will boot off of. A nice thing is that they are always-on by default, so as long (and as soon as) there is power, they are on. They also have a locking barrel style power connector so it doesn't unplug by accident, but also accept non-locking connectors so it works with pretty much anything.

A common deployment for companies that need a bit more than just a single subnet and NAT (but don't have the budget) is a DC management switch (i.e. Quanta LB9 style), a VEP, a 10G DAC between them and off you go. Since there are two 10G SFP+ ports, you can use the other one for WAN, but you can also get a switch with enough 10G ports to just pass the WAN over a VLAN.

 

[abq]

I use them all over the place. They are essentially just normal SBC servers but without graphics. They have a USB port on the back that works as a serial console with normal UEFI access. It boots off of normal storage too, BSD and Linux all have drivers for everything as well, except the CPLD which doesn't mean much unless you want the power LED on the front to change color, and the LEDs on the ports to have different blinking patterns instead of default ones.

They also have USB ports which they will boot off of. A nice thing is that they are always-on by default, so as long (and as soon as) there is power, they are on. They also have a locking barrel style power connector so it doesn't unplug by accident, but also accept non-locking connectors so it works with pretty much anything.

A common deployment for companies that need a bit more than just a single subnet and NAT (but don't have the budget) is a DC management switch (i.e. Quanta LB9 style), a VEP, a 10G DAC between them and off you go. Since there are two 10G SFP+ ports, you can use the other one for WAN, but you can also get a switch with enough 10G ports to just pass the WAN over a VLAN.

@oneplane, thank you for the background on these & UEFI access suggestion. ...I'll try loading linux to start & test, then try loading opnsense for home firewall with a brocade switch.

 

[PigLover]

I think any C3000 series used VEP or SuperMicro 3xx series will do the trick here. By default those SoCs have plenty of I/O on board with 30Gbps fdx traffic in mind. Unless you enjoy pfSense I'd not use Netgate for this.

I'm not sure the C3000 is going to do it for you.

It appears that you are not having trouble with N6005's raw IO - the issue comes up with processing horsepower for your IDS.

The C3558 is MUCH slower than the N6005 both in single-core and multi-core bandwidth. Getting a "better" C3xxx gets you more cores but unless your traffic is really well balanced you are not going to spread traffic over them well enough to get much benefit.

I had a C3958 based system until about a year ago and throughput collapsed quickly on a 1gbe connection even with just a few rules in Suricata (ok - a few hundred rules - but not a real aggressive IDS).

Honestly the N6005 (R86s) I am using now handles modest IDS much better, though as you pointed out not well enough to keep up with a Comcast 1.25gb connection. It handles the bandwidth just fine. In fact, without IDS (or IPsec firewall) it would likely handle your 2.5gbps connection.

But to get the IDS you are going to have to get something with a bit more CPU umph. Even Netgate's Xeon appliances, which are still on D-15xx platforms, will probably struggle.

 

[oneplane]

If the ruleset is indeed very core-bound and more cores doesn't parallelise as much as we'd like to, you'll be looking at a 10th gen desktop class CPU or the a Xeon D from Hewitt Lake or newer. Trouble is mostly that everything from the C3000 era that is more powerful than a C3000 needs quite a bit of cooling and power, so that pushes you do more current gen stuff. Budgets become much more important at that point ;-)

 

[PigLover]

You could probably do it with one of the newer Lenovo TMM boxes (M90q) with an i5-10500 or i5-11500 and add in a X550-T2 10g-baseT card. I'd go with the X550 vs X540 in that form factor just due to heat dissipation. That should have the clocks you need for the IDS and still not break the bank. If you are concerned about heat not too hung up on form factor you could also get the same generation in the "mini" form factor (M90s or even M80s). Then you could save about $100 and go with the more power hungry X540 - just watch out for fakes when you pick one.

Thinking a bit more - the AMD Ryzen mini-PC with the 5825U CPU that Patrick reviewed might do the job if 2.5g links are all you need.