New Router Suggestions (Multi-Gig/10 GbE)

[mjturner]

Following this thread with some interest. I'm currently using an OEM Dell R220 (Xeon E3-1220v3) with 4x1GigE interfaces (2xIntel 82571EB PCIe, 2xBroadcom BCM5720 onboard) running OpenBSD as my firewall/router and am expecting the installation of 1Gbps fibre in the next few weeks (initially 1Gbps down/110 up, moving to symmetric once a better provider is available later in the year).

Inter-VLAN routing performance of the R220 has been fine (I can pretty much get wire speed), but I do wonder if I'm going to hit some limits shortly if there are multiple 1Gbps streams as OpenBSD pf is still not completely multi-threaded. First step would be to move some of the inter-VLAN routing over to my switches (a pair HP V1910s), which should help, and then I plan on bringing my Brocade ICX6610 online and finally moving some of my systems to 10GbE.

 

[adman_c]

Following this thread with some interest. I'm currently using an OEM Dell R220 (Xeon E3-1220v3) with 4x1GigE interfaces (2xIntel 82571EB PCIe, 2xBroadcom BCM5720 onboard) running OpenBSD as my firewall/router and am expecting the installation of 1Gbps fibre in the next few weeks (initially 1Gbps down/110 up, moving to symmetric once a better provider is available later in the year).

Inter-VLAN routing performance of the R220 has been fine (I can pretty much get wire speed), but I do wonder if I'm going to hit some limits shortly if there are multiple 1Gbps streams as OpenBSD pf is still not completely multi-threaded. First step would be to move some of the inter-VLAN routing over to my switches (a pair HP V1910s), which should help, and then I plan on bringing my Brocade ICX6610 online and finally moving some of my systems to 10GbE.

Based on my experience with my pfsense firewall on a i3-8100T (which has incredibly similar single- and multi-threaded passmarks to your CPU), you'll hit CPU limits before you get to 10gbit linespeed, but it's well above 1gbit. Doing a simple iperf3 across vlans I was able to see a bit less than 7gbit throughput with my firewall.* So you have quite a bit of headroom before you run into a performance ceiling with your current hardware. However, since you already have the hardware, it's worth moving some/all of your L3 routing to the 6610 so all of that stuff can move at linespeed regardless of what your firewall is running.

*This was running pfsense virtualized on proxmox, with the 10gbit NIC passed through. So probably a bit higher if you're running bare metal.

 

[mjturner]

Based on my experience with my pfsense firewall on a i3-8100T (which has incredibly similar single- and multi-threaded passmarks to your CPU), you'll hit CPU limits before you get to 10gbit linespeed, but it's well above 1gbit. Doing a simple iperf3 across vlans I was able to see a bit less than 7gbit throughput with my firewall. So you have quite a bit of headroom before you run into a performance ceiling with your current hardware. However, since you already have the hardware, it's worth moving some/all of your L3 routing to the 6610 so all of that stuff can move at linespeed regardless of what your firewall is running.

Thanks for the feedback. I think pfSense can probably squeeze a bit more out of the hardware than OpenBSD as FreeBSD's pf implementation is multi-threaded (which is why it's still stuck at OpenBSD 4.5-style syntax due to the divergence), but I'm hoping the E3-1220v3 will still be good for a few gigabits of traffic once my WAN connection gets that fast. Moving to the switch for L3 routing is definitely the thing to do, I just got a bit burnt last time I tried - I managed to mess things up terribly which I why I stuck with the firewall doing everything.

 

[adman_c]

Thanks for the feedback. I think pfSense can probably squeeze a bit more out of the hardware than OpenBSD as FreeBSD's pf implementation is multi-threaded (which is why it's still stuck at OpenBSD 4.5-style syntax due to the divergence), but I'm hoping the E3-1220v3 will still be good for a few gigabits of traffic once my WAN connection gets that fast. Moving to the switch for L3 routing is definitely the thing to do, I just got a bit burnt last time I tried - I managed to mess things up terribly which I why I stuck with the firewall doing everything.

FWIW, I've adopted a hybrid approach to my inter-vlan routing. I have 3 "trusted" vlans handling my servers, infrastructure, and PCs--those I route via my Brocade switch with no ACLs--everything can reach everything. Linespeed routing FTW. But I'm also lazy and I didn't wish to re-create all of my firewall rules for my "untrusted" vlans--IOT, guest, etc. So I left routing (and access rules) for those on my firewall. Given that I don't route between trusted and untrusted all that frequently, and when I do I care less about full performance, I'm happy to take the performance hit for ease of maintenance. So really, other than making virtual-router interfaces on the switch and pointing default gateways for my trusted networks at the switch, there was very little additional config on the switch to get this working. Oh, and I guess I had to make a transit VLAN on the switch and then run all of my VLANs tagged from the switch to pfsense, but pfsense handled that no problemo.

 

[ReturnedSword]

My main stopping point for doing a TMM based router at the moment is concern with heat build up on copper 10 GbE quad NICs. Once I am able to upgrade my core switch to 10G I’d definitely go with a Lenovo Tiny and a 10G fiber NIC.

 

[adman_c]

My main stopping point for doing a TMM based router at the moment is concern with heat build up on copper 10 GbE quad NICs. Once I am able to upgrade my core switch to 10G I’d definitely go with a Lenovo Tiny and a 10G fiber NIC.

Honestly I'd be a bit concerned with the heat build up for a dual copper 10 Gbase-T NIC, let alone a quad. My little SuperMicro dual SFP+ runs at 3-5w and gets plenty toasty in that small space. Hey, at least the M90q and P340 have ventilation in the top cover!

 

[ReturnedSword]

Honestly I'd be a bit concerned with the heat build up for a dual copper 10 Gbase-T NIC, let alone a quad. My little SuperMicro dual SFP+ runs at 3-5w and gets plenty toasty in that small space. Hey, at least the M90q and P340 have ventilation in the top cover!

On my P330 which shares chassis with your P340, and the chassis does have vent holes on the cover for the Quadro P400/P620 cards (though interestingly mine have P1000’s which use a heat pipe going to the main HSF channel). I’ve tried trying to shoehorn a slim fan inside the chassis but it doesn’t have enough clearance. It would be great if there is some way to make the chassis slightly taller, let’s say 20-30mm to accommodate 15mm/25mm fans.

Aren’t you worried about your NIC dying due to getting too hot? I had the impression that dual SFP+ cards were fine … but your report that it gets toasty scares me a bit hah!

 

[adman_c]

On my P330 which shares chassis with your P340, and the chassis does have vent holes on the cover for the Quadro P400/P620 cards (though interestingly mine have P1000’s which use a heat pipe going to the main HSF channel). I’ve tried trying to shoehorn a slim fan inside the chassis but it doesn’t have enough clearance. It would be great if there is some way to make the chassis slightly taller, let’s say 20-30mm to accommodate 15mm/25mm fans.

Aren’t you worried about your NIC dying due to getting too hot? I had the impression that dual SFP+ cards were fine … but your report that it gets toasty scares me a bit hah!

Well, given that my NIC has no way of giving me a temp readout, I’m not worried about what I don’t know! When I looked it up, the Tmax for the Intel chip in the card is something like 120C, so I’m just YOLOing it for now. Oh, and I have the M720q, which lacks any ventilation over the PCIE/SATA area. Given how small the heatsink on my NIC is, I’m guessing even the little bit of ventilation from the top cover holes in the P330/340 would be plenty for a dual SFP+ card. They basically sip power.

 

[ReturnedSword]

Well, given that my NIC has no way of giving me a temp readout, I’m not worried about what I don’t know! When I looked it up, the Tmax for the Intel chip in the card is something like 120C, so I’m just YOLOing it for now. Oh, and I have the M720q, which lacks any ventilation over the PCIE/SATA area. Given how small the heatsink on my NIC is, I’m guessing even the little bit of ventilation from the top cover holes in the P330/340 would be plenty for a dual SFP+ card. They basically sip power.

It seems like a waste of a P330/P340, or even M920q to use as a soft router, unless it was had for a great deal. However I’m almost certain the top vented panel from the P330 and up can’t be used in the Mx20q. I could be wrong though!

I had a thought of cutting out my own 40/60mm fan hole in the panel. But alas.. that would make it ugly I guess I would have to re-weigh my options. A part of me says “just cut the hole and be done with it.”

 

[adman_c]

It seems like a waste of a P330/P340, or even M920q to use as a soft router, unless it was had for a great deal. However I’m almost certain the top vented panel from the P330 and up can’t be used in the Mx20q. I could be wrong though!

I had a thought of cutting out my own 40/60mm fan hole in the panel. But alas.. that would make it ugly I guess I would have to re-weigh my options. A part of me says “just cut the hole and be done with it.”

Yeah, I wouldn't get a P340/M90q for a router. My 8th gen i3/i5 M720qs were less than $400 all in and have plenty of juice, even running the firewall virtually. A p340/M90q with a 10th/11th gen i7 is appealing to me to replace my primary proxmox host though. Boot from a SATA DOM, have mirrored nvmes for VM storage, and 10gbe for replication/ha. Do I have a reason for this? No, but it sure seems nifty.

 

[ReturnedSword]

Yeah, I wouldn't get a P340/M90q for a router. My 8th gen i3/i5 M720qs were less than $400 all in and have plenty of juice, even running the firewall virtually. A p340/M90q with a 10th/11th gen i7 is appealing to me to replace my primary proxmox host though. Boot from a SATA DOM, have mirrored nvmes for VM storage, and 10gbe for replication/ha. Do I have a reason for this? No, but it sure seems nifty.

I had tried to theorize various ways to utilize the SATA port or M.2 E Key 2230 slot in the P3x0 whilst still leaving the PCIe slot free, but couldn't think of anything that could fit. Including, SATA to mSATA, E Key to M Key x1 flexible riser adapter. It's pretty tight in there once the lid is closed. Tbh I'm surprised that until this day no one had thought to manufacture a E Key x1 2230 NVMe, or made a simple "stacked" adapter that raises the height of the installed adapter/drive rather than going on a ribbon.

 

[adman_c]

I had tried to theorize various ways to utilize the SATA port or M.2 E Key 2230 slot in the P3x0 whilst still leaving the PCIe slot free, but couldn't think of anything that could fit. Including, SATA to mSATA, E Key to M Key x1 flexible riser adapter. It's pretty tight in there once the lid is closed. Tbh I'm surprised that until this day no one had thought to manufacture a E Key x1 2230 NVMe, or made a simple "stacked" adapter that raises the height of the installed adapter/drive rather than going on a ribbon.

Looks like this person got a SATA DOM plus 2x 10gbe to fit in their m90q.

 

[ReturnedSword]

Looks like this person got a SATA DOM plus 2x 10gbe to fit in their m90q.

That’s a pretty tight fit! I’ll have to try it out and see if a SATA DOM + adapter would interfere with my P series units’ P1000 GPUs. I had also considered removing the chassis from a 2.5” SSD and going “naked” drive, but concerned about electrical shorts to the exposed PCB just dangling around.

Well, since my Topton adventure doesn’t seem to be working out, I have been looking for some Chinese Velkase or Lone Industries clones. It looks like I can build an i3 12th gen multi-gig router for about $700, or under $1,000 for an i5. This would blow out any Jasper Lake system. My only problem is trying to find a suitable chassis, with full size/slim 80-92mm fan slots, and preferably a DC-ATX PSU with external power brick. It seems Flex ATX chassis are more prevalent, but if I go that route I’d have to deal with annoying tiny fans in the PSU.

 

[Parallax]

That’s a pretty tight fit! I’ll have to try it out and see if a SATA DOM + adapter would interfere with my P series units’ P1000 GPUs. I had also considered removing the chassis from a 2.5” SSD and going “naked” drive, but concerned about electrical shorts to the exposed PCB just dangling around.

I posted this in the P350 Tiny deal thread but it's relevant here :

A simpler option might be what eBay sellers seem to call a "half slim SSD" which should plug directly into the SATA connector without needing an adaptor, and it has no case so it's pretty small, the same or less than what you show here. SATA to SATA DOM adapters are impossible to find at the moment in the UK and the SATA DOM units themselves generally are fairly expensive here.

If you're looking at ESXi 7 then it requires 32GB minimum for an SSD and will consume up to 138GB for a boot device, so 64 or 128GB looks optimal given ESXi seems to appreciate some scratch and swap space. A 128GB half slim is £24 (~$30) delivered here at the moment.

 

[Parallax]

As far as the risks of a naked drive are concerned, you could duct tape the underside since there don't seem to be any components there.

 

[Markess]

It looks like I can build an i3 12th gen multi-gig router for about $700, or under $1,000 for an i5. This would blow out any Jasper Lake system. My only problem is trying to find a suitable chassis, with full size/slim 80-92mm fan slots, and preferably a DC-ATX PSU with external power brick. It seems Flex ATX chassis are more prevalent, but if I go that route I’d have to deal with annoying tiny fans in the PSU.

@ReturnedSword , as you know from all the help you've given me lately, I know less than the square root of 0 about firewalls/routers. But I do know a little bit about hardware in general ...

In case you haven't used any 12th Gen yet, 12th Gen i3 compares pretty well with Gen 10/11 i5s. Not as powerful as an i5-11600k that's been tweaked, but still probably enough for a router? Prices have dropped lately too. i3-12100F (do you need a GPU after the initial setup?) is $107 today after promo code at Newegg. i3-12100 is $125 (both U.S. pricing). And unlike the Z690 boards that came out at launch, B660 and H610 are much more affordable. So $700 is easily in reach.

For a chassis, have you considered one that uses an SFX PSU? SFX have larger (mostly 92mm) fans, so not as buzzy when they're spinning. Depending on the model, one with a partial-Zero RPM fan curve might not spin up at all. As a smaller form factor, there's SFX chassis out there that should be be around the desired 5L size.

One of my novelty builds had a Corsair 450w SFX PSU powering a Supermicro X9DRT-F, an E5-2640v2, and 64GB of DDR3 RDIMM. Probably a safe bet that it drew almost as much power at idle as a 12th Gen i3 + NIC(s) running pretty hard? The power supply rarely spun its fan up, and when it did it was very quiet. A 600w, or similar, partial-zero RPM SFX PSU while less efficient at the lower end of power curve, will be able to ramp up even higher without the fan coming on.

One concern I'd mention...My desktop has an i5-12600K and its been pretty well behaved on Linux with kernel 5.13 (after a couple updates). I understand kernel optimizations are ongoing to fix remaining bugs that I, luckily, haven't really encountered yet. 5.15 (out in the wild in Ubuntu 22.04) is better still, but I'm reading that it still has some rough edges. I'm not sure how fast things are going on the BSD side though, if you plan to use pfSense or similar? Again, I know ZERO about routers, but you probably need/want to have a router with no rough edges at all??

Cheers!

 

[glow]

That’s a pretty tight fit! I’ll have to try it out and see if a SATA DOM + adapter would interfere with my P series units’ P1000 GPUs. I had also considered removing the chassis from a 2.5” SSD and going “naked” drive, but concerned about electrical shorts to the exposed PCB just dangling around.

Well, since my Topton adventure doesn’t seem to be working out, I have been looking for some Chinese Velkase or Lone Industries clones. It looks like I can build an i3 12th gen multi-gig router for about $700, or under $1,000 for an i5. This would blow out any Jasper Lake system. My only problem is trying to find a suitable chassis, with full size/slim 80-92mm fan slots, and preferably a DC-ATX PSU with external power brick. It seems Flex ATX chassis are more prevalent, but if I go that route I’d have to deal with annoying tiny fans in the PSU.

I have been looking at the Lazer3D HT5 case for a pair of very similarly-goaled builds. Just cannot really stomach the >$100 price for a plastic case at the moment.

 

[ReturnedSword]

I posted this in the P350 Tiny deal thread but it's relevant here :

A simpler option might be what eBay sellers seem to call a "half slim SSD" which should plug directly into the SATA connector without needing an adaptor, and it has no case so it's pretty small, the same or less than what you show here. SATA to SATA DOM adapters are impossible to find at the moment in the UK and the SATA DOM units themselves generally are fairly expensive here.

If you're looking at ESXi 7 then it requires 32GB minimum for an SSD and will consume up to 138GB for a boot device, so 64 or 128GB looks optimal given ESXi seems to appreciate some scratch and swap space. A 128GB half slim is £24 (~$30) delivered here at the moment.

Great tip! Thank you @Parallax because I hadn’t even thought of that. Leave it up to me and I’ll think of the most complicated way to accomplish something simple I’ll have to have a poke around and see what reputable brands are out there. These half slim SSDs look strangely like recent gen SATA SSDs that have their 2.5” enclosures removed.

As far as the risks of a naked drive are concerned, you could duct tape the underside since there don't seem to be any components there.

I prefer not to use duct tape if possible as it gunks up whatever it’s applied on… I suppose I could try kapton tape first.

 

[ReturnedSword]

I have been looking at the Lazer3D HT5 case for a pair of very similarly-goaled builds. Just cannot really stomach the >$100 price for a plastic case at the moment.

I also don’t like the high price of the acrylic or 3D printed chassis, but I suppose whoever makes them have to be paid. Lazer3D is reputable though and it seems plenty of people like them for SFF builds.

I was thinking more along the lines of Velkase 3/5 or Lone L5 (this would be perfect!) but unfortunately both are out of production asa year or two back. The Lone L5 would be the best as it has a straightforward layout. HHHL card directly in the x16 slot, and since it was made for ITX sized dual slot GPUs, a simple x4 M.2 riser can enable usage of the second HHHL slot. Then the 2 x 80mm slim fan slots up front provide great cooling as well.

I’ve been looking around AliExpress and Taobao off and on for a suitable clone. Sadly one of the reasons both case lines went out of production is that with the RTX 3000 gen ITX sized GPUs are no longer really viable, so the SFF community has moved on to larger folded sandwich chassis…

 

[ReturnedSword]

@ReturnedSword , as you know from all the help you've given me lately, I know less than the square root of 0 about firewalls/routers. But I do know a little bit about hardware in general ...

In case you haven't used any 12th Gen yet, 12th Gen i3 compares pretty well with Gen 10/11 i5s. Not as powerful as an i5-11600k that's been tweaked, but still probably enough for a router? Prices have dropped lately too. i3-12100F (do you need a GPU after the initial setup?) is $107 today after promo code at Newegg. i3-12100 is $125 (both U.S. pricing). And unlike the Z690 boards that came out at launch, B660 and H610 are much more affordable. So $700 is easily in reach.

For a chassis, have you considered one that uses an SFX PSU? SFX have larger (mostly 92mm) fans, so not as buzzy when they're spinning. Depending on the model, one with a partial-Zero RPM fan curve might not spin up at all. As a smaller form factor, there's SFX chassis out there that should be be around the desired 5L size.

One of my novelty builds had a Corsair 450w SFX PSU powering a Supermicro X9DRT-F, an E5-2640v2, and 64GB of DDR3 RDIMM. Probably a safe bet that it drew almost as much power at idle as a 12th Gen i3 + NIC(s) running pretty hard? The power supply rarely spun its fan up, and when it did it was very quiet. A 600w, or similar, partial-zero RPM SFX PSU while less efficient at the lower end of power curve, will be able to ramp up even higher without the fan coming on.

One concern I'd mention...My desktop has an i5-12600K and its been pretty well behaved on Linux with kernel 5.13 (after a couple updates). I understand kernel optimizations are ongoing to fix remaining bugs that I, luckily, haven't really encountered yet. 5.15 (out in the wild in Ubuntu 22.04) is better still, but I'm reading that it still has some rough edges. I'm not sure how fast things are going on the BSD side though, if you plan to use pfSense or similar? Again, I know ZERO about routers, but you probably need/want to have a router with no rough edges at all??

Cheers!

All my recent stuff is AMD, so the idea would be my first foray back with a new Intel system since 2017.

Even the cut down dual core 12th gen Celeron/Pentium are probably more than overkill for a router. An i3-12100 doubles the cores and bumps up the cache. The non-F would be useful if I ever repurpose the CPU, and sometimes I like to connect directly to the console as my router sits next to all my gear.

In terms of chipsets, as long as the motherboard has 2 x4 M.2 slots it’s good enough. The other important part would be quality VRMs (for 65W or less CPUs, nothing overkill needed here). This is so in a low airflow system the VRMs don’t cook.

SFX (non-L) greatly increases the size of the chassis to a minimum of 7-ish liters, with 9-12L being more of the norm. I’d love to get a custom router with two HHHL slots under 3L but that’s a stretch. 4-5L is more reasonable, with or without a Flex PSU. DC-ATX is much preferred as it leaves room for some slim fans, or leftover volume to use a bigger CPU HSF.

With Linux AFAIK Alder Lake with heterogeneous cores would be finally ironed out with kernel 5.17… but that hasn’t made it into Ubuntu yet. Not that that would help that much because I’ll be using FreeBSD hehe. I’d really prefer a homogenous CPU core, so the “small” Alder Lake die is perfect.

Actually, I would love more than anything for Intel to hurry up with Alder Lake-N, which would be homogenous all E-cores… but as usual Intel probably won’t have any available to buy until a year from now. Jasper Lake was announced in January 2021, and Tremont which it was based on released even earlier… but were not available in mass until just a month ago.