New Router Suggestions (Multi-Gig/10 GbE)

[IamSpartacus]

The problem with the whole "Have your switch do most of the work" notion is that for high bandwidth L3 switching you typically need a large, power hungry (and thus loud) switch.

If someone can suggest a lower power L3 switch that can do 10Gb L3 switching I'll jump on it. Until then, I'll stick with my router on a stick.

 

[sko]

any cisco will do acl/policy-based routing at line-speed or very close to that.
My stack of 3750X does ACL- and policy-based routing between VLANs at 9.3-9.8GBps (iperf3).


edit: was a bit in a hurry earlier, so here's the full explanation:

That stack of 3750X handles some local routing for 10GBit gear (servers, main desktop). Only relatively basic rules from/to distinct hosts/IP ranges are allowed, everything else falls through to a "set next-hop" rule that forwards it to the OpenBSD router (which only has 1GBit links).
The 9.3-9.8GBps figures were tested between the desktop and storage server - only setting up the ACLs and route-maps at the switch and changed the default route on those systems to use the switch as the default gw. No special tuning (e.g. everything still runs on an MTU of 1500) or shedding load off the network.
Best that crappy mikrotik CSR305-1G4S+ (yes, I know, I was stupid...) could make in pure L2 forwarding was barely above 8GBps peak, average ~7.5GBps and dropping WAY more if utilizing all 4 ports.


I also acquired a juniper EX3300-48P a while ago and am still testing with policy-based routing (but can't get it to set a 'next-hop'; it just drops everything on the floor...) and it also did (practically) line-speed on the 10G links.
So yes, routing in ASIC is FAST and doesn't need a switch that sounds like a turbine. The 3750X stack is audible but bearable (esp. considering there are 3 PSUs and 4 additional fans running...) and the fans in the EX3300 can be easily swapped with something ~half the airflow, as the stock fans are running at lowest speed 90% of the time anyways and emit an annoying PWM-noise at such low speeds. I've used a pair of SUNON MF40201VX-G99 which are whisper silent and temps are well within the normal range even while testing with 8 APs @30W...

 

[IamSpartacus]

I run Cisco at work. I have ZERO interest in running Cisco at home. But you bring up another point. ACL's. Configuring ACL's on Cisco is a huge PITA compared to something like pfsense/OPNsense.

 

[IamSpartacus]

any cisco will do acl/policy-based routing at line-speed or very close to that.
My stack of 3750X does ACL- and policy-based routing between VLANs at 9.3-9.8GBps (iperf3).


edit: was a bit in a hurry earlier, so here's the full explanation:

That stack of 3750X handles some local routing for 10GBit gear (servers, main desktop). Only relatively basic rules from/to distinct hosts/IP ranges are allowed, everything else falls through to a "set next-hop" rule that forwards it to the OpenBSD router (which only has 1GBit links).
The 9.3-9.8GBps figures were tested between the desktop and storage server - only setting up the ACLs and route-maps at the switch and changed the default route on those systems to use the switch as the default gw. No special tuning (e.g. everything still runs on an MTU of 1500) or shedding load off the network.
Best that crappy mikrotik CSR305-1G4S+ (yes, I know, I was stupid...) could make in pure L2 forwarding was barely above 8GBps peak, average ~7.5GBps and dropping WAY more if utilizing all 4 ports.


I also acquired a juniper EX3300-48P a while ago and am still testing with policy-based routing (but can't get it to set a 'next-hop'; it just drops everything on the floor...) and it also did (practically) line-speed on the 10G links.
So yes, routing in ASIC is FAST and doesn't need a switch that sounds like a turbine. The 3750X stack is audible but bearable (esp. considering there are 3 PSUs and 4 additional fans running...) and the fans in the EX3300 can be easily swapped with something ~half the airflow, as the stock fans are running at lowest speed 90% of the time anyways and emit an annoying PWM-noise at such low speeds. I've used a pair of SUNON MF40201VX-G99 which are whisper silent and temps are well within the normal range even while testing with 8 APs @30W...

Oh so you're not actually doing L3 routing between VLANs on the Cisco. My Mikrotik CRS354 can do 40Gbps L2 forwarding without issue. But I've yet to find a power efficient switch that can do 10Gb+ L3 routing between VLANs that isn't power hungry.

 

[sko]

I run Cisco at work. I have ZERO interest in running Cisco at home. But you bring up another point. ACL's. Configuring ACL's on Cisco is a huge PITA compared to something like pfsense/OPNsense.

True, ACLs /w route-maps are a bit "special" to get your head around.

I've been using PF on OpenBSD and FreeBSD for years - and despite pf/OPNsense using it behind the scenes, I absolutely hate doing that in a GUI. It can't get any more impractical, complex and opaque from what originally is a very plain, simple and human-readalble syntax.

 

[sko]

Oh so you're not actually doing L3 routing between VLANs on the Cisco. My Mikrotik CRS354 can do 40Gbps L2 forwarding without issue. But I've yet to find a power efficient switch that can do 10Gb+ L3 routing between VLANs that isn't power hungry.

The switch is the default gw for those hosts and all rules are IP and/or port-based; although it's not stateful routing - so yes, it's not fully-fledged and thorough L3 routing, but I only wanted to handle the most part of the traffic between the 10G-connected hosts there.
Everything more complex or to the outside or 1G-part of the network can go through the OpenBSD router, where I can simply use PF... The network isn't that huge and busy that this would impose a considerable bottleneck. Most local traffic is kept inside the same VLANs/subnets anyways...

 

[IamSpartacus]

The switch is the default gw for those hosts and all rules are IP and/or port-based; although it's not stateful routing - so yes, it's not fully-fledged and thorough L3 routing, but I only wanted to handle the most part of the traffic between the 10G-connected hosts there.
Everything more complex or to the outside or 1G-part of the network can easily go through the OpenBSD router, where I can simply use PF...

So you're getting 10Gb speeds between hosts in different VLANs on that switch or are the hosts in the same VLAN?

 

[sko]

So you're getting 10Gb speeds between hosts in different VLANs on that switch or are the hosts in the same VLAN?

also between different VLANs
most 'heavy lifting' (e.g. NFS) is kept within the same VLAN, but I have services running e.g. in a DMZ that's also accessible from the outside, but ofc I want to access them at 10G if traffic is local...

 

[ReturnedSword]

@sko Wow that sounds like an impressive setup. Going forward I probably will still be on mostly L2 and fairly flat, so my needs aren’t that high. I do plan to do some light VLAN spanning, but nothing major. I believe IIRC, Mikrotik can handle a bit of L3.

 

[oneplane]

I'm mostly doing just L2 stuff on the switch, maybe some 802.1x, and then have a fat trunk to an OpnSense box to deal with all the firewall and routing. Unless you're doing something special at ome, 2x10Gbe Twinax will do just fine. Hell, unless you're going 5Gbps bi-directional between VLANs, 1x10Gbe will work just fine too.

 

[spikeb]

Here are some hardware scenarios I've come up with:

2.5 Gbps Only

$408 - Topton N6005 Quad i225V appliance [link]:

• $277 - Topton barebones
• - Intel N6005 2.0GHz/3.3GHz; 4513 PassMark
• $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
• $50 - 500GB Western Digital Blue SN570
• - Quad i225V (integrated)

Looks pretty awesome. Seems like it's going to be years before home internet connections (at least in the USA) will be commonly above 2.5 Gbit.

My plan is one of these, I'm waiting on servethehome to post their review, they had a two similar models reviewed, but without the Intel N6005. Apparently they ordered the N6005 unit and are waiting for it. I believe it was mentioned that it's using the desirable rev3 version of the Intel Ethernet chip.

My plan is VyOS, Linux+Shorewall, or maybe OpenSense on the topton. Planning on Comcast modem -> Topton -> switch with 10G (for NAS) and 2.5G for the rest (desktops, 2xUbiquiti wifi, and a few Reolink cameras). Not sure if I'll virtualize the firewall (as discussed in the ServerTheHome article), but I will likely run DNS (unbound), maybe a cache, possible a Wire Guard end point, and maybe even a minecraft server for kid and her friends.

I'll use VLANs on the firewall so I tag each port of the 10G+2.5G switch as needed to let me separate the trusted (like the desktops I admin) from random consumer equipment I don't trust. I definitely want to keep the switch and router configs in version control. Sure it's overkill on the router, but I suspect I'll find a use for it over it's useful life.

 

[ReturnedSword]

Looks pretty awesome. Seems like it's going to be years before home internet connections (at least in the USA) will be commonly above 2.5 Gbit.

My plan is one of these, I'm waiting on servethehome to post their review, they had a two similar models reviewed, but without the Intel N6005. Apparently they ordered the N6005 unit and are waiting for it. I believe it was mentioned that it's using the desirable rev3 version of the Intel Ethernet chip.

My plan is VyOS, Linux+Shorewall, or maybe OpenSense on the topton. Planning on Comcast modem -> Topton -> switch with 10G (for NAS) and 2.5G for the rest (desktops, 2xUbiquiti wifi, and a few Reolink cameras). Not sure if I'll virtualize the firewall (as discussed in the ServerTheHome article), but I will likely run DNS (unbound), maybe a cache, possible a Wire Guard end point, and maybe even a minecraft server for kid and her friends.

I'll use VLANs on the firewall so I tag each port of the 10G+2.5G switch as needed to let me separate the trusted (like the desktops I admin) from random consumer equipment I don't trust. I definitely want to keep the switch and router configs in version control. Sure it's overkill on the router, but I suspect I'll find a use for it over it's useful life.

In my area I can already get 5 Gbps fiber, but it’s pretty expensive. 1 Gbps/$80, 2 Gbps/$130, 5 Gbps/$180. I’m already paying $75/mo for crappy Charter Spectrum 400/20 Mbps, so just need to get moving on switching. Big block for me is that AT&T fiber does not officially allow real bridge mode, and on their newer combined ONT/router units, the unofficial method is not even possible. There’s a fake bridge mode, however the connection would still be constrained by the provided equipment’s small NAT table.

I would’ve went with some suggestions in this thread with having the switch manage VLANs, but my network infrastructure isn’t there yet. I’ll change to a new router with a multi-gig fiber NIC later, and maybe give the Topton box to my brother.

Looks like lead time has been increased for the N6005 variant again. Topton reported they have in-stock of the N5095 and N5105 though. If I were to rebuy I probably would go with a N5105 variant. I was blinded by “little more perf on N6005!” when I purchased.

 

[zer0sum]

In my area I can already get 5 Gbps fiber, but it’s pretty expensive. 1 Gbps/$80, 2 Gbps/$130, 5 Gbps/$180. I’m already paying $75/mo for crappy Charter Spectrum 400/20 Mbps, so just need to get moving on switching. Big block for me is that AT&T fiber does not officially allow real bridge mode, and on their newer combined ONT/router units, the unofficial method is not even possible. There’s a fake bridge mode, however the connection would still be constrained by the provided equipment’s small NAT table.

I would’ve went with some suggestions in this thread with having the switch manage VLANs, but my network infrastructure isn’t there yet. I’ll change to a new router with a multi-gig fiber NIC later, and maybe give the Topton box to my brother.

Looks like lead time has been increased for the N6005 variant again. Topton reported they have in-stock of the N5095 and N5105 though. If I were to rebuy I probably would go with a N5105 variant. I was blinded by “little more perf on N6005!” when I purchased.

I can't see a reason you would need full passthrough on the newer BGW320-500.
Your firewall has a public IP, and the upstream gateway is the AT&T CPE device. I get sub 1ms pings to the gateway

Depending on how many devices you have you may never get anywhere near the limit of 8192 sessions.
I only have about 40 devices on my internal network and I never use more than 10% of it

 

[ReturnedSword]

@zer0sum Does the BGW320-500/505 crap its NAT table when running open connection heavy situations (e.g. torrent seeding)?

 

[zer0sum]

@zer0sum Does the BGW320-500/505 crap its NAT table when running open connection heavy situations (e.g. torrent seeding)?

You can for sure blow it up with torrents, but I stopped using those a long time ago when I made the switch to Usenet

You can see the effect on the NAT table with a single torrent client with 5 linux iso's running

 

[ReturnedSword]

You can for sure blow it up with torrents, but I stopped using those a long time ago when I made the switch to Usenet

You can see the effect on the NAT table with a single torrent client with 5 linux iso's running
View attachment 22809

I have a couple RSS feeds active to seed new releases for various OSS ISOs, where older releases get de-activated eventually, so when the torrent client picks up a new build my concurrent connections goes up by quite a bit. Certainly more than 8,192. I’ve always been a big supporter of OSS, so it’s my little way of helping the community. It would be a shame if I couldn’t do that anymore

As for torrents for media, I don’t use torrents for that purpose, so switching to Usenet wouldn’t be as useful for me. All my hoarded media originated from physical copies I ripped myself. I still have boxes of movies I’ve collected over the years I have not ripped yet

 

[zer0sum]

I have a couple RSS feeds active to seed new releases for various OSS ISOs, where older releases get de-activated eventually, so when the torrent client picks up a new build my concurrent connections goes up by quite a bit. Certainly more than 8,192. I’ve always been a big supporter of OSS, so it’s my little way of helping the community. It would be a shame if I couldn’t do that anymore

As for torrents for media, I don’t use torrents for that purpose, so switching to Usenet wouldn’t be as useful for me. All my hoarded media originated from physical copies I ripped myself. I still have boxes of movies I’ve collected over the years I have not ripped yet

The only thing I've seen the NAT table struggle with is torrents, and if I had that requirement I'd be looking at an external vps to use as a seed box

 

[ReturnedSword]

Seed box pricing goes up fairly quickly though. Storage is also an issue as it seems seedboxes usually don’t have plans with a massive amount of storage. Well we will see, I’m still on crappy DOCSIS

 

[ReturnedSword]

I just got a notification from USPS that I’ll be getting my Topton N6005 box on Tuesday .

I originally ordered on April 7 for $277, so it will take a total of 55 (!) days to receive the product. Topton‘s excuse was that there are shortages of the N6005 SoC, and they didn’t get any supplies until May 15. This was surprising, since Jasper Lake was announced over a year ago IIRC. I wonder if Intel has some manufacturing issues or capacity constraints? My order shipped out exactly on May 15th. The N5095 and N5105 were plentiful in stock though beforehand. If I had to do this again I probably would’ve bought the N5105 variant.

Topton threw in a VESA mount (not that useful for a router/firewall), and a $20 refund for the trouble.

 

[ReturnedSword]

I had received the Topton unit on May 28th, however have not have had time to do any testing until a few days ago. I'll do a full write up, but I'm not getting a great feeling about the product.

Edit:

https://forums.servethehome.com/index.php?threads/topton-jasper-lake-quad-i225v-mini-pc-report.36699/