Switching suggestions for 10gb L3 core routing and PoE++ auxiliary

[Phlesh]

I'm looking for any suggestions on the best switch(es) to purchase for a new home setup.

In light of being able to get multi-Gb WAN into my home, I want to be able to do 10gb+ switching as the core of my network. At the same time, I'm going to have a plethora of PoE devices (cameras, AP's being the main ones, but plenty of others over time).

I'm having a hard time identifying switches that have 12+ 10gb RJ45 ports (or SFP+) and can do 10gb+ L3, and also a lot of PoE++ ports. My question here is whether I should be looking instead to get two distinct switches: one for PoE (probably 24 ports or more) and another for the 10gb L3 core (could be fewer ports... perhaps as few as a dozen for now).

Any suggestions on a combination of switches that make sense, and if possible, any single family that offers a stacking solution to present as a single switch? Or is there actually a single switch that will basically accomplish what I want (even if some minor compromises are required, such as fewer 10gb ports)?

In particular, understanding what brands and model families are most likely to be the best path will help, because the space of enterprise gear is so massive that it's hard to even start narrowing things down.

Thanks in advance for any advice!

 

[nickf1227]

In a production environment it's always best to separate your core/distribution layer from your access traffic. If you are talking about specifically for a home lab, you can obviously scale that back a bit and try to do everything all in one and no one will judge

With that said, how many POE++/802.3bt ports do you think you need? I personally haven't had the need for anything more than 802.3at, with most of my POE equipment still 802.3af. if you really don't need Poe++, just get an ICX 6610, it checks all of your other boxes. There's a whole thread about this here that @fohdeesha started.


Even still most enterprise products that are going to be bt power and have as many SFP+/10 gigabit ports you want are still supported and for sale...which means you'll be paying alot of money for what you are looking for.

My recommendation would be to invest in your core with something hefty like a VDX 6740 or a Nexus 5K or Nexus 3K. Deals for any of these switches can be had between 200-400 USD depending on the day on eBay.

Then you will have a shit ton of ports for anything in your lab and any high end machines you have... You can break the golden rule and put access traffic on your core because it's your house. Then you can get an access switch that has any amount of POE++ you need..maybe a small 8 port deal, and a different access switch for everything else in your house.

 

[Phlesh]

Thanks @nickf1227.

I've looked a little bit at the Brocade line (including that forum post you referenced). Given I want some PoE++, it seems like it isn't necessarily the best option for an all-in-one switch. Your recommendation to think of things in a two-switch setup (core vs. access) makes some sense to me and might be the way to go -- in which case your suggestion to go for a beefy non-PoE that's 10gb+ and combine it with whatever size PoE++ auxiliary switch I need makes sense. The VDX 6740 looks interesting for the core... do I need to worry about a license for that if I buy one off of Ebay? And then what reasonable options (if any) are there for the PoE++ switch?

I am doing a count of the PoE/PoE+/PoE++ devices that _will be_ in the network in the not-too-distant future. I'm making an assumption that 802.3bt will be a requirement for any WiFi 6E AP that I'd install (as soon as those are widely available). Everything else is probably 802.3at at most for the foreseeable future.

• 8 IP cameras (802.3af/at)
• Up to 12 WiFi 6E AP's (802.3bt assumed) -- in reality, probably end up with a handful of 6E in important locations and a few 5Ghz for broad coverage, but I'd like this overplanned if I can get a switch for all of it at once
• Up to 4 miscellaneous powered devices for convenience at wall jacks (802.3af/at for foreseeable future)

So we're looking at PoE+ for up to 12 devices, and PoE++ for up to 12 devices. This is a little bit overkill on the PoE++ -- could get away with probably 4 or 8 for the next 2-3 years or more. But I would love to not have to install a new switch in 5 years if I can help it.

Options seem to be:

• 24-port PoE++
• 12-port PoE+ and 12-port PoE++ (either single switch or two switches)
• 24-port PoE+, with understanding I will have to get a new switch when I want to deploy PoE++

Thoughts?

 

[nickf1227]

What aps are you planning on purchasing that you feel that you need 802.3bt?

Any APs drawing that much power are going to need 802.3bz multigig as well, and in fact APs that need only 802.3af will still benefit from 2.5 or 5 gig ethernet.
This AP will work 90% with 802.3af:
Aruba 510 Series Wi-Fi 6 (802.11ax) Indoor Access Points | Aruba (arubanetworks.com)

This AP only requires at, and even works at half capacity on af:
Cisco Catalyst 9115 Series Wi-Fi 6 Access Points Data Sheet - Cisco

This one requires at:
RUCKUS®R650: Indoor Wi-Fi 6 (802.11ax) 4x4:4 Wi-Fi Access Point with 2.5Gbps backhaul and 6 spatial streams (ruckuswireless.com)

This one requires at:
Access Point WiFi 6 Long-Range – Ubiquiti Inc.

The only bt requirement I can find is for the Cisco 9130, which is an 8x8 custom engineered AP. and it DOES work totally unhindered on at:
Cisco Catalyst 9130AX Series Access Points Data Sheet - Cisco

This is probably the highest-end AP in existence right now.
All I am saying is that you're assuming Wifi 6e will bring a requirement it probably won't bring, because even in the enterprise poe++ is hard to find.

EDIT: Aruba has released a Wifi6e AP:
630 Series Wi-Fi 6E Indoor Access Points | Aruba (arubanetworks.com)
Which works on both at and bt, with limited features being disabled on at. This is their highest-density and highest-performance option. If you down rev to the future release of the 610, you will only need at for full features. The 505 was full Wifi6 at only af...I don't see why there won't be a low-end AP 605 that is the same for 6e.


The short version of my answer is that if you are trying to future proof your setup, you are NOT looking for out-of-support used enterprise gear, you are looking for current production enterprise gear and you are going to pay substantially more for it, because even in current production enterprise gear these are not standard features, these are "upgraded" SKUs. Aruba makes a buch of multigig switches, they have SKUs for the older 2930m and 3810 that have multigig ports, but I don't think either of those have 8023bt ports on their normal SKUs. Cisco is a similar story, where you can get "UPOE" ports, but they are not the standard SKUs. That makes it difficult to find second hand, as that technology is not ubiquitous.

If you ask me multigig is more important than POE++, but even that depends on the density of your environment. I also feel that if your wireless infrastructure needs that much backhaul, you really should be hard-wiring more of your devices. That is doubly true if this is your house.

If you are looking for a recommendation, the best deal I can think of seeing was for a Brocade ICX 7150ZP, but they generally dont go below $1k on eBay.

As far as the VDX, there are some users here who can help with licensing. I personally do not recommend the copper version of that switch because it is loud and power hungry, the SFP+ version is great though. I still think the 6610 gets you where you need to be. There's something to be said about the 80-20 rule. You can get 80% of the way there on a shoe-string budget, but you're going to pay through the nose to get that extra 20%.

 

[RTM]

Generally speaking, I believe we are in between standards at this point.
2.5Gbe and 802.3bt have yet to break through and become widely and cheaply accessible when it comes to switches.
Sure you can find some, but they are usually very expensive.

I think we are getting there, Ubiquiti's USW-Enterprise-24-PoE for one is a step in the right direction (no 802.3bt support though).

 

[Phlesh]

Thank you for the additional thoughts.

I am hardwiring a bunch of things in the house. Basically every room has hardwires and smurf tubes for more wires. So, no issue there! Still, when WiFi 6E is generally available, I will probably want to install it rapidly -- hence the desire to go ahead and have a switch in place that will enable full-power 6E right away.

My research on 6E AP's was not very extensive, but one of the first ones I came across was the one you pointed out from Aruba. The Aruba AP was the one giving me some thought toward 803.bt being a soft requirement for full-powered 6E AP's -- and obviously that is not going to be universally true, as you're showing me. So, maybe I'm overdoing it on thinking about 802.3bt at this point and should leave that for "future me" to worry about with a secondary switch when the time is right.

I'm mentally skipping over 802.3bz (2.5 Gb / 5 Gb) because I'm assuming I go straight to 10 Gb with all the hardware that wants multi-gig. Maybe that's a bad assumption? Thinking of certain PC's, the future 6E AP's, and maybe some video-over-IP scenarios -- for all of those, I assumed I would be able to deploy 10 Gb-capable NIC's, AP's, and associated devices. And again, maybe that's an awful assumption. There are two immediate places I would deploy 10 Gb, which are a primary PC and a workstation laptop, and for both of those, I know I can get 10Gb interfaces.

So, again, you are maybe showing me the light here: ignore PoE++ for now. When the devices are there, get an access switch specifically for those devices, and ensure it supports the transfer rates (e.g., 802.3bz) that they require.


Let's move forward from that point. I want a core switch that's going to do 10 Gb L3. I want an access switch that does PoE+ Gb. I want an access switch that does 2.5Gb / 5 Gb if possible, and with PoE+ as a bonus if possible.

What would the minimum combination of switches be that provides both low-cost upgrade paths and lowest starting cost? Let's assume we start with something like the VDX 6740 as the core and work up from there. Is there another core switch that gets me the multi-Gb at lower total cost (and let's not forget RU's, power, heat...) than a secondary switch? And/or another core switch that gets me PoE+ at lower total cost than a secondary (or tertiary...) switch? As an example of a single switch that maybe gets me "all the things" right now, there's the ICX 7150-48ZP that you pointed out. It's pricey, but again, the TCO on multiple switches (power, heat, and upgrade cost when the secondary/tertiary may want to go to PoE++) needs to be considered here.

Or, alternatively, is there another lower cost secondary switch that gets me both 2.5 Gb/5 Gb and PoE+, and in combination with the core switch would have reasonable total power draw/heat output? This could be a smaller switch for sure -- 24 ports would do it.

 

[nickf1227]

Hey man,
Just trying to help you spend your money wisely, it's no issue.
You can architect this a dozen different ways, and really it is ultimately up to you.
I personally have a Brocade ICX switch as my L3 core, with some other smaller HP/ICX switches on different floors in my house. I did not have the luxury of having previously run telco lines to reuse or any conduit to feed wire through.


I used to have 2 HP E3800 stacked as my core, and needed more than the 8 ports of 10 gigabit ethernet available for my servers and workstations. I purchased the ICX I have now and a VDX 6740 on eBay for relatively inexpensive. The "data center" switch is separated from my home network above at L3. Using the magic of OSPF I have separate redundant routes/interfaces to the rest of my home network and to my two ISPs. So in effect, I have a modified collapsed core design, and neither in my home network nor my datacenter am I following the two or three tier architectures. But this is my house and I accept that.

Now, as far as TCO, spending more up front vs electricity and heat costs, let alone the WAF (wife acceptance factor), and how many touch points you will have to plan for that's all up to you really. Surely you can purchase something from Mikrotik or Unifi and get "newer" hardware that is more energy efficient per-port-per-gigabit but you are getting lower quality SMB hardware that may or may not have a higher AFR (annual failure rate) than used enterprise gear form Cisco, HP or even Brocade/Ruckus (who, despite all appearances, I would never purchase for a new production environment). If you are looking for the set it and forget it type of thing, and you want something to run forever, spend the money on enterprise. If you like to tinker and you will probably end up replacing it three times over the next five years, you can get the Mikrotik stuff.

I;ve brought up the ICX 6610 a few times now and I really think it's something worth considering. lets do some maths.

The VDX I had brought up will draw a maximum of 110 watts per it's spec sheet. I got mine for 200 dollars.


If we assume worst scenario at 100% load we are looking at:


The average price for electricity in the UNited states is 10.42 cents, we'll assume 11.


Which is about 106 dollars annually, or 9 dollars a month:


Then let's add in the unicorn switch, the 7150 ZP as your access switch:


With no POE devices for our calculation purposes, lets then add the 89 watts listed in the spec sheet.


Add another 86 dollars annually or 7 dollars a month:


So now we are at ~$192 annually or $16 a month in switches. What do you get with that? An enterprise-class access switch with all the bells and whistles you asked for and a data-center class super low latency core/aggregation switch with advanced L3 features like OSPF and other dynamic routing protocols. This follows a 2-tier design and is technically closer to what should be considered "best practice". This setup will cost all-in $1200 with the above mentioned $192 annual cost.
Add in a pfsense box like a SG-5100, a few APS and your all set.

The other option I have been trying to present is an ICX 6610. In terms of homelab it's really a class of it's own. You get 48 ports of 802.3at POE+, 16 ports of SFP+ 10 gigabit and advanced L3 features like OSPF and other dynamic routing protocols. You will be using a collapsed core design, but it's your house. This setup all-in is under $200 and has an anual cost equivalent to just the VDX at $106. This is in 1RU instead of 2RU. Sure, you lose multigig and poe++, but does it matter? Maybe it will in 5 years...

 

[Phlesh]

That's a great analysis, thanks. One point of confusion, though -- why couldn't I just go with the 7150ZP in a collapsed core design as well, foregoing the VDX 6740? Wouldn't that be the best apples-to-apples comparison to the collapsed core design with the ICX 6610? There are some pluses and minuses (fewer 10 Gb SFP on the 7150ZP, but get some 2.5 Gbps and PoE++). Still 1 RU, more comparable energy/heat budget. What am I missing there?

The ICX 6610 is certainly looking attractive, though I do like the idea of having a beefy core switch I can leave in place for a long time. It's a tough call!

 

[nickf1227]

That's all your call man, I don't see anything wrong with that option either if you want to spend that money up front.

 

[NateS]

I'm mentally skipping over 802.3bz (2.5 Gb / 5 Gb) because I'm assuming I go straight to 10 Gb with all the hardware that wants multi-gig. Maybe that's a bad assumption? Thinking of certain PC's, the future 6E AP's, and maybe some video-over-IP scenarios -- for all of those, I assumed I would be able to deploy 10 Gb-capable NIC's, AP's, and associated devices. And again, maybe that's an awful assumption. There are two immediate places I would deploy 10 Gb, which are a primary PC and a workstation laptop, and for both of those, I know I can get 10Gb interfaces.

Unfortunately, that is a bad assumption. The vast majority of 10g gear on the used market was designed well before 2.5/5g even existed, so they're not compatible. But you do have the right idea going straight to 10g, installing NICs as necessary, because we're in a weird place in the market where the faster technology is actually far cheaper at the moment. There's mountains of used 10g enterprise gear being sold, but the used market for 2.5/5g gear basically doesn't exist yet because it's too new.

The tricky bit is wifi 6 APs, as most of them will not have 10g capable ports or upgradable NICs. I'm seeing them mostly with 2.5g ports, since that's sufficient backhaul bandwidth, and also works over cat5e, so it's a drop-in replacement for many buildings.

Personally, when designing my network, I went with 6610s, since the price/performance really can't be beat, and I plan to add a small secondary 2.5g POE switch when I upgrade to wifi 6 (which isn't urgent for me).

And for POE, keep in mind that injectors are a thing too; if you need a bunch of ports of regular POE+ but only a few POE++, injectors on the specific ports may be a cheaper option than a full second switch.

 

[whiskytangofoxconn]

Don't skip over 2.5/5gbe. ISP's offering residential speeds in excess of gigabit will be a way to both charge more on a monthly basis, and a way for all of the device manufacturers to force people to buy new hardware (modems, routers, etc.). In my opinion, multi-gig is here to stay (for a little while at least).

PoE++ isn't a bad thing. Houses today are different than they were a few years ago. Siri and Alexa turn things on & off and low power LED lights allow people to use PoE switches and injectors for stuff like under-cabinet lighting or similar.

 

[Phlesh]

@nickf1227 -- I ended up going with the ICX 6610. Appreciate your help!

My next quest is for an appliance or appropriate hardware for running a router. Suggestions from anybody would be appreciated there too!

 

[Blue)(Fusion]

@nickf1227 -- I ended up going with the ICX 6610. Appreciate your help!

My next quest is for an appliance or appropriate hardware for running a router. Suggestions from anybody would be appreciated there too!

Depends what you are asking of the router.

Simple edge NAT/PAT router? Doesn't need much. Throw in Suricata and/or Sensei and you're looking at requiring more and more performant hardware. I'm presently connecting a humble 100/10Mbps ISP through a Check Point T-160 (4600) with OPNSense, OpenVPN server and client, Sensei, and Suricata. Suricata definitely starts to stretch the old dual-core Pentium E6500 thin. I also had to upgrade to 8GB of RAM.

 

[Phlesh]

@Blue)(Fusion -- thanks. What if I want to go dual-WAN, with at least one of those links being 10GbE, and third 10GbE port to the switch? Is there a server (preferably low-power) that gives me the ability to have dual PSU's and a PCI slot for a 4x SFP+ NIC?

 

[Blue)(Fusion]

@Blue)(Fusion -- thanks. What if I want to go dual-WAN, with at least one of those links being 10GbE, and third 10GbE port to the switch? Is there a server (preferably low-power) that gives me the ability to have dual PSU's and a PCI slot for a 4x SFP+ NIC?

It all depends on your budget. You can get a powerful homebuilt server with a newer, more efficient CPU, capacity for multiple PCI-E dual 10G NICs, and so on.

With L3 on your switch and only edge routing on pf/OPNSense with 3x 10G SFP+, get whatever will fit the 10G SFP+ NICs of your choosing. It won't require much performance for just NAT/PAT routing and a few simple things (e.g. DNS, DHCP, SNMP, squid). If you want Suricata, Sensei, and fq_codel traffic shaping on a multi-gigabit WAN, you might as well build your own high core-count system with lots of RAM and an SSD or 2.

Regarding virtualized vs physical pf/OPNSense, I ran both for years. Virtualized was nice to consolidate, but going physical had a noticable increase in performance and made it much easier for the "just turn it off and turn it on" fix for when I am not home and things go awry (they haven't yet*).

EDIT to add:
Here's a more powerful model of what I have running OPNSense on:

Checkpoint 4800 T180 8 Port Gigabit Firewall Appliance USED | eBay

Find many great new & used options and get the best deals for Checkpoint 4800 T180 8 Port Gigabit Firewall Appliance USED at the best online prices at eBay! Free shipping for many products!

www.ebay.com

It will not be able to do Suricata on a multigig WAN, but you can probably run Sensei with little issue. The above linked one appears to have the add-on 2x 10Gig SFP+ module, if you can settle for only two. It does have dual PSU.

 

[Phlesh]

I decided to grab a used R210 II with the E3-1220L proc, and will go at this from the bare metal angle for now. It's so low power draw that I'm going to be happy running just the firewall on it and nothing more for now. I think I'm going to start with OPNsense.

I believe all I need now is a PCI riser, the SFP+ quad NIC, and... nothing else I guess?

Thank you all for the help!