Trying to plot out my next network upgrade

[TeeJayHoward]

I recently consolidated from a networking rack and a compute rack into back-of-rack networking. I was blowing breakers too often, heating up the entire basement, and the electric bill was starting to frustrate the wife. So I consolidated several U of switches into one, and lost most of my redundancy and a good chunk of my speed. Well, it's a home network. No biggie, right? I can always shut down a few servers and spin up the "lab" when I want to tinker or learn... Right?

Yeah... No. I'm missing my fiber switch something awful. I've got 10 servers (8 compute, 1 storage, 1 controller) which were, prior to this "upgrade", running over dual 10GbE fiber links that're now stuck at bonded 1GbE copper. And even in the glorious times before the 10GbE links, my ancient 40G Infiniband could occasionally restrict things. I've been going backwards for WAY too long. I've now got just a single Catalyst 3850 with 2x 10GbE uplinks (one to my workstation, one to my NAS) sitting reverse/Top-of-Rack. I'd really like to upgrade this, but I'm not sure what I'm looking for actually exists. I could swap in a dual 40GbE module, but that wouldn't help where I really need it - the compute nodes. I could break that out and get those servers running at 10GbE, but even then I'm pushing up against the edge of several-generation-old hardware and downgrading my storage/NAS link. I need an upgrade. Maybe even an entire network overhaul. So I'm looking for a single 1U switch with...

12+ 10GBase-T w/ PoE at a bare minimum for the house wiring. The more, the better. Probably half my house has PoE devices running @ 1GbE, and I'd really like an excuse to swap out the Cat6 for Cat8. IIRC I've got 16 copper lines running from the house to the switch now, but only 12 are really in use. The damned PoE is the biggest problem. Anything with SFPs means you can't run PoE, and I'm heavy into that ecosytem.

As for high speed links... I've got 8 servers I'd like to see running Inifiniband or at minimum some sort of low-latency >10GbE link - they can be short-cable passive copper or optical connections. Think HPC/clustering here, with a focus on latency over bandwidth. I've also got 1 workstation about 50' away that I'd like to run at the same speed as the servers, but latency isn't as important - although now that I think about it, it might be fun to play with NVMeoF. The NAS and control nodes can probably run over whatever link that is too. Come to think of it, maybe they'd be fine over the 10GBase-T connections, even. I don't yet own the NICs or transceivers for anything over 40G, and fully expect to need to upgrade my current OM4 to something new. (Is it MTP12 time?)

So I'm trying to find a semi-modern 1U Infiniband switch with 10GBase-T PoE. That kinda doesn't exist from what I've seen. Then throwing in the standard home lab requirements - not sounding like a jet engine during takeoff, not dumping 10A @ 120V worth of heat, etc... I'm guessing I'm SOL.

Should I be looking into upgrading the compute nodes as well so I can run two switches, or did my Google-Fu fail me and the perfect PoE + IB switch is out there... Somewhere?

 

[BlueFox]

You want Ethernet and InfiniBand on the same switch? Yeah, that's not happening.

 

[TeeJayHoward]

You want Ethernet and InfiniBand on the same switch? Yeah, that's not happening.

That's about what I expected. But it doesn't HAVE to be InfiniBand. I just really like the low latencies IB provides and it'd be the dream. Even if we pull IB out of the equation, I can't find a 12-16 port 10GBase-T PoE (ideally PoE++ for upcoming WiFi7) + 8-16 port QSFP28/QSFP-DD/OSFP/etc switch. It seems like once you get above 10GbE, it's an either/or situation. You either get RJ45, or you get transceivers. If you're really lucky, you'll have a quad-port "uplink". That's about it.

edit: Crud, I forgot about the out-of-band management ports. That's another 10x 1GBase-T. So I guess I'd be looking for a 24-port 10GBaseT w/ PoE++ and 8+ transceivers. (Of whatever config is new enough to have some decent offload/low latency.)

The more I think about it, the more I'm coming around to the idea that I'm gonna have to sacrifice something. I may need to pull one of my JBODs and upgrade the storage node to a smaller unit. Maybe flash in the rack is finally gonna happen...

 

[Blinky 42]

Do you have space for 2x 1U switches one for high speed, and the other for POE / your current 1G switch?
Need any features in the switch beyond speed? VLANS / Routing / Multicast etc?

You could go with 25Gb to keep it faster than just 10G w/o a huge cost outlay. There are a few 1/2 width options as well - get one for the POE and one for the higher speed networking. Pretty low practical demand for the vendors to make POE + high speed network as the volume isn't there to justify a SKU like that.

 

[TeeJayHoward]

Do you have space for 2x 1U switches one for high speed, and the other for POE / your current 1G switch?
Need any features in the switch beyond speed? VLANS / Routing / Multicast etc?

You could go with 25Gb to keep it faster than just 10G w/o a huge cost outlay. There are a few 1/2 width options as well - get one for the POE and one for the higher speed networking. Pretty low practical demand for the vendors to make POE + high speed network as the volume isn't there to justify a SKU like that.

Thinking about the half-width randomly brought to mind another issue with Infiniband. I'd need an IB->Ethernet gateway. Which, of course, means more rack units used and more Base-T ports. Damnit, there's a reason I had my network separated out into a different rack!

But yeah, 2x half-width would be fine as long as they're fairly low power. I technically have negative 2 rack units right now. I could justify moving to a larger rack and not care about how many U are used if I could get the power consumption of two switches to be smaller than the current 1-switch setup. My house HVAC and my server room are on the same circuit and I've barely got it under the limit now. I'm even at the point where I'm counting draw from unused PCIe cards. I've been told in no uncertain terms that running a new line down to the server room is not allowed.

...Which, again, brings up a new problem. If I go PoE++ for future APs, I'm gonna pop that breaker anyway. Dad gummit! Maybe it's time to split the rack and spread it around the house. 4U in bedroom 1, 4U in bedroom 2... You don't think the wife would notice if there's a server hidden behind the treadmill, do you?

 

[kapone]

Two switches...one...something like a Mellanox SX6036...will do 56gb ETH/IB/RDMA (with the right Mellanox cables of course)...two...anything with enough POE/POE+/POE++/xxx ports with at least dual 10g uplinks.

I mean, the Mellanox gives you 36x 40/56gbe ports or with breakout cables 64x 10gbe ports or...some combination between these two and idles at 35w...with software control of fans. At their lowest setting, if you're more than a few feet away, it's not gonna be an issue.

Problem solved?

 

[BackupProphet]

You want Ethernet and InfiniBand on the same switch? Yeah, that's not happening.

Mellanox SX6036 does that. And it works really great too.

 

[BlueFox]

Mellanox SX6036 does that. And it works really great too.

It is a bit of a unicorn in that regard, though last I checked, anything with copper (and PoE for that matter) like the OP wants just doesn't exist.

Really need 2 devices, not one, with that feature wish list.

 

[Greg_E]

Might be an option for your POE++ needs, look for Extreme Networks stuff on ebay. I got a new in box 5420m 48 port (gigabit) POE ++ with dual 900 watt supplies for $400, only has 4 SFP28 and two more "stacking" ports that you can use as 10g SFP+ (SFP-DD for stacking). And even with this, gigabit might be slowing you down with wifi7, not a subject I've really studied yet.

For the higher speed stuff, not sure what to recommend.

I have Extreme 5420m at work (with service contracts) so that was one factor in buying it. The other was I just got tired of dealing with Cisco, when you don't work with it often enough, you have to switch gears and go back... Just too much to handle, and I wanted higher power for POE. They are decent switches, I just wish that had a way to register your switch for lab use, and be able to get the firmware updates. Enterasys was way better in this regard, but Extreme bought them to gain some sort of feature and now it's gone.

 

[Tim111]

Brocade ICX6610

• 24/48 1gbE copper (PoE available)
• 16x 10gbE (8x SFP+ in the front, 8x via 2 QSFP+ breakout ports on the rear)
• 2x 40gbE (separate from the previously
• Mellanox sx6036 36 40/56 gb ethernet with license.
• This is what i run and works great.

 

[nabsltd]

12+ 10GBase-T w/ PoE at a bare minimum for the house wiring. The more, the better. Probably half my house has PoE devices running @ 1GbE, and I'd really like an excuse to swap out the Cat6 for Cat8. IIRC I've got 16 copper lines running from the house to the switch now, but only 12 are really in use. The damned PoE is the biggest problem. Anything with SFPs means you can't run PoE, and I'm heavy into that ecosytem.

You just won't get this in one switch at any decent price point. We have some $5000+ retail 24 port Aruba switches at work with Nbase-T PoE which would be perfect for you.

Just split up the "server" and "access" switches like everybody else does. Get a 12-24 port 2.5Gbit PoE switch for access and whatever 10-40Gbit switch you want for the server network.