Does an 8+ slot PCIe 4x motherboard exist?

[Agrikk]

I recently came across an article about VyOS and now I am intrigued about building my own router/firewall, but the port count on my existing switches is rather high. I plan on populating a motherboard with dual-port 10g fiber HBAs and 4-port 1gig copper NICs.

I've seen plenty of mining motherboards with tons of PCIe 1x slots, but 1x will throttle the throughput on these NICs.

Has anyone come across motherboards with higher slot densities?

 

[BlueFox]

Supermicro X9DRX, X10DRX, or X11DPX-T come to mind.

 

[Agrikk]

Oh wow. Those are cool looking boards. A pity they are dual CPU, though. They will be a little overpowered for a simple switch.

 

[BlueFox]

There aren't enough PCIe lanes to go around on most platforms with a single CPU. Pretty much looking at dual CPU only and you would need to populate both.

You may also find that software switching performance can be lackluster in comparison to hardware with an ASIC.

 

[Agrikk]

Yeah, I expect as much. This whole thing reeks of a Bad Idea but I'm considering doing it anyways. I'm hoping that I can replace three devices (Sonicwall NSA 4500, Force10 10g switch, PowerConnect 6248P gig switch) and their resulting crazy-loud fans with something home built and quieter but still performant.

 

[kapone]

Much easier to take an existing switch (that has the ports you want) and water cool it. It's going to be a custom job, expensive (not that bad though ~$200-400), and will make it quiet without sacrificing network throughput.

 

[NablaSquaredG]

Would you be okay with a bit of Frankensteining?

Because in that case I'd it might be an idea to use two or three China-PCIe-Switches (e.g. 119.0US $ |ANU28PE16 NVMe SSD Riser SFF8643 zu SFF8639, 8 Port 12Gbs,(nicht mit kabel, nicht unterstützung LSI 8643*2 zu 8639*2 kabel)|Add-On Karten| - AliExpress) in case the board does not support bifurcation or just an PCIe->SFF8643 Expander (e.g. 26.0US $ |ANU04PE16 NVMe Controller SFF8643 SFF8639 4 port PCIe 3,0 X16 SSD Exp Riser (nicht mit kabel, nicht unterstützung LSI 8643*2 zu 8639*2)|Add-On Karten| - AliExpress) if the board does support bifurcation and some SFF8643 -> PCIe x4 adaptors.
You could then even use a mATX EPYC board like this one: ASRock Rack > ROMED6U-2L2T

 

[nivedita]

Much easier to take an existing switch (that has the ports you want) and water cool it. It's going to be a custom job, expensive (not that bad though ~$200-400), and will make it quiet without sacrificing network throughput.

LTT actually did this: Watercooling a network switch

 

[Agrikk]

Much easier to take an existing switch (that has the ports you want) and water cool it. It's going to be a custom job, expensive (not that bad though ~$200-400), and will make it quiet without sacrificing network throughput.

The easiest thing would be to cut a 120mm or 140mm hole in the top of each of my existing devices and unplug the 40mm fans increasing the airflow while reducing the noise, but that adds about 1U to each device. I'm trying to avoid that because my cabinet is filling up. But it would be the easiest way to go about quieting down everything.

LTT actually did this: Watercooling a network switch

Okay this is badass. Not sure I'd want to take a 1U switch and turn it into a 5U device, but what the hell.

 

[Neil Jefferies]

Supermicro H11SSl-NT is close - 7 slots 16/8x plus 2 onboard 10GB ports

 

[zer0sum]

quiet a few EPYC motherboards out there with 7 x PCIe Gen4 slots

Enterprise and Industrial Grade Motherboards for Automation, AI, IoT, Edge and Cloud Computing

World Wide Enterprise Server and Workstation Provider

www.prowesscomputing.com

And even a couple of Xeon ones with PCIe Gen3

ASRock Rack WC621D8A-2T C621 CEB

World Wide Enterprise Server, Workstation, and Components Supplier

www.prowesscomputing.com

Supermicro X11SPA-T C621 E-ATX

World Wide Enterprise Server, Workstation, and Components Supplier

www.prowesscomputing.com

 

[am45931472]

Dont do it. dont turn a pc into a switch. its not worth it. performance is terrible. reliability is terrible. Just look through the forums here and buy a brocade icx 7250 or icx 6610 if you dont need it to be too quiet and can stand some noise. ICX6610 can be had for like 150$ on ebay.

 

[Agrikk]

Dont do it. dont turn a pc into a switch. its not worth it. performance is terrible. reliability is terrible. Just look through the forums here and buy a brocade icx 7250 or icx 6610 if you dont need it to be too quiet and can stand some noise. ICX6610 can be had for like 150$ on ebay.

This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?

 

[lunadesign]

This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?

I was in a similar boat last year, hoping to have a single uber-quiet Xeon-D pfSense box play firewall, router and switch but also found the switching performance was lousy. I don't have numbers but everything you hear about ASICs ruling this space (network switching) is absolutely true. I ended up using the pfSense box for firewall and routing but am using fanless 24 port switches from Cisco Small Business and Mikrotik for switching. All of it has worked out great so far.

 

[Agrikk]

I was in a similar boat last year, hoping to have a single uber-quiet Xeon-D pfSense box play firewall, router and switch but also found the switching performance was lousy. I don't have numbers but everything you hear about ASICs ruling this space (network switching) is absolutely true. I ended up using the pfSense box for firewall and routing but am using fanless 24 port switches from Cisco Small Business and Mikrotik for switching. All of it has worked out great so far.

Well, I'm not sure I'm happy or sad that you've talked me out of it.

I was kinda looking forward to a weird project like this, but am happy to not spend the dosh on motherboard/cpu/ram/case/fiber HBAs/4-port gig NICs.

 

[Agrikk]

I'll be looking for a quiet 120mm or 140mm fan and some passive heatsinks for the various chipsets instead.

 

[nivedita]

This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?

This is a recent example. There's also more systematic performance measurements with DPDK. According to the latter, it looks like roughly speaking you need to dedicate at least one core to each 10G port.

 

[nivedita]

Also, there's the Mikrotik 16-port switch which can run with fans off at low utilization, plus the 8-port version which is fanless if you want a quiet switch. The ICX7150 is also fanless for non-POE models if you want a combination of gigabit and 10G.