Does an 8+ slot PCIe 4x motherboard exist?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
I recently came across an article about VyOS and now I am intrigued about building my own router/firewall, but the port count on my existing switches is rather high. I plan on populating a motherboard with dual-port 10g fiber HBAs and 4-port 1gig copper NICs.

I've seen plenty of mining motherboards with tons of PCIe 1x slots, but 1x will throttle the throughput on these NICs.

Has anyone come across motherboards with higher slot densities?
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,030
1,439
113
Supermicro X9DRX, X10DRX, or X11DPX-T come to mind.
 

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
Oh wow. Those are cool looking boards. A pity they are dual CPU, though. They will be a little overpowered for a simple switch. :D
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,030
1,439
113
There aren't enough PCIe lanes to go around on most platforms with a single CPU. Pretty much looking at dual CPU only and you would need to populate both.

You may also find that software switching performance can be lackluster in comparison to hardware with an ASIC.
 

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
Yeah, I expect as much. This whole thing reeks of a Bad Idea but I'm considering doing it anyways. :p I'm hoping that I can replace three devices (Sonicwall NSA 4500, Force10 10g switch, PowerConnect 6248P gig switch) and their resulting crazy-loud fans with something home built and quieter but still performant.
 

kapone

Well-Known Member
May 23, 2015
1,095
641
113
Much easier to take an existing switch (that has the ports you want) and water cool it. It's going to be a custom job, expensive (not that bad though ~$200-400), and will make it quiet without sacrificing network throughput.
 
  • Like
Reactions: nasi

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,285
778
113
Would you be okay with a bit of Frankensteining?

Because in that case I'd it might be an idea to use two or three China-PCIe-Switches (e.g. 119.0US $ |ANU28PE16 NVMe SSD Riser SFF8643 zu SFF8639, 8 Port 12Gbs,(nicht mit kabel, nicht unterstützung LSI 8643*2 zu 8639*2 kabel)|Add-On Karten| - AliExpress) in case the board does not support bifurcation or just an PCIe->SFF8643 Expander (e.g. 26.0US $ |ANU04PE16 NVMe Controller SFF8643 SFF8639 4 port PCIe 3,0 X16 SSD Exp Riser (nicht mit kabel, nicht unterstützung LSI 8643*2 zu 8639*2)|Add-On Karten| - AliExpress) if the board does support bifurcation and some SFF8643 -> PCIe x4 adaptors.
You could then even use a mATX EPYC board like this one: ASRock Rack > ROMED6U-2L2T
 
  • Like
Reactions: nasi

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
Much easier to take an existing switch (that has the ports you want) and water cool it. It's going to be a custom job, expensive (not that bad though ~$200-400), and will make it quiet without sacrificing network throughput.
The easiest thing would be to cut a 120mm or 140mm hole in the top of each of my existing devices and unplug the 40mm fans increasing the airflow while reducing the noise, but that adds about 1U to each device. I'm trying to avoid that because my cabinet is filling up. But it would be the easiest way to go about quieting down everything.

Okay this is badass. Not sure I'd want to take a 1U switch and turn it into a 5U device, but what the hell. :D
 

zer0sum

Well-Known Member
Mar 8, 2013
846
472
63
quiet a few EPYC motherboards out there with 7 x PCIe Gen4 slots

And even a couple of Xeon ones with PCIe Gen3 :)
 

am45931472

Member
Feb 26, 2019
87
17
8
Dont do it. dont turn a pc into a switch. its not worth it. performance is terrible. reliability is terrible. Just look through the forums here and buy a brocade icx 7250 or icx 6610 if you dont need it to be too quiet and can stand some noise. ICX6610 can be had for like 150$ on ebay.
 

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
Dont do it. dont turn a pc into a switch. its not worth it. performance is terrible. reliability is terrible. Just look through the forums here and buy a brocade icx 7250 or icx 6610 if you dont need it to be too quiet and can stand some noise. ICX6610 can be had for like 150$ on ebay.
This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?
 

lunadesign

Active Member
Aug 7, 2013
256
34
28
This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?
I was in a similar boat last year, hoping to have a single uber-quiet Xeon-D pfSense box play firewall, router and switch but also found the switching performance was lousy. I don't have numbers but everything you hear about ASICs ruling this space (network switching) is absolutely true. I ended up using the pfSense box for firewall and routing but am using fanless 24 port switches from Cisco Small Business and Mikrotik for switching. All of it has worked out great so far.
 

Agrikk

Member
Sep 6, 2013
39
3
8
Oakland, CA
I was in a similar boat last year, hoping to have a single uber-quiet Xeon-D pfSense box play firewall, router and switch but also found the switching performance was lousy. I don't have numbers but everything you hear about ASICs ruling this space (network switching) is absolutely true. I ended up using the pfSense box for firewall and routing but am using fanless 24 port switches from Cisco Small Business and Mikrotik for switching. All of it has worked out great so far.
Well, I'm not sure I'm happy or sad that you've talked me out of it. :D

I was kinda looking forward to a weird project like this, but am happy to not spend the dosh on motherboard/cpu/ram/case/fiber HBAs/4-port gig NICs.
 

nivedita

Member
Dec 9, 2020
41
22
8
This is literally the problem I am trying to solve. I have Force10 and PowerConnect 6248 switches and a Sonicwall NSA 4500 firewall appliance and the shriek of their 40mm fans has gotten old. I was toying with the idea of building a switch to cool it with 120mm fans.

Is there any data about the performance hit of running a PC-based switch?
This is a recent example. There's also more systematic performance measurements with DPDK. According to the latter, it looks like roughly speaking you need to dedicate at least one core to each 10G port.
 

nivedita

Member
Dec 9, 2020
41
22
8
Also, there's the Mikrotik 16-port switch which can run with fans off at low utilization, plus the 8-port version which is fanless if you want a quiet switch. The ICX7150 is also fanless for non-POE models if you want a combination of gigabit and 10G.