Does Anyone know if there is any free software that runs on x86 machines that basically turns your pc into a network switch using the NICs in the pc?
Build your own switch
- Thread starter macrules34
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I'm curious, what's the goal?
Slow performance that also burns a lot of extra power, maybe?
Easiest would be just do a Linux Network Bridge, same as is done e.g. on Proxmox VE (Virtual Machines sharing the same Network / Network Switch).
You have free Options if you want a GUI, though historically these are more marketed as Routers: OpnSense, OpenWRT, VyOS, etc. Or just roll your own on top of whatever Linux (or FreeBSD/OpenBSD/NetBSD for that Matter) Distribution of your Liking if you are confortable with the Command Line.
Realistically this will only be useful / efficient up to gigabit Speeds. You will be able to achieve multi-gigabit (possibly up to 10gbps) but it will use lots of CPU Power, or you will need to rely on ROCE v2 or similar offloading Technologies.
Not everything that can be done should be done .
You have free Options if you want a GUI, though historically these are more marketed as Routers: OpnSense, OpenWRT, VyOS, etc. Or just roll your own on top of whatever Linux (or FreeBSD/OpenBSD/NetBSD for that Matter) Distribution of your Liking if you are confortable with the Command Line.
Realistically this will only be useful / efficient up to gigabit Speeds. You will be able to achieve multi-gigabit (possibly up to 10gbps) but it will use lots of CPU Power, or you will need to rely on ROCE v2 or similar offloading Technologies.
Not everything that can be done should be done
.CPU usage isn't that bad with Linux bridges, I have my laptop bridged through my desktop so they can have a 25G link rather than the 10G the rest of my desk network runs and the laptop will hit a CPU bottleneck on speed tests at 7ish Gbps before there's any noticeable load on my desktop.
Still not an amazing idea in most cases, just get a switch.
Still not an amazing idea in most cases, just get a switch.
A 40GB switch will be too noisy for the an office environment, I need some quiet. Like a desktop pc.
Some Devices can be set in
The only advantage would be if you only need a couple of Ports and you don't really need much Bandwidth, since you will have an overhead (CPU cycles unless
While I personally hate close-sourced Devices and most of >=10gbps stuff is Proprietary (Mikrotik, UniFi, etc), OpenWRT has some good Support for e.g. ONTi ONT-S508CL-8S / XikeStor SKS8300-8X is quite good Supported (provided you don't need crazy L2/L3 Features).
In any Case, the NIC / Switch Chip is rarely fully open source, so you somehow move the Issue one layer down, but it's still there.
netdev Mode (I heard Intel E810 is one of those, probably a Mellanox ConnectX-5/6 might as well, to be verified).The only advantage would be if you only need a couple of Ports and you don't really need much Bandwidth, since you will have an overhead (CPU cycles unless
netdev Mode) and when using multiple NICs you will be limited by PCIe Bandwidth too. Likely you'll have Issues with PPS Rate, not only Bandwidth in [gbps].While I personally hate close-sourced Devices and most of >=10gbps stuff is Proprietary (Mikrotik, UniFi, etc), OpenWRT has some good Support for e.g. ONTi ONT-S508CL-8S / XikeStor SKS8300-8X is quite good Supported (provided you don't need crazy L2/L3 Features).
In any Case, the NIC / Switch Chip is rarely fully open source, so you somehow move the Issue one layer down, but it's still there.
Somebody here did a NOCTUA Fan Mod of the Mellanox SX6036 plus some extended Fan Shroud, he seemed relatively happy with that
.I’m no good at the modification of equipment, unless it’s plug and play.Somebody here did a NOCTUA Fan Mod of the Mellanox SX6036 plus some extended Fan Shroud, he seemed relatively happy with that
Mikrotik is probably your best bet, 40G is going to be rough in a desktop because you can only get two ports per x8 slot and that quickly puts you into workstation/server board territory, which is louder and costs more than a typical desktop, plus getting 40G between NICs in a bridge going across the PCIe bus and through the CPU isn't going to be easy.
I’m not looking for blazing fast speeds, just something that can do iscsi And typical network switching quietly.
I think first you need to define your Requirements such as:
- How many Ports
- Which Bandwidth (1gbps/10gbps/40 gbps)
- Switching Capacity in PPS / MPPS (this will be much lower than what an ASIC can do IMHO)
- Which CPU and Motherboard are you looking into (number of PCIe Slots, PCIe Generation, PCIe Lanes)
- What Management do you need
I wouldn't dream on achieving 40gbps Speeds on a DIY non-ASIC Router with traffic passing through the CPU.
Unsure if
netdev would work on such old Devices, probably not.ROCE v2 is for L3 (Routing), not sure if there is something else besides
netdev for L2 (Switching).9x 40GB ports on one system/switch.
Dell desktop (tower) Xeon system with 5+ PCIe slots.
Basic command line management is fine, don’t need a gui.
Is it only Ethernet you need or also Infiniband ?
That's going to be 5 x dual port NICs. At 10gbps (maybe also 25gbps) you could at least get away with 3 x quad-port NICs which seems already more manageable.
CPU generation ... "Xeon" is a little bit generic ... is it a Sandy bridge E5-2620 or a Xeon E2286G or whatever ?
Just with the NICs I personally have a bad Feeling, you will need significant airflow over the Heatsink or a side Fan, so unless you have significant front-back Airflow (AKA server Noise) and you don't want to install noisy blower type fans on each NIC, you might not even have the space to install a 40mm Fan on each Heatsink (or something like that).
Not sure if/which NIC supports like P2P Mode across the same PCIe Root Complex, that would help with offloading.
At the very least I would expect 10-15W for each dual Port NIC, even at Idle / low-usage. That's already 50W-75W for NICs alone, say 15W for Motherboard, 5W for RAM and maybe 15W for CPU at idle. You are already in 110W-135W at Idle.
Most likely all 40gbps NIC don't support ASPM thus your Idle Power Consumption will be quite high (Package PC3 at best, most likely Package PC2). FYI Package PC6 or higher is needed in order for Power Consumption to drop considerably. Keep in mind that it will increase Latency a bit though.
It might be relatively quiet if you can DIY well, but it won't be power efficient for sure.
The SX6036 is much lower than that
Or mine with just 2 LR4 Transceivers plugged in. Fan Script to Mod Fan Noise. Still irritating, just not a jet engine.
The whole idea is to have a switch that is tolerable in an office setting while using 40 or 10 gbe NICs.
Most 40gbe NICs are going to be PCIe 3.0. That means you'll need 8 lanes of PCIe per port to get the full speeds, or 72 lanes total. You're looking at a full blown server build to get that many PCIe lanes, or paying way more to get gen4 cards. Even if you're fine with running dual port cards at 64gb/s PCIe, that's still a total of 40 lanes (4 dual port x8 + 1 single port x8). There's no way a system that can drive that ends up being cheaper and easier to build than a switch.
This might have made sense many years ago when 10gbe switches were still fairly expensive but early 10GbE NICs were dime-a-dozen, but there's just no way it makes sense now.
If 10gbe is fine, that makes it even worse compared to a real switch, either an already-quiet switch or a Noctua-modded switch. Given that you'd have to install cooling fans on a server anyway (and it would likely be louder), I don't see that coming out ahead in any scenario. A CRS326-24s-2q can be found used for $350-400, has 24x 10gbe and 2x 40gbe, lets you adjust fan speeds to a degree, and can be Noctua modded with about the same amount of effort as installing those same fans in a PC. You can certainly get a switch with fewer ports for less as well.
This might have made sense many years ago when 10gbe switches were still fairly expensive but early 10GbE NICs were dime-a-dozen, but there's just no way it makes sense now.