Building a 10GB home network - switch advice

[locoliberty]

Hello! Been a long time reader, from time to time, of the main sites posts and videos along with various other places. Im excited to start a homelab 10GB LAN sponsored by work.

The goal is 10GB Ethernet so devices have plenty of bandwidth to the server. The server you say, yes, well, its just a over build glorified emby box for my currently 30TB horde of 4k. The problem with 4k is when you get something like 4k77 without noise reduction it starts getting real close to the limits of 1G ethernet.

Current network is a netgear R9000 running VOXEL firmware with SPF to the server and a CM2000 for internet. The CM2000 was a recent upgrade and what kicked this all off, comcast(ew) recently upgraded the network and those with 1G plans not get 1.4... if they can support it.

Server current/planned specs:

https://pcpartpicker.com/list/jY47bh

The server currently has a Mellanox MCX311A-XCAT for SFP+

the gaming PC has 2.5g nic, and the streamer is straight ethernet 1g. I would like everything on the network to be able to communicate without bottlenecks

So what about the new network? I am wanting to run PFsense on the server in a VM. The PFsense VM will get one of the intel x550 NICs and the motherboards NIC. the motherboard NIC will connect to the modem, the intel x550 will connect to the switch for router duties. The servers second x550 port will connect to the switch for its "normal" communication of the Ubuntu system providing linux images via torrent and running emby. I would like to run a WIFI6 POE access point for wireless.

I think ive got "most" of my hardware sorted but im not exactly sure on a reasonable switch. something 8-16 ports + 2 uplinks(1 for pfsense, the other free).
Port needs immediately will be:
2x 2.5Ge(wifi and desktop)
1x 10Ge or SPF+ (server)
1x 1G streaming device
1x 10G SPF+ pfsense

So far ive really liked the MS510TXUP for this setup, but im curious what other options ive missed

a picture of the server because consumer hardware has some RGB to mesmerize the cats.

 

[Vesalius]

This is not what you asked for but would recommend caution with Netgate/pfSense and look at OPNsense if *sense is the direction you would prefer to go. If you have not seen them, read over the 2 recent threads below on this forum so you can make an informed decision before putting your faith in negates hands at the moment.

https://forums.servethehome.com/index.php?threads/freebsd-13-wireguard-pfsense-2-5.32252/

https://forums.servethehome.com/index.php?threads/netgate-removed-all-mention-of-pfsense-from-their-website-major-regressions-in-pfsense-2-5-and-newer.32984/

 

[locoliberty]

Yea, i may easily switch to OPNsense. May fight the fight to not use more netgate gear but my co-worker doesnt listen, so ill have to setup with pfsense to present my learning and such to justify buying the gear.

 

[Vesalius]

If you are willing to buy used, have the space for a rackmount and don’t need silent, a Ruckus icx7150 48zp would give you everything you listed in a switch with a lot of room to grow. Would be significantly more expensive new, but can be had in the 500-700 range used.

 

[locoliberty]

I do need fairly silent/quiet operation as the server setup is just in the office area at home. That would be cool though

 

[NateS]

Have you considered the MS510TX (non-UP) or MS510TXPP-100NAS (same thing but with PoE) version? It's a bit of a weird one in that it has 4x1G, 2x2.5G, 2x5G, and 2x10G (1 copper and 1 SFP+), but it sounds like that matches your needs almost exactly with a bit of room to grow. It's also $269 (or $349 for the PoE version), instead of $599 for the -UP, and I imagine it draws less power too.

 

[locoliberty]

Ive thought of those, but ive got the budget for the -up so I figure why not

 

[locoliberty]

This is not what you asked for but would recommend caution with Netgate/pfSense and look at OPNsense if *sense is the direction you would prefer to go. If you have not seen them, read over the 2 recent threads below on this forum so you can make an informed decision before putting your faith in negates hands at the moment.

https://forums.servethehome.com/index.php?threads/freebsd-13-wireguard-pfsense-2-5.32252/

https://forums.servethehome.com/index.php?threads/netgate-removed-all-mention-of-pfsense-from-their-website-major-regressions-in-pfsense-2-5-and-newer.32984/

On this *sense is because of a work, I can simply replace it after ive done the project. Are there other, possibly better, systems I could virtualize onto my server for my network?

 

[Vesalius]

Better is relative and *sense is one of the better multitool firewall/routers with a gui. Other options all linux based that would likely be faster if routing at 10G:

1. VyOS - CLI at the moment, but fast, lightweight has paid support tiers and free tiers (i am exploring this one)
2. Untangle - Gui based, has paid support tiers and free home tier I think.
3. Sophos - Gui based, also has paid support tiers and free home tier (Home Edition is limited to 4 cores and 6GB ram)
4. Danos - also CLI, functionality still being documented, Debian-based distribution that makes use of the Data Plane Development Kit (DPDK) libraries, also tinkering with this one in a VM. Should route well above 10g

 

[locoliberty]

Better is relative and *sense is one of the better multitool firewall/routers with a gui. Other options all linux based that would likely be faster if routing at 10G:

1. VyOS - CLI at the moment, but fast, lightweight has paid support tiers and free tiers (i am exploring this one)
2. Untangle - Gui based, has paid support tiers and free home tier I think.
3. Sophos - Gui based, also has paid support tiers and free home tier (Home Edition is limited to 4 cores and 6GB ram)
4. Danos - also CLI, functionality still being documented, Debian-based distribution that makes use of the Data Plane Development Kit (DPDK) libraries, also tinkering with this one in a VM. Should route well above 10g

I could probly get away with Sophos home edition. Dont really want to give it more than 4 cores anyways, im not doing anything "fancy"

 

[Bert]

I spent quite sometime on tinkering this. Don't spend time on pfSense as it won't work a swicthc.

VyOS would work as a switch but it is an overkill.

I was told to consider OpenSwitch. I haven't tried it but it seems like this way you can use the host for other purposes so it is pretty need idea. For Router purposes, I heard OpenWRT as an option.

Please be mindful for the difference between switch and router.

 

[newabc]

Better is relative and *sense is one of the better multitool firewall/routers with a gui. Other options all linux based that would likely be faster if routing at 10G:

1. VyOS - CLI at the moment, but fast, lightweight has paid support tiers and free tiers (i am exploring this one)
2. Untangle - Gui based, has paid support tiers and free home tier I think.
3. Sophos - Gui based, also has paid support tiers and free home tier (Home Edition is limited to 4 cores and 6GB ram)
4. Danos - also CLI, functionality still being documented, Debian-based distribution that makes use of the Data Plane Development Kit (DPDK) libraries, also tinkering with this one in a VM. Should route well above 10g

Last year when I saw Sophos, it was using Snort other than Suricata. That was why it limited home edition to 6GB ram(Last year's Snort still needed to run as processes, 200Mbps per process, or passed traffic if over. The newest Snort release implies thread.).

Update: In the Sophos forum, the Snort 3.0 on Sophos is still on Dev(link).

 

[jdnz]

I do need fairly silent/quiet operation as the server setup is just in the office area at home. That would be cool though

go have a look at the QNAP desktop switches (m408 and m2108 line) - either 8*1gbe+up to 4 combo 10/5/2.5/1 ports or 8 * 2.5gbe + 2 combo 10/5/2.5/1 ports.

I've got a m408-4c on the desk in my home office and it's fan is basically silent even under sustained 10gbe loads

 

[GHenschen]

I can speak to the Netgear MS510TX vs the MS510TXM as I own both. The MS510TX has worked flawlessly and despite having one fan I have never heard it or seen the fan light come on. It is silent and dependable. More recently I purchased the MS510TXM, which apparently is the successor to the TX. This switch has been nothing but problems. For starters look at the pictures of the new model. They show three fans and a light gray case for both the POE and non-POE version of the switch. What you get for the non-POE model has two fans and a darker gray case. I asked Customer Support about this and they said they would have to talk to marketing. Only the POE model gets the three fans according to them. They couldn't explain the case color difference.

The next issue is noise. With no load the unit runs the two fans at 2500 rpm and is hardly "whisper quiet" as claimed in the literature. If I added just one 10Gbase-T transceiver to one of the uplink ports the unit started cycling the two fans to 5000 rpm every 5-10 minutes. It was very noisy at 5000 rpm and the cycling just accentuated just how noisy. The 5000 rpm is just 50% duty cycle on the fans. 2500 rpm was 25% duty cycle. I can only imagine what 75% or 100% duty cycle would be like. Those with POE get three fans and a lot more heat than this model to dissipate, so expect more heat and noise issues than I encountered.

To cap things off, the MS510TXM I received dropped ports at random. This problem would occur multiple times per week. I own four other Netgear switches and an ORBI router/WAP and I have never had port drops occur on these. I opened a case with Netgear Support since the unit was still in the 90 day window for free support. They changed a few network settings remotely and things improved for a few days. When the problem returned the next week I asked for a replacement unit. I am on my second unit now and this one still idles at 2500 rpm but at least I can add a transceiver without triggering the 5000 rpm fan cycling. This second unit also drops ports randomly and I have another case with Netgear that has dragged on for a month with no determination what is causing the port drops and how to prevent them. My advice is steer clear of the MS510TXM. Due to the port drops and the heat and noise issues I experienced with the low power non-POE model, I would not touch the POE model with ten meter cattle prod.

 

[locoliberty]

I can speak to the Netgear MS510TX vs the MS510TXM as I own both. The MS510TX has worked flawlessly and despite having one fan I have never heard it or seen the fan light come on. It is silent and dependable. More recently I purchased the MS510TXM, which apparently is the successor to the TX. This switch has been nothing but problems. For starters look at the pictures of the new model. They show three fans and a light gray case for both the POE and non-POE version of the switch. What you get for the non-POE model has two fans and a darker gray case. I asked Customer Support about this and they said they would have to talk to marketing. Only the POE model gets the three fans according to them. They couldn't explain the case color difference.

The next issue is noise. With no load the unit runs the two fans at 2500 rpm and is hardly "whisper quiet" as claimed in the literature. If I added just one 10Gbase-T transceiver to one of the uplink ports the unit started cycling the two fans to 5000 rpm every 5-10 minutes. It was very noisy at 5000 rpm and the cycling just accentuated just how noisy. The 5000 rpm is just 50% duty cycle on the fans. 2500 rpm was 25% duty cycle. I can only imagine what 75% or 100% duty cycle would be like. Those with POE get three fans and a lot more heat than this model to dissipate, so expect more heat and noise issues than I encountered.

To cap things off, the MS510TXM I received dropped ports at random. This problem would occur multiple times per week. I own four other Netgear switches and an ORBI router/WAP and I have never had port drops occur on these. I opened a case with Netgear Support since the unit was still in the 90 day window for free support. They changed a few network settings remotely and things improved for a few days. When the problem returned the next week I asked for a replacement unit. I am on my second unit now and this one still idles at 2500 rpm but at least I can add a transceiver without triggering the 5000 rpm fan cycling. This second unit also drops ports randomly and I have another case with Netgear that has dragged on for a month with no determination what is causing the port drops and how to prevent them. My advice is steer clear of the MS510TXM. Due to the port drops and the heat and noise issues I experienced with the low power non-POE model, I would not touch the POE model with ten meter cattle prod.

Hmmm part of me wants to take this as a grain of salt, but this isnt an amazon review, so I should probly take it more seriously. This sucks as those models are almost perfect for me.

 

[GHenschen]

Hmmm part of me wants to take this as a grain of salt, but this isnt an amazon review, so I should probly take it more seriously. This sucks as those models are almost perfect for me.

I thought the MS510TXM would be the perfect complement to the MS510TX. It just hasn't worked out that way. I have two NAS devices as part of my network and am trying to establish a content feed to the two NAS units at higher speeds (2.5, 5 and 10 Gbe). I then use the lower speed NAS ports for content delivery from the NAS units. With the port drops on the TXM still being investigated, Netgear Level 2 support has me only using one port on the NAS units for now. I had to stop using the 2.5 Gbe content feed connection to my Synology NAS and the 1 Gbe content delivery port on the QNAP, while they investigated. One step forward and two steps backward. I added the MS510TXM to get more 2.5, 5 and 10 Gbe connections for the future, but instead it has brought instability and unwanted rearrangement to my network. A Level 3 support tech is suppose to get involved next week. We will see if that helps the situation.

You are correct about mine not being an Amazon review. I did not order from Amazon. In retrospect, I should have.

I bought the MS510TXM from the Netgear store February 5th. A week after I placed my order I got an email saying my order would ship in a couple days. They charged my credit card at that time. A week went by and no shipping notice with a tracking number. I opened a case to find out my order status. They told me the unit was on backorder and they could not give me an estimated delivery date. They closed my case. Another week goes by and another call. Still no answers, I ask for escalation, they waffle and only tell me they will pass my concerns up to their supervisor. They won't give me a name or phone number. They close my case again. Two more calls and another case later I finally get a unit allocated and shipped. By the time they send me the tracking number the unit shows up on my doorstep the same day they give me the shipping confirmation and tracking number. The date was March 15th. It took Netgear 40 days to fill my order from their store! They only waited a week to charge my credit card.

While I was waiting those many weeks for Netgear to ship my order I notice distribution channels have the MS510TXM model in stock and ready to ship. I could have gotten the unit much faster from a number of sources other than the Netgear store. The only advantage the Netgear store had was a lower price due to a "one time" 10% discount. I thought I would get quick shipment from Netgear because the Netgear store must have stock of a new Netgear model, right? Knowing what I know now, I will not order from the Netgear store again. Lesson learned.

 

[Iaroslav]

Not about the exact case, but these new mikrotiks are just awesome:

CRS317-1G-16S+RM | MikroTik

Smart Switch, 1 x Gigabit LAN, 16 x SFP+ cages, Dual Core 800MHz CPU, 1GB RAM, 1U rackmount passive cooling case, Dual Power Supplies

mikrotik.com

CRS312-4C+8XG-RM | MikroTik

Switch of the future: the first MikroTik product with 10G RJ45 Ethernet ports and SFP+

mikrotik.com

 

[newabc]

Not about the exact case, but these new mikrotiks are just awesome:

CRS317-1G-16S+RM | MikroTik

Smart Switch, 1 x Gigabit LAN, 16 x SFP+ cages, Dual Core 800MHz CPU, 1GB RAM, 1U rackmount passive cooling case, Dual Power Supplies

mikrotik.com

CRS312-4C+8XG-RM | MikroTik

Switch of the future: the first MikroTik product with 10G RJ45 Ethernet ports and SFP+

mikrotik.com

By the way, the SwitchOS is much better than RouterOS on switching on my CRS305-1G-4S+IN.

 

[GHenschen]

By the way, the SwitchOS is much better than RouterOS on switching on my CRS305-1G-4S+IN.

I purchased a CRS305-1G-4S+IN recently and mine came with RouterOS enabled. It took me a while, and several attempts, to get into SwitchOS mode, but I finally made it and I agree that for my immediate needs SwitchOS was the way to go. I have to bone up on RouterOS before trying it out. I found some Microtik training materials about RouterOS on the internet so I will see what I can learn from that. I love the switch design. No fan, POE capable, redundant power supplies and the switch is light and compact with very good ventilation. It won't win any beauty contests but it is very functional and reasonably priced. I wish they made an 6-8 port version of this particular design.

I tried it with three 10Gbase-T transceivers and the switch and the transceivers can get pretty hot. I was seeing 57-60C on the transceivers and 51-53C on the switch. I found a single AC Infinity S3 120mm multi-fan placed under the switch blowing up dropped the temps of the transceivers and the switch by 10-18C depending on the fan speed you choose. This fan is USB powered and nearly silent at the low speed and still very quiet at medium speed. The fan fits this switch very well and turns hot into just warm and makes it safe to load up the ports with at least three 10Gbase-T transceivers. I haven't tried four yet but when I buy my fourth one I will give it a go.

 

[newabc]

I purchased a CRS305-1G-4S+IN recently and mine came with RouterOS enabled. It took me a while, and several attempts, to get into SwitchOS mode, but I finally made it and I agree that for my immediate needs SwitchOS was the way to go. I have to bone up on RouterOS before trying it out. I found some Microtik training materials about RouterOS on the internet so I will see what I can learn from that. I love the switch design. No fan, POE capable, redundant power supplies and the switch is light and compact with very good ventilation. It won't win any beauty contests but it is very functional and reasonably priced. I wish they made an 6-8 port version of this particular design.

I tried it with three 10Gbase-T transceivers and the switch and the transceivers can get pretty hot. I was seeing 57-60C on the transceivers and 51-53C on the switch. I found a single AC Infinity S3 120mm multi-fan placed under the switch blowing up dropped the temps of the transceivers and the switch by 10-18C depending on the fan speed you choose. This fan is USB powered and nearly silent at the low speed and still very quiet at medium speed. The fan fits this switch very well and turns hot into just warm and makes it safe to load up the ports with at least three 10Gbase-T transceivers. I haven't tried four yet but when I buy my fourth one I will give it a go.

I am using DAC cables and SFP+ NICs with this switch. Maybe the traffic is keeping low, so haven't got so hot yet. But maybe the 10Gbase-T transceivers produce more heat when converting SFP+ to RJ45.